8
$\begingroup$

Abstract: "Back It Up" Policy for Answers

In this post, I propose updating the policy on Economics.SE for deletion of low-quality answers. While there has been some recent discussion regarding our site’s policy regarding deletion of low-quality answers, this question has come up on this site multiple times (reference provided below). A recent post has demonstrated that there is significant community support to develop more detailed guidelines for deletion of posts. Here, I review the policies adopted by other Stack Exchange sites and the rationale behind these policies and review criteria for how these policies might be successfully adopted on this site. In the end, I propose a stricter set of guidelines specifically for the deletion of answers. Specifically, I propose that we adopt the “back it up” principle adopted by over a dozen other stack exchange sites: a non-mathematical answer (or similar) must be backed up by citations or is subject to deletion. Here is an example. This policy reduces the amount of subjective judgement needed to determine whether a post needs to be deleted while also significantly increases the quality of posts on the site.

  • The question portion of this post reviews the policies of other sites and discusses the rationale for them. Upvote this question if you support a revision of our policy.
  • The answer portion of this post will contain specific policies that we might adopt. I post a specific policy based on an assessment of other sites’ policies. Upvote my answer if you support the specific policy that I propose. Also, feel free to submit alternative proposals.

Reviewing Answer-Deletion Policies Across Sites

On this site, we have discussed the problems associated with low-quality answers several times. I list these here.

A common thread among these posts is the tension arising from the inherent subjectivity of determining whether an answer is of low quality. Fortunately, this problem has been already been solved (or at least mitigated) on many other Stack Exchange sites through the adoption of the so-called “back it up” principle. For most sites, this means that an answer must be backed up with a citation of some sort. On our site, it might mean something slightly different. For example, an answer to a question of a mathematical nature must be backed up with mathematical argumentation (obviously). A question of a more “wordy” nature must be backed up with a reference or citation to, say, a peer-reviewed journal. As argued on another site,

“For anyone who is trying to do actual research, answers without sources are useless. I'm not even going to talk about how answers without sources are often inaccurate, but I've noticed that people who know where they got their information from usually don't misremember things or post incorrect facts.”

The rationale, summarized in an oft-cited blog post by Robert Carataino is that “opinion, by itself, is noise.” If an answer is not backed up, it should be subject to deletion. See here for a list of sites that have adopted a policy of this nature: How widespread is the "back it up" principle?

Criteria for a Successful “Back It Up” Policy on Economics.SE

The idea of “back it up” is universal. However, since each site on Stack Exchange is different, the particulars of implementing such a policy are subtly different. Here I review some criteria that such a policy should have to be as successful as possible.

  1. Because Economics.SE contains mathematical and computational questions, as well as discourse-based “wordy” questions, simply requiring all answers to contain citations is not a sufficient policy. However, the principle of “back it up” should be universal---since mathematical answers, say, can be backed up by mathematical argumentation. A successful policy on our site needs to articulate this.
  2. The policy should be effective! That is, it should reduce the visibility of unsourced answers and other answers of low-quality. One way to do this is to simply encourage users to downvote such answers. Given that this site attracts a lot of laypeople who are very willing to upvote unsourced answers, I believe that the policy should be to delete such answers.
  3. Such a policy should put as little burden on moderators as possible. Deletion of answers can only be done by moderators, with the exception of answers with a negative score. Given this, the community should be vigilant to downvote answers that do not satisfy this policy. (A downvote can be undone if the author fixes the answer via edits.) When such answers do not get the necessary downvotes, enforcement of the policy rests with the moderators. Thus, the policy should be such that a moderator can decide whether or not an answer should be deleted almost instantly and without effort. To accomplish this, there should be a simple, nearly programmatic rule for this. An example that might arise, for example, is that a moderator wants to allow for some sort of grace period for the author to revise the answer. In this case, a programmatic rule would be that an unsource answer MUST be deleted if it is older than 3 days, for example. By tying the hands of the moderator in this way, we in fact make their job easier.
  4. How should this policy be applied to very old answers? There are a lot of old, unsourced answers on this site. A successful policy must describe how to deal with these. My take is that they will either be fixed by the community slowly over time or that they will be eventually deleted. For example, over time community members edit them or will flag them to bring them to the attention of moderators and they will slowly be resolved. If we’ve tied the hands of the moderators, as described above, the solution will be simple. Remember, “opinion, by itself, is noise.”
  5. The policy should be short and simple. Short and simple policies are the most effective.
$\endgroup$
5
  • 1
    $\begingroup$ "Deletion of answers can only be done by moderators." This does not apply to answers with a negative score. $\endgroup$ – Michael Greinecker Mar 22 at 21:18
  • 1
    $\begingroup$ @MichaelGreinecker Oh. Thanks, this makes a pretty interesting difference. I've edited the proposal to reflect this. The new commentary that I've added is this: "Deletion of answers can only be done by moderators, with the exception of answers with a negative score. Given this, the community should be vigilant to downvote answers that do not satisfy this policy. (A downvote can be undone of the author fixes the answer via edits.) When such answers do not get the necessary downvotes, enforcement of the policy rests with the moderators. " $\endgroup$ – jmbejara Mar 22 at 21:30
  • 1
    $\begingroup$ With apologies if I have missed something, where is a statement of the current policy on low quality answers that you propose to update? In the Help Centre under "How do I write a good answer?" and "Why and how are some answers deleted?" there is only general guidance which does not appear specific to this site (unlike for questions where "What topics can I ask about here?" contains specific guidance for Econ SE). $\endgroup$ – Adam Bailey Mar 23 at 23:43
  • $\begingroup$ @AdamBailey I think jmbejara refers to this: economics.meta.stackexchange.com/questions/2084/…. Also note those help pages, except the ones for off/on topic cannot be edited by us they are generic, so it is common across sites to specify additional rules/policies on meta like for example here on skeptics: skeptics.meta.stackexchange.com/questions/1505/… $\endgroup$ – 1muflon1 Mar 24 at 8:25
  • 1
    $\begingroup$ Yeah. For better or for worse, a lot of site-specific policy lives in meta posts rather than on the FAQ. Here's a question that talks about it: meta.stackexchange.com/questions/279879/…. One comments on one answer talk about our situation: "some communities have a stronger back-it-up policy than others. It would be nice to let them create a page with that community-specific policy defined in one place that's more controlled and easier to find than a random Meta question." $\endgroup$ – jmbejara Mar 24 at 15:49
8
$\begingroup$

Here is my proposed FAQ on answers, the text is based on mashup of ideas in and under the two other answers of HerK and Jmbejara, as well as putting them into appropriate FAQ format (i.e. this FAQ has to be new user friendly), please feel free to comment on the FAQ or provide further answers that could be incorporated in. Once we agree on precise wording and policy we will create separate community wiki and update notices and other relevant materials with links to it.


FAQ on Answers:

I am a new user here on Economics.SE. What do I need to know about writing an answer here?

Welcome to Economics: Stack Exchange. Economics:SE is a question and answer site for those who study, teach, research and apply economics and econometrics. It's built and run by you as part of the Stack Exchange network of Q&A sites. With your help, we're working together to build a library of detailed answers to every question about economics.

Economics:SE is Different

Economics:SE different to many online Q&A sites and forums (such as mailing lists, bulletin boards, and the commenting systems on blogs.) It is also fairly different to many other non-science web sites in the Stack Exchange family, such as Stack Overflow, that you may be familiar with. As a result, the Economics.SE community can appear to be a little demanding on new users. Many — perhaps even most — first time users are surprised when their contributions are judged against an unexpectedly high set of standards. We urge you not to be disheartened by this. Have a look around, and you will quickly understand how we work. We hope that you will soon appreciate the value that these standards bring in ensuring that the answers on Economics:SE are of very high quality and reliability.

These standards can be summarized as simply as: "Back it up!" This means that answers on this site have to be supported either by references to reputable and/or scholarly sources or by formal (mathematical) model or by sharing empirical results backed up by replicable example of code and data.

1. Answers That Need References

The biggest surprise to new users may be our insistence on references in the answers. Many forums will appear to simply accept at face-value the word of a random internet denizen. Here, we expect to be able to independently check what you are saying, as we want to chase down the evidence, rather than relying on authority or personal expertise. You should expect that people will actually verify whether your references say what you claim they say. Ideally, we would like to see links to peer-reviewed evidence. This makes writing an answer harder, but the good answers are appreciated much more.

To be more specific, answer that:

  • contains a claim of an empirical or theoretical nature that concerns primarily with inferred information (i.e. information obtained as a result of some empirical or theoretical analyses), the sources cited must also be reputable and scholarly source;

  • addresses a data inquiry or data request, it must provide links or references to the data or to a data codebook or data guide;

  • claims about factual information (e.g. "Event X happened" or "So-and-so said this") be supported by reputable (or reputable and scholarly) sources.

What is reputable and scholarly source?

A source is reputable if its publisher or author(s) have the reputation within the economics profession for maintaining high standards of fact-checking, accountability, ethical reporting or production of high quality scholarly or professional work related to economics.

A source is scholarly if its publication requires a peer-review process.

Reputable and scholarly sources are:

a) peer reviewed articles published in reputable journals (i.e. predatory journals are not allowed, if in doubt refer to Scimago Journal & Country Rank of journals in economics or other fields).

b) peer reviewed handbooks or books that could be cited in reputable journals (but not undergraduate textbooks, save for circumstances mentioned under "Are There Any Exemptions?").

c) curated working paper series (e.g. NBER or MPRA that undergo limited peer review but not SSRN which does not - with exception of SSRN articles from authors affiliated reputable high ranking university; please refer to top 200 universities in QS Economics & Econometrics ranking).

d) technical reports produced by widely recognized government or supranational organization which specialize or heavily participate economics research (e.g. IMF, World Bank, ECB, BoE, Fed, OECD etc.).

What is only reputable source?

a) News articles (but NOT opinion pieces) in a respectable news outlet such as Associated Press, Reuters, Bloomberg, etc.

Does that mean that popular books or blogs from economists are not allowed on this site?

No, but these do not count as a support for an independent claim. You should feel free to quote or use material from these and reference them but they should not be the only sources used to back up a claim. For example, if you think that certain blog provides excellent intuition behind a certain result you can use reputable and scholarly sources as a support for the result itself and blog as a source for intuitive explanation.

2. Answers That are Supported by Theoretical Model

These are answers supported by mathematical argumentation and derivation from certain premises. You should still provide reference for the validity of premises unless they fall under exceptions listed under "Are There Any Exemptions?" or if question requests you to use certain premises (please note that graphical analysis, done rigorously, is also form of mathematical analysis e.g. supply-demand charts etc).

3. Answers That are Supported by Empirical Model/Estimation

These answers should be explicit about functional form of their model and provide access to replicable code and data. Note code and data can be provided via link to external sites (such as GitHub) or link to external dataset (unless user combined several dataset or manipulated data in a way not reflected in the code).

Are There Any Exemptions?

Yes there are few exceptions from the rules above:

  • Common knowledge. What is common knowledge? Common knowledge is a knowledge that everyone or virtually everyone knows. Examples of common knowledge include: "Canada is country in North America", "USD is currency of USA", "1 is an odd number". Remember: whenever in doubt whether your statement is common knowledge, please follow the rules listed in previous sections!

  • Common knowledge in economics. What is common knowledge in economics? It is a knowledge that anyone who ever studied economics 101 should know. It is general knowledge you would find in introductory economics texts such as Samuelson: Economics, Mankiw Principles of Economics, Krugman: Economics for AP and so on... (e.g. statements like 'profit maximizing monopoly will produce quantity at a point where marginal revenue equals marginal cost'). Remember: whenever in doubt whether your statement or claim is common knowledge in economics, please follow the rules listed in previous sections! (Note: you can provide citations to introductory economics texts as a proof of claim being common knowledge, if challenged).

  • Common knowledge in mathematics, statistics & econometrics. What is common knowledge in in mathematics, statistics & econometrics? It is a knowledge that anyone who ever studied in mathematics, statistics & econometrics 101 should know. It is general knowledge you would find in introductory texts such as Sydsaeter et al: Essential Mathematics for Economic Analysis or Newbold et al: Statistics for Economics and Business or Stock and Watson Introduction to Econometrics etc... Remember: whenever in doubt whether your statement or claim is common knowledge in mathematics, statistics & econometrics, please follow the rules listed in previous sections! (Note: you can provide citations to introductory texts as a proof of claim being common knowledge, if challenged).

  • When you are writing an answer for question with tag references are encouraged but not required. Note: tag should not be abused and only used for informal questions about workings or practices within economics profession. Repeated abuses of tag after warning might result in moderator intervention.

  • When you are writing an answer for question with tag and original poster specifies that they are looking for a reference to literature not mentioned in the previous section.

What Happens if I do not Follow These Rules?

Your contributions can get flagged and receive moderator post notice

enter image description here

Answers that are at least 3 days old and are still not backed up will be deleted.

What Should I do if I See an Answer That does not Follow These Rules?

Please consider either:

  1. Edit the answer and add missing sources if you know sources that would support the answer.

  2. flag the answer using:

  • low quality flag

or

  • needs moderator intervention flag and write in "needs support/references/sources" or an equivalent of that (feel free to specify which paragraphs lack sources).

I Flagged Old Question for Lack of Sources and the Flag was Declined, What is Going on?

We implemented this policy on insert date. Previously, we only encouraged the 'back it up!' principle but did not enforced it.

As a result this site might have some old but great answers that do not fully comply with these rules. While we would ideally want to bring these old answers to our current standards, some of the past users might no longer be active and it would not be fair to delete their great contributions.

Consequently, moderators reserve right to apply the 'back it up!' policy on case by case basis for any answer posted before insert date (unless the post already received post notice in the past).

However, you should still feel free to edit these old answers and add sources that support the answer.

Conclusion

If you never visited Economics:SE these rules might seem a little bit demanding for an internet site. However, if you stay with us for a bit you will see that we take economics seriously and provide great answers and thoughtful answers. If you will stay with us longer, you will soon start to be disappointed that other forums don't demand the same standards.


Some remarks

  • the FAQ above tries to be a compromise as such I think everyone will dislike some part of it but please let's not let perfect be the enemy of what can be done.

  • the FAQ above copies the scholarly and reputable definitions from HerrK but not authoritative. This is because I can foresee that there will be borderline cases where someone's favorite economist is not well known or heterodox economist and there will be arguments about that. Unless we would wanna go the same way as Physics.SE and only allow mainstream science this might be bad idea and it might lead to fights and too much work for mod team to adjudicate who is authoritative and who is not.

  • it does not allow entry level textbooks (save as proof of common knowledge when challenged), this makes it bit more strict and in line with original jmbejara proposal even if it does not go as far).

  • it also allows for bit of ambiguity (one of the michaels points)

  • it changes wording to at least 3 days, I will strive and also encourage other mods to do it after 3 days but were are volunteers there might be days (summer vacation times) where our vacations might overlap or we might accidentally miss an answer in periods of high activity so I prefer having there at least 3 days to avoid complaints that someone's post got bit more time before deletion.

  • it allows for case by case dealing with old answers (I think this is only fair to past answers - especially for users that are not active right now but might want to come back to Economics.SE in the future).

EDIT:

I added provision for allowing use of graphs to support theoretical answer. In addition, I removed the right of moderators to delete very bad answers sooner than 3 days.

PS: This post is in essence just mash up of Herr K and jmbejara answers, thus credit should be yours guys. If this does matter to both or one of you, you can both post parts of this answer into community wiki which will record the contribution as being from you (community wikies allow for multiple contributors). Of course, this will be extended to any other users whose ideas might shape this FAQ.

$\endgroup$
9
  • 2
    $\begingroup$ I hope you don't mind that I made some cosmetic changes to some English expressions. But overall I like what you've written and I'd be happy to support your version. $\endgroup$ – Herr K. Mar 25 at 1:46
  • 1
    $\begingroup$ @HerrK. I don’t mind thanks $\endgroup$ – 1muflon1 Mar 25 at 7:19
  • 1
    $\begingroup$ I like this alot. In particular, I think it does well in keeping an inviting tone even though the rules are strict. Two suggestions: I would soften the SSRN claim. To something "SSRN papers need not be reputable and scholarly." Often, there are other criteria such as authorship that clarify the scholarliness. Also, I would add a proviso that what can be taken to be common knowledge might vary with the context of a question. $\endgroup$ – Michael Greinecker Mar 25 at 19:33
  • $\begingroup$ @MichaelGreinecker thanks, I would not mind softening the SSRN policy, but can you suggest a way to do it? My issue is that while I think many SSRN contributions are good (and I myself have working papers listed there) I seen people posting there literally just essays with very little scholarships and they somehow still listed. Perhaps I could add provision unless the SSRN paper is published elsewhere? Regarding common knowledge I would rather prefer to have some stated areas (although we can expand it if you think it is necessary), because I think that would be abused too much. $\endgroup$ – 1muflon1 Mar 25 at 19:48
  • 1
    $\begingroup$ @1muflon1 Maybe requiring authors of SSRN papers with affiliations from university or reputable research institution. Regarding the common knowledge thing, I was more worried about more advanced questions, where more things can reasonably be taken to be common knowledge. If someone asks how Bayesian Persuasion proofs with a compact metric type space work, the answer will utilize results that go beyond Sysaeter and Hammond's books. $\endgroup$ – Michael Greinecker Mar 25 at 19:57
  • $\begingroup$ @MichaelGreinecker ok thats a good suggestion for SSRN, I will add that in. Regarding the common knowledge, I think in that case it is better to rather reference some handbook (non-entry level handbooks are still allowed) I would hope that would not be too troublesome. This is because in such cases it might be difficult to evaluate for us mods if it indeed is common knowledge and then that could lead to disputes if someone does not like an answer $\endgroup$ – 1muflon1 Mar 25 at 20:03
  • 1
    $\begingroup$ @1muflon1 I don't think mods need to evaluate technical answers to advanced research topics. I can't imagine that such answers will cause much controversy. In the context of graduate microeconomics, the first welfare-theorem might well be assumed to be common knowledge. And if a reference is needed, good online lecture notes would be a more useful reference than sending the reader to a hard-to-find textbook. $\endgroup$ – Michael Greinecker Mar 25 at 20:27
  • $\begingroup$ @MichaelGreinecker well I would not mind also doing further adjustments but I dont like the wording of it being context specific because that will be definitely abused. I would even say perhaps let it be as it is. Just to be clear I do not plan to be a person who would just go from question to question and just wait for a 'gotcha' moment to past the post notice on a post, if there will be high quality answer people will likely not flag it and it will not have a problem and in rare cases if they arise I can first just leave comment with gentle reminder I think this should not be too cumbersome $\endgroup$ – 1muflon1 Mar 25 at 20:37
  • $\begingroup$ @1muflon1 I think that is reasonable; it should work out fine this way. $\endgroup$ – Michael Greinecker Mar 25 at 20:47
5
$\begingroup$

Definitions

The following definitions of terms are adapted from here:

A source is reputable if its publisher or author(s) have the reputation within the economics profession for maintaining high standards of fact-checking, accountability, ethical reporting or production of high quality scholarly or professional work related to economics.

A source is authoritative if its publisher or author(s) have established expertise in the relevant subject area(s) as evidenced by their past scholarly publications or comparable professional accomplishments in the said area(s).

A source is scholarly if its publication requires a peer-review process.

The idea for introducing these three definitions is to require

  • claims about factual information (e.g. "Event X happened" or "So-and-so said this") be supported by reputable sources, and
  • claims about inferred information, i.e. information obtained as a result of some empirical or theoretical analyses (e.g. "Most of X has property Y" or "X causes Y") be supported by either reputable-and-authoritative or reputable-and-scholarly sources.

The reason for accepting "reputable-and-authoritative" sources is that opinions of a well-established expert, on a topic of their expertise, deserve serious consideration by virtue of them being authoritative--a usually hard-earned status, even if we may end up disagreeing with them. Moreover, if one day Al Roth comes here and answers questions on market design, his answers would carry weight even if they are not properly sourced. The reputability requirement is there to rule out self-proclaimed "experts" in some niche areas. Likewise, scholarly sources should generally be acceptable, and I throw in a reputability requirement there to rule out articles from dodgy journals.

The definitions purposefully leave room for accepting good quality professional sources. If we are serious about including them as part of our core user base, non-academic professional economists, e.g. PhDs working in the "industry" --- banks, consultancies, think tanks, governments, international organizations, etc. --- may have a slightly different set of trusted sources than what academic economists deemed acceptable. While such trusted professional sources would typically not be scholarly, they should still pass some form of reputability and authoritativeness criteria --- or so I would think. Input from member professional economists on this point would be especially welcomed.

Examples satisfying the above definitions

For completeness, I provide examples for all possible combinations of the three criteria. I don't expect full consensus on my placement of the examples, but a lot of the potential disagreements won't matter for the purpose of formulating our policy, which only requires the distinction between being reputable, reputable-and-authoritative, and reputable-and-scholarly.

Reputable but not necessarily authoritative or scholarly:

  • News articles (but NOT opinion pieces) in a respectable news outlet such as Associated Press, Reuters, Bloomberg, etc.

Authoritative but not necessarily reputable or scholarly:

  • A working paper on a niche topic by an otherwise "well-established" economist (e.g. Bart Lipman's 12-year-old working paper)

Scholarly but not necessarily reputable or authoritative:

  • An article by a little known author in a very lowly ranked journal

Reputable and authoritative but not scholarly:

  • A "popular science" book by a well-known and well-established economist (e.g. Al Roth's book)

Reputable and scholarly but not authoritative:

  • The first publication by a young economist in a decent journal

Authoritative and scholarly but not reputable:

  • An article on a niche topic in a non-economics journal by an otherwise well-established economist (e.g. Arthur Robson's secret handshake paper)

Reputable, authoritative, and scholarly:

  • Many articles in decent journals
  • Many books by well-known academic publishers

Policy Statement

Given the above definitions and in light of @jmbejara's answer and the comments under it, I propose the following revised version of the "Back it up" Policy:

Effective on <date>, each answer on Economics.SE must be appropriately backed up; otherwise it is subject to deletion. "Backing up" an answer means providing suitable support for the claims made therein. Specifically, if an answer:

  1. contains a claim of a mathematical or theoretical nature, that claim must be supported by mathematical arguments or reference to reputable-and-authoritative or reputable-and-scholarly sources;
  2. contains a claim of an empirical nature, that claim must be supported by referring to reputable sources; in addition, if the claim concerns primarily with inferred information, the sources cited must also be either authoritative or scholarly;
  3. contains a claim of a computational nature, that claim must be supported by code or references to such;
  4. addresses a data or reference request, it must provide links or references to scholarly or authoritative sources, unless explicitly requested otherwise by the asker.

Answers that are 3 days old and are still not backed up WILL BE DELETED. Community members should flag posts that don’t meet these standards as “low-quality answers” to bring them to the attention of moderators.

I should note that this version is itself subject to revisions and improvement.


Remarks

  • The inclusion of an "effective date" give us some leeway in dealing with "old violations".
  • The policy require claims to be supported, so an answer with multiple claims would call for separate support for each of them.
  • Newspapers are allowed for facts only.
  • On inferred information, the following "reasonable" examples would clear the bar:
    • most common journal articles
    • most working papers by professors tenured at major universities
    • NBER WP series
    • academic books and handbooks published by major academic publishers
    • papers presented at major conferences/workshops/seminar series
    • most graduate textbooks
  • Cases that require judgment/discussion:
    • widely adopted undergraduate textbooks (e.g. Mankiw)
    • reports by (trusted) government, industry or international bodies (e.g. IBIS World Industry Reports, publications by the IMF/World Bank/OECD)
    • trade journals
    • <...>
$\endgroup$
13
  • $\begingroup$ I like a lot of that is written here but you somehow omitted also theoretical claims, claims about mechanisms (which are empirically testable but not necessarily empirical themselves), in that definition. There we should have rule as for that inferred information or that one should simply write out the model formally $\endgroup$ – 1muflon1 Mar 24 at 8:02
  • $\begingroup$ Also additional issue if there will be user with nick Al Roth how do we verify it’s actually the author, I seen several Keyneses, Walrases and so on here. $\endgroup$ – 1muflon1 Mar 24 at 8:09
  • 1
    $\begingroup$ @1muflon1: Perhaps we can broaden the definition of "inferred info" to include theoretically derived statements as well? As in: "inferred information (i.e. information obtained as a result of some empirical or theoretical analyses)". Correspondingly, point 2 in the policy would read: "contains a claim of an empirical or theoretical nature..." $\endgroup$ – Herr K. Mar 24 at 14:28
  • $\begingroup$ If the real Al Roth do become a member, I think it would be fairly easy to tell from the quality of their first few answers. More generally, my point is that if for example someone cites (correctly, not out of context) a blog post by Roth on a relevant issue, that citation, albeit not scholarly, should be given serious consideration. $\endgroup$ – Herr K. Mar 24 at 14:39
  • 1
    $\begingroup$ I prefer this version. But I think the phrasing in 1. and 2. needs some adaption. For mathematical claims, one needs to give a mathematical argument (1.), for theoretical claims, one needs a reputable source (2). These should really be exchangeable, and the dividing line is not clear. $\endgroup$ – Michael Greinecker Mar 24 at 20:14
  • $\begingroup$ @MichaelGreinecker: There is indeed some overlap between math and theoretical claims. But I think we shouldn't rule out "verbal" theories (à la Coase (1937)), even though they are few and far between nowadays. $\endgroup$ – Herr K. Mar 24 at 21:07
  • $\begingroup$ More specifically, I intend for mathematical claims to be something like "Cobb-Douglas function is concave" and for theoretical claims something like "Population diversity causes more conflicts" or "Torture is ineffective". $\endgroup$ – Herr K. Mar 24 at 21:07
  • $\begingroup$ @HerrK. That makes sense. I still think one should allow references for a mathematical claim, at least references with actual proofs. $\endgroup$ – Michael Greinecker Mar 24 at 21:10
  • $\begingroup$ I tried to combine your answer with the other one into a compromise FAQ consider having a look $\endgroup$ – 1muflon1 Mar 24 at 22:53
  • $\begingroup$ "Newspapers are allowed for facts only." will be abused to no end by trolls. $\endgroup$ – Kitsune Cavalry Mar 24 at 23:01
  • $\begingroup$ @MichaelGreinecker: I edited the grouping in the policy to reflect your suggestion. In any case, I think it's better to have overlaps than gaps in the coverage of citation requirements. $\endgroup$ – Herr K. Mar 25 at 1:02
  • $\begingroup$ @KitsuneCavalry: Possibly. But trolls will abuse whatever policy we put out, and we should not expect trolls to abide by it. Policies are intended to bring clarity to moderators and site users with whom we can reason. Moreover, sometimes reputable news outlets may be the only source of reliable factual information. A very quick search only led me to eg1 and eg2, but I suspect answers that rely on factual info can probably be found somewhere under the macroeconomics tag. $\endgroup$ – Herr K. Mar 25 at 1:18
  • $\begingroup$ You've made some good arguments for why "back it up" should include more than just strictly peer-reviewed articles. Thanks! $\endgroup$ – jmbejara Mar 25 at 16:56
2
$\begingroup$

The “Back It Up” Principle on Economics.SE

Policy Specifics

Policy Statement: Each answer of Economics.SE must be backed up in some way, or it is subject to deletion. Answers that are 3 days old and are still not backed up WILL BE DELETED. “Backing up” an answer depends on the type of question. Questions of a mathematical or computational nature must be backed up by mathematical argumentation or code. Questions about data or data sources must be backed up by links or references to the data or to a data codebook or data guide. All other types of questions must be backed up by references to credible academic sources. Examples of such sources include articles from a peer-reviewed journal or high-quality working paper series (e.g. NBER); peer-reviewed handbook chapters; or advanced-level, scholarly textbooks. References to newspapers or similar publications do not count. If an answer makes several significantly independent claims, each must be supported by references. Community members should flag posts that don’t meet these standards as “low-quality answers” to bring them to the attention of moderators.

Discussion of why the policy was formulated in this way

  • As discussed in the question, requiring deletion after the unsourced question is 3 days old makes the job of the moderator easier. If their hands are tied, they don’t have to worry about offering grace periods or offending. Community members should be able to alert an author of the deletion policy and 3 days should be enough time to revise. Also, authors will still be able to see their deleted posts, so they can always repost later if they are able to find a citation later.
  • I have chosen 3 days specifically because it is a rather short time period. If the time period is longer, then the answer will likely get lost down deep in the feed. It is easiest to delete while the answer is fresh, so that moderators don’t have to remind themselves to come back to it later.
  • Theoretically, the community will bring such posts to the attention of moderators. This means that a moderator can easily go through their queue of flags and quickly delete answers that don’t meet our standards.
  • Note that moderators may always choose to delete an answer that doesn’t meet these standards sooner than 3 days, subject to their judgement. This policy only requires the answer to be deleted after 3 days. However, this policy does not limit their ability to use their best judgement where appropriate.
  • For non-mathematical questions (and the like), this policy requires citations to credible academic sources. Newspapers and other such references do not count for the following reason, because such references are likely useless for the target audience of this site. This site is supposed to be a site for experts or those doing economics professionally. See this recent post. For a student, even a first-year graduate student and an advanced undergraduate, writing a research paper using references to newspaper articles is generally unacceptable. For graduate students, academics, and professionals, only peer-reviewed journals or high-quality working papers are useful. This also has the benefit of avoided discussions of bias among different news outlets. Technically, the one exception embedded in the policy is if the newspaper article is used as a citation for data. That is, the newspaper article contains a graph or figure that is used to back up the answer given.
  • Note that neither books nor Wikipedia should not generally count as peer-reviewed sources (unless the book is a reputable, scholarly textbook). Also note that high quality textbooks will include references to peer-reviewed articles. If you're using a textbook, this will make it easy to find the relevant peer-reviewed article to cite and it is encouraged that you provide the references to the underlying articles rather than the textbook itself.
  • The policy does not specify what happens to very old unsourced posts. My intention is that these will slowly resolve themselves over time. Either community members will slowly edit them over time or they will eventually be flagged by the community and will be deleted.
  • This policy complements a related policy for questions developed in another meta post regarding "expert-level questions".
  • The policy description that I have provided above is relatively short. It is one short paragraph that can be included in the help section of the site. It covers most of the cases that this site will encounter. Answers that don’t fit any of the cases will be left to the discretion of the community and the moderators. The underlying principle is the same, though. All answers must be “backed up.”
$\endgroup$
59
  • 3
    $\begingroup$ An excellent proposal, I was actually working on my own version of this yesterday but did not managed to finish it yet. I fully agree on the broad points. I further suggest that we learn from other sites off-topic on-topic statements and FAQ such as Physics.SE, Biology.SE or even from Skeptics.SE - Skeptics.SE have very good FAQ on citing, I actually already contacted Skeptics.SE moderators yesterday to ask for permission to copy some parts of that for our site since naturally here provision has to be made for mathematical answers, in addition, I think it has to be said publicly that $\endgroup$ – 1muflon1 Mar 20 at 9:22
  • 2
    $\begingroup$ PPS: here is link to physics.se help center: physics.stackexchange.com/help/on-topic. PPPS: I would also add provision not just for high quality working paper sites but also high quality books e.g. handbooks/textbooks. I dont see why citing Hamilton: Time Series Analysis or Romer Advanced Economics or Mostly Harmless Economics (books cited in many published papers should not be also included). Yes those books will include citations to papers directly but often they are more than just sum of the individual citations and they also offer knowledge in organized way $\endgroup$ – 1muflon1 Mar 20 at 9:26
  • 1
    $\begingroup$ @1muflon1 Thanks! I like what you describe about updating "off-topic" and "on-topic" statements and the FAQ. I'm looking forward to seeing that happen. Thanks for doing that! $\endgroup$ – jmbejara Mar 20 at 18:20
  • 2
    $\begingroup$ I agree with the proposal. Seems like a way to increase the quality of content on the site. $\endgroup$ – EconJohn Mar 21 at 21:31
  • 1
    $\begingroup$ Some arguments to not allow textbooks to count towards the requirement: (1) If the textbook is of high quality, you should be able to find the underlying peer-reviewed reference. (2) If you're writing a paper, textbooks references are not very helpful since peer-reviewed papers almost never cite textbooks. Answers on this site should be helpful for those trying to write quality research papers. (3) I'm afraid of people writing answers where they cite the "applications" section of their undergraduate econ textbooks. Such references don't meet the higher standard that I think we want. $\endgroup$ – jmbejara Mar 22 at 17:00
  • 2
    $\begingroup$ @1muflon1 If someone cites Hamilton or Mostly Harmless, my guess is that the answer will likely satisfy "back it up" by virtue of containing mathematical reasoning. E.g., an answer might discuss pitfalls associated with diff-in-diff identification without writing a bunch of equations. This would still be considered mathematical reasoning. An answer doesn't have to contain a full-on proof to be deemed "mathematical reasoning." The requirement that an answer contain a reference to a peer-reviewed article applies only to those answers that are purely discourse---answers that are just "talk." $\endgroup$ – jmbejara Mar 22 at 17:01
  • 2
    $\begingroup$ painstakingly going through old paper and retyping what is there for a quote especially where the information is not that different (and Romer book is citable). Next some knowledge was actually first published or properly organized in a book. Consider Friedman and Schwartz Monetary History of US, Keynes General Theory, Friedman Theory of Consumption Function. Yes in recent times books are bit sidelined and sometimes are just amalgamation of papers but there are exceptions like Woodford Interest and Prices or Gordon The Rise and Fall of American Growth, such books in my opinion might even have $\endgroup$ – 1muflon1 Mar 22 at 20:26
  • 1
    $\begingroup$ @jmbejara yes that was good post. I think we will make FAQ like they have on skeptics because not all pages in help center are editable by mods $\endgroup$ – 1muflon1 Mar 22 at 20:47
  • 1
    $\begingroup$ @1muflon1 Re: accessibility of articles for download. You mention that academics will have access but others will not. The counter-argument is that the ideal target audience for the site is academics, professionals, and graduate students. Enthusiasts probably won't care as much to actually look up the papers and read them. Even if they do, a motivated person can always find a way to download these papers for free somewhere. Often on the author's personal website. $\endgroup$ – jmbejara Mar 22 at 20:47
  • 1
    $\begingroup$ Yeah, it gets difficult at that point. What constitutes an independent claim? I haven't addressed this, but again this leads to the "unwritten and unspoken" aspects to a policy. There is some inevitable subjectivity here. $\endgroup$ – jmbejara Mar 22 at 21:15
  • 1
    $\begingroup$ @jmbejara well, I think you have me almost convinced but I still worry bit about users who might wanna abuse such unspoken policy. I think even if we go for your proposal fully we should at least provide some guidance on this in FAQ so the policy is not completely unspoken. I hope some other users will also share their opinion on this $\endgroup$ – 1muflon1 Mar 22 at 21:18
  • 1
    $\begingroup$ @1muflon1 Sounds good. Trolls are frustrating for sure. I personally don't see this aspect of policy being a prime target though, but you can never know. Let's see what others suggest. As always, it's been very nice to talk/work through these things with you!! $\endgroup$ – jmbejara Mar 22 at 21:32
  • 2
    $\begingroup$ References to handbook series (eg 1 and eg 2) can be particularly helpful and sometimes sufficient to answer some reference-request type questions. Moreover, the JEL regularly publishes book reviews and listing of new books. It would be a bit ironic not to acknowledge---as a matter of site policy---the value of books catalogued by none other than the Journal of Economic Literature. $\endgroup$ – Herr K. Mar 22 at 22:35
  • 2
    $\begingroup$ @1muflon1 I agree that we should tell users that theoretical answers need arguments and empirical claims evidence. If they lack this, they may be deleted. But the proposal is that any answer that does not mechanically satisfy the criteria needs to be deleted. I am primarily against such a mechanical deletion policy, moderation needs judgement and moderation. $\endgroup$ – Michael Greinecker Mar 22 at 23:20
  • 3
    $\begingroup$ @jmbejara I think "references to credible academic sources" should be enough and required for empirical claims. But not allowing books would throw out the baby with the bathwater. Truman Bewley's book "Why Wages Don't Fall during a Recession" has more than 2000 citations on Google Scholar. If we exclude books, we basically exclude many areas close to other fields such as economic history or history of economic thought. I think the risk that people cite case studies from undergraduate econ textbook is rather minor. $\endgroup$ – Michael Greinecker Mar 23 at 0:49
-1
$\begingroup$

I present here a number of features which I believe a policy on low-quality answers should have (it is not a fully worked-out policy).

1. The policy should have regard to the needs of all the groups in our strapline “For those who study, teach, research and apply economics and econometrics”.

The site does not currently attract enough experts and some degree of change to make it more useful to experts seems sensible. But change should not be at the price of making the site significantly less useful to others, or significantly devaluing their past efforts. Compromise between the needs of different groups is required. Those who would like a radically different kind of site tailored primarily to the needs of experts do have the option of setting up a new site, either within Stack Exchange or elsewhere.

2. The policy should include explicit guidance on the boundary between what should fall within the scope of the “back it up” principle and what can reasonably be taken to be common knowledge so does not need to be backed up.

This has been well addressed in the answer by 1muflon1. I would just add that what is common knowledge can be interpreted in two ways. One is what is common to all or most of the site community. Another is what is common to the sub-community of those likely to be interested in a particular question. Favouring the latter interpretation, I suggest that, for an expert question, the scope of what can be taken to be common knowledge is wide (but the quality of back-up for what isn’t common knowledge should be high). For a beginner question, conversely, the scope of common knowledge is narrower (but the quality of back-up for what isn’t need not be so high).

3. Where back-up is required, the type(s) of back-up that is/are appropriate will depend on the question and on the content of the answer.

The following quote is from this post by Monica Cellio on Meta SE:

Backing up an answer includes citing sources, making a logical argument, and sharing relevant personal experience.

With the caveat that personal experience will rarely be relevant on Economics SE, this is I suggest a good summary of the variety of forms which back-up may take. But not all forms of back-up are appropriate in all circumstances. A few examples:

a) Empirical claims (unless common knowledge) require citing sources. They could require logical argument as well (eg to draw a conclusion where a cited source just provides data).

b) Theoretical claims (unless common knowledge) require logical argument, but that could be by presenting an argument, or by citing a source containing an argument (though in the latter case it would be good practice to give a brief summary rather than a bare citation).

c) Textbooks and online teaching sites can usefully be cited in answers to elementary questions (and may well be more useful to questioners than peer-reviewed papers which for many can be difficult to access or understand). They will rarely be appropriate in answering expert questions (not so much because of doubts about reliability but because their contents, to experts, will probably be common knowledge).

d) Newspapers and online news sites will rarely be appropriate sources for citation (because they tend to misinterpret or sensationalize scientific findings), but could be useful in answering questions on specific events too recent to have been considered in other sources.

e) Occasionally, questions specifically ask for a particular type of back up. Such requests should be respected.

4. Where back-up is by logical argument, the policy should not privilege mathematics over other recognised forms of reasoning, eg verbal or diagram-based.

This is not in the least to criticise the use of mathematical reasoning in economics. I am merely asserting that each of these forms of reasoning can be used well or used not so well, and that different forms of reasoning may be appropriate to different questions or to different questioners (ie where their level of knowledge and understanding can be inferred from the wording of the question).

Answers which merely use verbal reasoning to draw conclusions from common knowledge assumptions (and therefore contain neither citations nor mathematical formulae) should be judged on their merits and not automatically assessed as low-quality. By judging on their merits, I mean assessing whether the assumptions really are common knowledge and whether there has been a reasonable attempt to present a logical argument rather than just make assertions.

5. Compliance with the policy should be encouraged and enforced primarily by the community, rather than by use of moderator powers.

I understand the attraction of using moderator powers to deal with a case such as recently arose in which a low-quality answer to this question received a large number of votes giving it more prominence than several other very good answers. Such an answer does not give a good impression of the site, and for a case such as that, I think deletion using moderator powers is appropriate (see 6 below).

However, the maxim “hard cases make bad law” is relevant here. Although low-quality answers are quite common, most of them do not gain such prominence. For many such answers, the voting system works well in ensuring that they are listed below better answers. I have two suggestions to try to make it work better still. One is to positively encourage downvoting of low-quality answers. I don’t share the view occasionally expressed that it is rude to downvote without giving the reason in a comment. Giving a reason would be helpful to a new user, but I don’t think it should be considered essential if a regular participant posts an answer that is clearly in breach of published policy. Those of us with a decent amount of rep shouldn’t be concerned by the small loss of rep (-1) that downvoting an answer entails.

My second suggestion starts from the likelihood that, when low-quality answers receive a lot of votes, some of those votes are by people who are not regular users of Economics SE but have seen a question on Hot Network Questions. We might explore (I think it would have to be raised with Stack Exchange management) the possibility of excluding our site from HNQs. This has already been done for several other sites (see here). The grounds for requesting exclusion would be that some economic questions are liable to attract attention from those with more interest in some social, political or other agenda than in economic science and that their votes can be harmful to the site.

6. Deletion by use of moderator powers should be limited to answers which are a) clearly offensive, or b) amount to non-answers, or c) contain contentious empirical claims that are BOTH not adequately backed up AND of a kind liable to attract attention from those with more interest in a social, political or other agenda than with economic science.

Category (c) is deliberately quite narrow. The fact that a claim is a matter of academic dispute does not in itself qualify. For example, Richard Carson and Jerry Hausman have published opposing views on the reliability of contingent valuation as a method of valuing environmental assets (see here). A post taking one side or other in that dispute in itself would I think be unlikely to attract the wrong sort of attention. But if someone were also to make a claim about the value of a particular well-known environmental asset (New York’s Central Park, say) then it might be judged likely to do so, and deletion (in the absence of adequate back-up) would I think be appropriate.

It is I suggest fairly straightforward to identify topics that might be liable to attract the wrong sort of attention. An initial list might be any economic questions involving reference to race, gender, environment and climate change, and international disputes.

Category (b) is helpfully explained in Herr K's answer here.

7. Deletion of an answer, whether by use of moderator powers or by the community via the low quality review queue, should normally be based on careful judgment. Only in the most blatant cases should it be an instant decision.

I am not persuaded by the argument that a policy should be simple. A simple policy will almost inevitably have unfortunate results in some cases. It is true that if many answers are to be deleted by use of moderator powers, a simple policy will place less burden on moderators than a more complex one, and I recognise that the burden on moderators, who act voluntarily for no tangible reward, is an important consideration. My proposal in 6 above, limiting the scope of deletion by use of moderator powers, would address that.

8. Any application of a new policy to old answers should be preceded by a substantial grace period, allowing time for people to consider editing their answers to comply with the policy.

Many people have put a lot of effort into answers over the years, in many cases trying hard to comply with policies applying at the time and to tailor answers to the particular needs of questioners. Some would probably be inclined to react strongly if deletion of many of their answers were to result, either suddenly or progressively over time, in significant loss of rep.

$\endgroup$
16
  • 2
    $\begingroup$ I think you make some good points about use of diagrams and other techniques that we should incorporate but I also have some criticism on some points. On point 1. Originally the Economics.SE site had subtitle "For professional and academic economists and analysts" implying much narrower scope - this was changed by meta discussion, no reason why now it could not be changed back. If back then those people who wanted wider scope were not told to set up their own new site why should now the people who would narrow the scope down do so? $\endgroup$ – 1muflon1 Mar 25 at 15:40
  • 1
    $\begingroup$ on the point 5 - 6 note that moderators not just community is quite heavily involved in deleting answers that do not follow some community standards on sci stacks. On physics even nonmainstream answers get deleted by mods (to make it clear I would oppose such policy on economics.se as I think heterodox answers/questions have their place here), so this would really not be huge expansion of power. In fact currently we are allowed to do so we just choose to self regulate ourselves. The policies above state objective criteria that can be followed I think by most people if they are just willing $\endgroup$ – 1muflon1 Mar 25 at 15:44
  • $\begingroup$ On the Monica Chello quote "Backing up an answer includes citing sources, making a logical argument, and sharing relevant personal experience." Monica is/was excellent mod but she used to mod sites where personal experience is more useful like Judaism or workplace etc. I dare to say that in science stacks like Physics.SE, Biology.se, Economics.SE personal experiences are not very helpful and scientifically not valid form of evidence. I can see them be helpful in soft-questions where the back it up policy could have exemption as I proposed $\endgroup$ – 1muflon1 Mar 25 at 15:48
  • 1
    $\begingroup$ In fact on the codidact I had some exchanges with monica on these topics for proposed site for economics there (see here) and she seems to agree that in order to create community the site should primarily be set up to attract experts - as Jmbejara often mentions experts attract wider audience not the other way around. 8 I fully agree with you, this is why I made provision for my proposal to apply the new policy to past answers only on case by case bases (meaning only to very bad answers, I would definetly not want to target excellent $\endgroup$ – 1muflon1 Mar 25 at 15:50
  • $\begingroup$ or not just excellent but even good past contributions) $\endgroup$ – 1muflon1 Mar 25 at 16:03
  • $\begingroup$ PS: I edited my original proposal. Now graphs are also included, and I removed the provision that would allow mods to delete very bad answers sooner than after 3 days (of having post notice), I think that this should be sufficient check on any potential moderator abuse - a) it is enough to add sources for the user if the answer is based on some legitimate arguments b) 3 days is enough for an answer for any regular to see it, and thus see if post notice was appropriate or not and then complain on meta for cases where it was misapplied $\endgroup$ – 1muflon1 Mar 25 at 16:28
  • 1
    $\begingroup$ @1muflon1 I think there are some nice things in this answer, but I don't like the part of this that puts limits on the ability of moderators to delete answers. It would be nice if the community could do all of policing, but the site doesn't have a critical mass to make this happen. I feel like this requirement would make it very, very difficult to enforce the new policy. I don't think we need to worry about moderator abuse. There are already other checks in place. $\endgroup$ – jmbejara Mar 25 at 16:38
  • $\begingroup$ @jmbejara I dont think it would be hard to enforce. It would not be perfect but I am personally not opposed to have some constraints. I think that anyone who would ever review my mod decisions would find that I always erred on the side of caution and acting too little rather than too much, but not all mods share this view, and even if I plan to mod for long time I wont be able to do that forever so I can see Adams concerns $\endgroup$ – 1muflon1 Mar 25 at 16:43
  • $\begingroup$ Also, IMO, graphical argumentation that appears in economics really is a form of mathematical argumentation. I don't think it needs to be spelled out. @1muflon1 I think your answer before the last edit (some 14 minutes ago) was great. $\endgroup$ – jmbejara Mar 25 at 16:43
  • $\begingroup$ @jmbejara thank you, I will consider rewriting it, you are right that graphical argumentation is a part of mathematical argumentation, but at the same time Adam (judging by most of his answers) is good economist but he somehow missed that as well I think some other people might miss it too so maybe just reminding users that it is part of mathematical argumentation is not so bad $\endgroup$ – 1muflon1 Mar 25 at 16:45
  • $\begingroup$ @1muflon1 I don't think it would work at all. Take the example of gender inequality question. From my reading, under this "community only" policy, you would not be able to delete any of those answers. The community member with high enough reputation can only delete an answer if it has a score of -1 or lower. Because of all the laypeople voting, there is no way that answer would get to -1, even if we had dozens of new experts on our site. $\endgroup$ – jmbejara Mar 25 at 16:46
  • $\begingroup$ @jmbejara no I disagree. 1. We would be able to delete the answer that was deleted by Kitsune (after 3 days - just not immidately) as that answer was just basked by opinion peace not directly by research. 2. next there are also couple of other answers that are not backed at all that could be deleted some already have post notices those would not pass this new policy as well. 3. Some things would probably fall through the cracks but no policy will be perfect there could always be abuse $\endgroup$ – 1muflon1 Mar 25 at 16:47
  • 1
    $\begingroup$ @1muflon1 Right. Your policy is different from Adam's. Your answer allows it to happen after 3 days. $\endgroup$ – jmbejara Mar 25 at 16:48
  • $\begingroup$ @1 muflon1 In response to your first comment. The change now proposed is significantly different from that in 2016. I could point to the fact that the 2016 proposal received more net upvotes (15) than have any of the current proposals - though there is time for that to change. More importantly, the site itself did not significantly change in 2016; rather, the subtitle was changed to reflect what the site at that time had already become. $\endgroup$ – Adam Bailey Mar 25 at 19:40
  • 1
    $\begingroup$ 1. but also in the past it seems meta was frequented by more users. All past meta posts got more upvotes then recent ones, last y and this y almost no post got more than 10 upvotes. 2. Note this proposal here does not change the scope of a site, my point 1 was just rebuttal to your point 1, but having a stricter policy on citation does not mean; a) we would disallow non-expert questions - this proposal is just for answers. B) disallow non-professionals from participation. It just follows the practices of other sci stacks and weeds out opinion answers that are discouraged on SE network anyway. $\endgroup$ – 1muflon1 Mar 26 at 9:11

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .