Monday, September 28, 2020

JoEP's excellent editorial on standards and practices

I was so thrilled to read this editorial in the Journal of Economic Psychology. I think they've hit the right balance on every point except possibly 14, despite having to balance disparate traditions from psychology and economics. Here's the text distilled into a list:

  1. Quality replication studies are encouraged and JoEP has a section dedicated to them (which is slowly increasing in usage.)
  2. They have added a "brief reports" section to the journal "to speed up the pace of research by conveying short, concise points without the need to artificially inflate the discussion to meet some arbitrary length standard."
  3. Null results are welcomed, especially as replication/brief report papers, but need to be accompanied by properly conducted power analysis.
  4. Pre-registration will not be considered a pro for publication. The details and justification for this point are fantastic so you should read section 1.2 in its entirety.
  5. Multiple studies in a single paper are also not considered a pro for publication; scientific logic dictates how many studies are appropriate.
  6. Making data and relevant code available will be enforced going forward, except in exceptional circumstances.
  7. At the same time, "data free-riding" is discouraged.
  8. "Authors need to disclose any use of deception and the necessity for its usage will be evaluated."
  9. "In studies of economic decision making incentivization is the standard practice, and this is the standard we adhere to."
  10. While in an ideal world we would abolish significance categories altogether, for now p<.05 is a one star effect, p<.01 is two stars, and p<.001 is three stars. Given that p values are themselves random variables, three categories is about the right level of precision for reporting them.
  11. Effect sizes should be presented with the highest possible clarity, for example by standardizing regression coefficients.
  12. "Multiple comparison corrections or strategies of avoiding multiple comparisons should be used", noting that the necessity of multiple comparisons depends on the specificity of the hypothesis being tested.
  13. "We ask authors to state in writing that they have reported all implemented experimental conditions ... and disclosed all measured variables; as well as how their sample size was originally determined. We also plan to add additional guidelines concerning indiscriminate removal of participants from the sample."
  14. Submissions at JoEP are not anymore anonymous.

I hope that none of these are particularly controversial, even though standard practice most certainly does not reflect them and they are inconsistently enforced even when required. 

I've highlighted 4 and 13 because 4 is certainly the most controversial, and 13 is relevant to that controversy. In fact it's the key to the solution. Those who object to mandatory pre-registration complain that it places an undue burden on the researcher and restricts the natural exploratory research process. Those who favor pre-registration counter that the pre-registration doesn't bind you to do exactly what the documents lays out; it simply requires you to explain the deviation or update the pre-registration. And to that I say: if you honestly don't want to hamper the exploratory research process, then it should be sufficient to adhere to 13. If all designs tried and refined are described, if all measured variables are reported, etc, then a careful reviewer can easily detect fishy statistical practices including all of the practices mentioned in the editorial as motivating the pre-registration movement. And it's much easier to detect these practices if this information is concisely included in the paper itself rather than having to dig through a separate pre-registration document(s); I guarantee this digging simply won't happen. Proposals to require these kinds of statements have been around since long before the pre-registration craze and I wish there were more than a handful of referees and editors out there sporadically enforcing the practice.

I'll skip commenting on the rest and refer you to the editorial instead, since it does such a good job. Except, on point 14. It's not that I even disagree with the decision, given the reality of the situation. I'm just frustrated that this seems to be an important issue that journals have collectively thrown up their hands over due to the reality of the situation. There must be other improvements that can be made. E.g. authors should be able to optionally remove all identifying information from their submissions, temporarily remove drafts from the internet where possible, and reviewers can be asked to promise not to look up the authors or papers online until after their reviews are submitted; of course this is unenforceable but would go along way towards establishing a norm. This is only 30 seconds of brainstorming; surely the profession as a whole can come up with better solutions than all-out surrender.

And to those who insist that author identity and affiliation don't actually affect the results of peer review, I invite you to conduct the following study that would be able to actually provide RCT evidence: Authors from top institutions and/or with particular notoriety in their field partner with authors from lower-ranked institutions and/or lesser name recognition who work in the same field. Journals are asked for permission prior to the study to, at some point in the future, submit genuine research papers with only the author and institution falsified, on the condition that the author and institution will be corrected, regardless of outcome, when the paper is either rejected or accepted. Any papers written by involved authors who want to submit to one of the involved journals will be assessed by the authors themselves and perhaps by a third party(ies) as plausibly submitted by another author from the opposite category. If so, it is submitted randomly under one or the other name. In the case of multiple-author papers, the lead author would be the single name on the submission until the review process is concluded, at which point it will be corrected to include all others. Of course that's only the basic gist, but you get the idea.

This would be much more logistically feasible to arrange by a senior faculty with sway at a wide range of journals, so how about one of you put your money where your mouth is? :) Even if the study is failure due to reluctance from journals or from researchers, that's an interesting finding of mismatch between stated and revealed beliefs...

Thursday, July 30, 2020

economics is a disgrace

I can't (yet) bring myself to do the multi-tweet reply thing, so I will link to this instead.

If you are an academic economist and somehow haven't yet read Claudia Sahm's post, you are obligated to do so now.

I admit I have been dismissive of complaints about the culture in the past, and I want to explain why and how I have come to recognize the problems, in case it helps anyone else bridge the understanding gap as well.

Most complaints about the culture of economics are about treatment of women and minorities specifically. These accusations invite defensiveness, because they are easy (in isolation!) to re-interpret as bad behavior that has nothing to do with gender or race. I have reacted that way myself because I personally have not experienced anything I would call sexist, and have seen plenty of effort to help women and minorities. In the absence of real statistics about how advisors treat minorities versus non-minorities, about the disparity in feedback those groups receive (properly controlling for other factors of course!) it's easy to write off bad behavior as unrelated to identity. Perhaps I should be more trusting of personal accounts that are interpreted as racially/gender motivated, but I'm naturally skeptical. And I will probably get in trouble for saying this (maybe this will be the 1000th blog post I don't publish out of fear of getting in trouble for attempting to discuss something sensitive...) but I honestly don't think that skepticism is a bad thing. Better evidence would go a long way not just to convince me of the problem, but to understand the problem better and make effective changes. Skepticism that demands such evidence is, I believe, a good thing.

However, over the last year or two I've realized that I'm very oblivious to the effect of my gender on my interactions. I'm obviously a woman but that fact is entirely irrelevant to my identity; I don't notice when I'm the only woman in the room, I don't hear comments about women as applying to me, and it always catches me by surprise when my gender is explicitly referred to as a factor (usually 'we need you for this committee because there has to be at least one woman.') It never occurs to me that particular interactions might be influenced by my gender, so I don't experience them as sexist. But the deluge of accounts that have, by the participants, been interpreted as sexist, prompted me to reconsider my interpretation. That has changed my prior belief about the likelihood that accounts of bad behavior are, in fact, due to gender/race/etc.

More importantly, however, I also got a little bit outside of the economics bubble and saw just how terrible our culture is by comparison. Regardless of whether any of our cultural problems are solely or particularly problematic for women and minorities, our culture is nasty, overly competitive*, overly cliquish**, ludicrously hypnotized by the power of the Top 5 journals to cast the ultimate judgement of the quality of research (rather than the quality of research that happens to fall into the narrow fads that are currently deemed to represent 'general interest'), dismissive, elitist, rude, and negative.

My work lies at the border of social psychology and behavioral economics, and UQ is fortunate to have an amazing social psychology group right next door to the School of Economics. I therefore started attending, at least when the topic veered close enough to my interests, the social psychology group's weekly seminars. I can't even express how different they feel to an economics seminar. Instead of dourly entering the judgement room where we will listen to a presenter and attempt to score points by identifying faults in their research, their seminars are friendly, positive, constructive, understanding, encouraging, and just a huge breath of fresh air by comparison.

Economists go to work with an attitude problem. The toxic culture is what results. If it's blatantly obvious in seminars, it has to be much more detrimental in every other context***. It certainly disproportionately affects women and minorities, but regardless, it hurts everyone.

Lastly, I want to point out that while Claudia's post is excellent and brave, the full extent of the problem will not be recognized without more comprehensive data, and that will not be forthcoming as long as people in a position to report incidents fear retaliation. We need a way to anonymously share our accounts (experienced or witnessed). That isn't EJMR...

~~~~~

* Sure, competition that leads to better research is great. Competition for the sake of winning is not. A very well-respected economist at a "top 5 school" once told me that if anyone ever criticizes your work, the only way to respond is to immediately double down and go on the offensive. The sadder thing is that, within the culture, that's probably true.

** It's amazing how much easier it is to get an email reply from economists you don't know when your email signature is from a "top 5 school", as mine was in grad school, than when it's from a lesser-known institution. Sure, I understand signaling, but if your judgment of potential value of of interaction is based so much on the signal sent by institutional affiliation, you are being immorally lazy. Or if your judgment of potential value of interaction is based on "how likely is it that this person will be in a position of power over me in the future", well... if that isn't a sign of a culture problem, I don't know what is.

*** This is why our publication process is so godawful compared to other fields. Refereeing/editing should not be a game in which to score points or an avenue to promote personal interests!

Monday, March 9, 2020

so wrong, in every way

Bloomberg spent $500 million on his campaign. Multiple people on TV actually said or agreed with the statement that he could have instead given $1 million to every American. This is so unfathomably wrong, it's hard to describe. It's true that you don't need a calculator to figure out that it's wrong, but that's being generous - you barely need any common sense to know this is wrong. Here are some things that are as wrong (yes, I calculated):
  1. "I'm Native American, because my mom is Native American. Well not my mom, but my great-great-great-great-great-great-great-great-great-great-great-great-great-great-great-great-great-great-great-grandmother."
  2. "We live in Manhattan, so I told my son he has to stay on this block if he rides his tricycle. I found him in Argentina. After he had already triked around the entire planet 7 times."
  3. "This recipe called for 1 tsp of salt, but I accidentally put all of the salt for sale in all of the Wal-marts in Texas in instead."
  4. "I thought this was a dinosaur bone, but actually it's my grandpa's baby tooth."
  5. "I'm prepping for a 10-year quarantine in case coronavirus gets really bad. I've got all my food supplies ready: one almond should do the trick."
  6. "I told you to meet me on the stairs of the Met museum, and you thought Perth was close enough?"
  7. "The richest man in the world could give everyone on the planet $18. An average guy in Somalia could quit working for 24,000 years with that $18."
  8. "Bloomberg spent $500 million on his campaign. That's the entire GDP of the USA for 20 years." 

Now imagine... If a professional journalist can be this wrong, how accurate do you think your political facebook memes are?