Wednesday, February 25, 2015

the rationality of heuristics

I recently joined a very interesting reading group and we're working our way through the new book "Evolution and Rationality: Decisions, Co-operation and Strategic Behaviour". The discussion about chapter 5 (by Brighton and Gigerenzer) was very thought-provoking. The chapter discusses the difference between "small world" and "large world" problems. The former are problems in which we are certain of the underlying processes, such as playing roulette. Large worlds are too complicated to be certain of the underlying probabilistic processes, perhaps too complicated to be certain of which processes are relevant at all, and the whole thing may not even be stationary. For example, playing the stock market.

The gist of the paper is that trying to model behavior in large worlds by deriving the optimal, rational thing to do, is misguided. This approach works well in small worlds but in large worlds it's highly likely that you'll specify the problem incorrectly. Heuristics can work better than complicated statistical bayesian reasoning. There's a great example of guessing which of two German cities is larger, based on a vector of attributes such as whether it has a university, whether it's on a river, whether it's located in the industrial belt, etc. In this case, a simple take-the-best heuristic, which looks only at the most relevant attribute that can distinguish between the two towns in question, outperforms an SVM.

This is a strong statement: not only can you understand actual choices better if you allow yourself to consider non-bayesian agents, you may understand what is actually optimal better.

I'll say that again a different way because I think it's that important: When we observe people behaving in a way that seems suboptimal, we should not infer that people are violating the rational/bayesian/vNM agent model. We should first question whether we truly understand the problem as well as we think we do.

This means that one very common response to psychologists' claims that people are non-(Bayesian/vNM/rational) by economists who are trying to rescue homo economicus, while clearly true in many cases, is sometimes not even necessary to resort to. In particular, heuristics are often seen to be rational because they are the optimal balance between mental calculation costs and accuracy. As the German city example proves, heuristics may in fact be a better approach to large world problems than a more sophisticated statistical analysis.*

Heuristics therefore definitely belong in the basket of reasons behind one of my favorite soapboxes: respect revealed preferences! Behavioral economics is too often seen as an excuse for all kinds of intervention in choices in order to "help" people optimize. But trying to do that is problematic for all kinds of reasons, including that it is very hard to prove that people are actually making mistakes. Observing demand for commitment devices is one of the rare cases where we can definitely say that restricting the choices of some people would make them better off. Such clear cases are few and far between.**

~~~

*My other objection to this frequent assertion (which I do believe is true in many cases, just not so many) is that critics of economics don't understand that most economic models are "as-if" models, and many economists have started to forget it, probably partially in response to all the negative press about classical economics that fixates on the implausibility that people actually make the calculations we model. But predicting the trajectory of a baseball is difficult to calculate, yet humans instinctively can do it very very well. Predicting the trajectory of a frisbee in gusty wind may not even be analytically tractable but somehow humans can do it reflexively. So why is it so hard to believe that humans are as good at optimizing their utility as they are at optimizing their frisbee catching? High mental calculation costs are not implied by analytically complicated problems.

**Not that situations in which people can be helped are rare, but situations in which we're sure there is room to help, and that by trying we won't make things worse, are rare.

Tuesday, February 17, 2015

retention in online education

I'm nowhere near a labor economist but in another life I would study education. Online education is particularly fascinating to me since I personally love it. I've done 8 or 10 coursera classes and started and abandoned about as many more. I love being able to learn about new subjects in a structured and multi-faceted* way again; it's like being able to take all the exploratory freshman college courses you want.

One of the first reasons people dismiss online education is something I wish wasn't a valid reason but I understand why it is. There's a major barrier to it being taken seriously as a credential, which means it's not going to be a substitute for traditional education anytime soon and will have trouble being financially viable.** But, as long as these credential-free courses exist despite lack of lucrativity***, I think that's great! They're purely about learning (what a concept!). The people in the course are self-selected based on genuine interest. The message boards consistently contain more fascinating and in-depth discussions than anything I've encountered in actual college classes. If I'm too busy one week, I'll skip a homework set, do it later at my leisure, my "grade" will suffer, but who cares.

I've lately heard a couple people dismiss online education for a reason I think is more bizarre, though.**** Retention rates. Yeah maybe it sounds bad when you excitedly advertise your course as having 700,000 signups but then have to later admit that only 10% of them completed it, but signups shouldn't be the metric in the first place. One of the great things about online education is that it's easy to peruse, test the waters, watch a few videos, try a couple weeks of homeworks, and then quit if it's not what you're looking for. Only one class that I've quit has been because I got too busy with other things and let it slide; the others were all because I made a conscious decision that it wasn't worth my time. Low retention rates (on average) is exactly what you want if what you care about is actual learning rather than credentials. If a particular class has a high retention rate that's a good sign about the quality of the course, but if retention rates are high across the board, people probably aren't exploring enough or are disproportionately motivated by the sheepskin.

~~~

*Structured meaning that each lecture builds on the last, allowing you to get to deeper truths vastly more easily than trying to piece together 30 isolated wikipedia articles. And multi-faceted meaning not just reading a book, but actually going through problem sets and taking quizzes and such. Highly important for information processing and retention.

**Coursera is falling over itself to change this; you can now often pay to get a "verified certificate of completion" instead of the standard free pdf download, or you can get a special certificate "with distinction" if you achieve a certain grade, etc. I have no idea if these are helping them make money yet but that's obviously the goal.

***Which I'm sure won't last forever; right now it's driven by a lot of excitement and curiosity and altruism but that won't sustain it once it's thoroughly clear that the current model won't make money. But hopefully the huge body of work that has already been done can be recycled indefinitely.

****Sort of like people dismissing charter schools because they're, on average, performing worse than traditional schools. That's exactly what should be expected!

Thursday, February 12, 2015

help, I have no idea who you are

This is just gonna be me whining that my life is sooo hard, so I suggest moving along :)

I'm really, really bad at recognizing people. I heard this (not that funny) joke at a comedy show once: "It's so weird how people always say they're bad with names. Like, opposed to what? How could you be good with names but bad with faces? Would you just walk around with a list of names in your head and no association to the people around you? Saying names randomly hoping the person you're looking for will hear? Haha that would be ridiculous."

Yes. That's exactly what it's like.

I first realized this as an undergrad when I tried to participate in an experiment involving facial expressions. I'm not exactly sure what they were testing (they don't usually tell you) but the experiment consisted of a training period in which you learned to easily distinguish four very similar faces (just the faces, no hair or anything). Then once you could tell them apart immediately and consistently, there was some other association test I don't even remember. Anyway, the whole thing usually took around 30-45 minutes. After an hour I still hadn't passed the training test. I came back for a second session, finally passed the training test after about another 45 minutes, and did the experiment.

I suspect I never noticed before that because I was too socially oblivious (until quite awhile after that, honestly) to notice awkward interactions that would clue me into something being wrong when I treated nonstrangers like strangers. And also, especially coming from small schools and a small town, you don't accumulate very many acquaintances until college in the first place.

But I'm definitely not faceblind: I also saw some internet talk about faceblindness once with a quiz for the audience in which people were supposed to keep track of how many celebrities they could recognize without hair and out of context. The point was that it's a lot harder than you expect. But I was awesome at it. Faces make a deep impression in my memory, but only after a long time and a lot of exposure.

With new self-awareness I've tried to compensate. I try really hard to remember features other than clothing (my subconscious default, apparently). Hair is salient and at least divides people into a few clearly demarcated bins, by color and/or length, but those are coarse divisions and there have been many failures in which someone I know reasonably well got a haircut or color and I didn't realize they were the same person for half a conversation or more. There are a few people I recognize by their glasses, but that's obviously dangerous since they change and people wear contacts. Height and build is much too coarse of a differentiator, and you really want to be able to recognize faces on their own anyway. A very small percentage of people have such distinctive faces that there's any detail I can fixate on and remember.

Failing much actual improvement in recognition, I use compensating devices. On the job market I think I spent just as much time studying people's photos as I did learning about their research. When going to conferences or other departments, etc, I try to look up what people look like who I know I should recognize (which mitigates approximately 1 in 8 embarrassing scenarios, but that's better than nothing.) I made face flashcards of everyone in my department before starting work, from which I easily learned names but they didn't help with faces one iota; people look too different in person than in photos. When meeting people in crowded places, I try not to make direct eye contact with anyone, maybe stare at my phone or elsewhere, to give them a chance to flag me down before I fail to find them.

These devices aren't very effective so I still run into a lot of problems. The motive for paying such close attention to faces on the job market was a previous job interview experience in which (so I deduced in hindsight after the awkwardness became so palpable that I clued in to my mistake) I introduced myself to the head interviewer on three separate occasions within a few hours. I also routinely look directly at someone I should know, smile or stare blankly and keep going, and then they awkwardly say hi Vera how's it going! while I desperately try to deduce what I can safely say to this mysterious person. I can't count how many times I've had entire conversations with people who know me that I'd swear I'd never seen before in my life, in which I mess up and say "I used to live in southern California" to a former Caltech classmate or something like that. Many other times I've agreed to meet someone in a few minutes in some other place and, immediately after turning away, realized I wasn't going to be able to find them. I'm inspired to write this blog post at this particular time after a two day conference in which I've failed to recognize so many people within the appropriate amount of time (even with nametags! for the love of god, wear your nametags, in a prominent position, and in the correct orientation!) that I'm well on my way to offending members of every major Australian economics department.

Help! What do I do? Does anyone have any secret tricks?

In the meantime, if you know me and I look through you without recognition, please don't take it personally. I surely remember our previous interactions well, you just don't happen to have conveniently recognizable green eyebrows...

Saturday, February 7, 2015

excessive formalism

Why on earth does this paper contain a formal model?

I love economic theory and can expound ad nauseum on the many reasons why it is valuable, but I'm having a hard time seeing how any of them apply in this case. The authors' points are more easily stated qualitatively, more easily explained qualitatively, and the formal specification doesn't produce any unexpected consequences or require anything more than intuitive qualitative statements to analyze. In fact I think their own argument could be more convincingly made if they were free to explore more nuanced aspects than will fit into the formalism. So why?

On the one hand I wonder if I'm missing something because the authors are (deservedly) well known and respected. On the other hand, I think John List et al's advocacy of field experimentation has gone beyond the very reasonable assertion that many types of questions are better suited to field tests to the assertion that field tests should be the only approach*. Is this an attempt to seduce others to this view with "math"**?

*I swear I'm not putting words in their mouths. I quote, from the same paper: "Another group feels that natural field experiments are more generalizable, and that in many settings, this benefit outweighs the drawback of having limited control, meaning that we should focus our scholarly energy on natural field experiments." That "meaning that..." clause doesn't follow, sorry...

**I'll remove the quotes when "proofs" contain more than a couple entirely intuitive qualitative statements.