the practical significance of statistical significance

That’s a leading headline, so let’s be clear: It’s not zero. But the question is, can a difference that’s significant at p<0.05 support a lede like this?

THE GIST The rich don’t get how the other half lives.

The answer, of course, is “it depends what you’re measuring.” So what’s being measured here?

The NYT article, by Pamela Paul, is a summary of a recent paper in PSYCHOLOGICAL SCIENCE by Michael Kraus, Stephane Côté, and Dacher Keltner. The full text of the paper is available to the general public on Kraus’s Web site, so you can follow along if you want. This seems like as good a place as any for the usual caveat: I’m a cognitive neuroscientist, not a social psychologist, so my reaction is a product of, and functions on the level of, fairly broad-spectrum instincts and not specific expertise in the psychology of poverty or empathy.

As Paul mentions in the NYT article, Kraus et al. conducted three experiments. The first examined the relationship between educational attainment (used as a proxy for social class) and an emotional intelligence test that requires subjects to identify emotional expressions in faces. The second examined the relationship between subjective (i.e. self-reported) socioeconomic status and accuracy at judging the emotional expressions of an interlocutor. The third examined the relationship between a manipulation of perceived SES (instructions to think about oneself compared to the best-off or worst-off people in the USA) and a measure of empathic accuracy similar but not identical to the one in the first experiment. In each case, the experimenters found that subjects with lower SES, either actual or perceived, had higher empathic accuracy relative to those with higher SES.

So far, so good. Let’s look at the data.

HS-educated people have higher empathic accuracy than college-educated people.

People made to feel lower-class have higher empathic accuracy than people made to feel higher-class.

Don’t look at the SEMs. They say what the authors say they say. Look at the scales.

The empathic accuracy scale used in Experiment 1 is normed to a population mean of 100 and standard deviation of 15. They report each population’s mean in the text. The high-school-educated subjects had a mean empathic accuracy of 106.02; the college-educated subjects had a mean empathic accuracy of 99.40. This is a real difference; there are lots of subjects, so the estimate of the means is pretty good. But we’re talking about a difference of 7.62% on a test with 20 items, which I’d eyeball as about an item and a half. It’s also notable, by the way, that the college-educated subjects are not lower than the normed mean, they’re indistinguishable from it. It’s the high-school-educated students who are higher than the mean. There are any number of reasons for this, most obviously that their measure of empathic accuracy may have been normed on college students. What’s striking to me is that this feature of the data isn’t mentioned at all.

As you might have guessed, a similar analysis on Experiment 3 yields a similar result. The measure of empathic accuracy in Experiment 3 has 36 items; participants with lower manipulated social class got on average 27.08 items right, while those with higher manipulated social class got 25.23 right. This is a difference of 5.14% accuracy, even smaller than the difference in Experiment 1.

Please note: This does not mean the work is uninteresting, insignificant, or wrong. There are other features that make it significant, notably the authors’ claim that this is the first experiment to manipulate participants’ perception of their own social class. And, as far as I can tell, the differences are real. I wouldn’t believe them until they’re replicated (Jonah Lehrer has a very nice article on the decline effect that’s worth reading for anyone concerned about scientific epistemology), but that doesn’t mean the paper wasn’t worth publishing.

But it does mean that it is pure bullshit to say that this paper is evidence that “The rich don’t get how the other half lives.” It’s evidence that the rich are slightly worse at perceiving other people’s emotions — whether those others are rich or poor. This isn’t inconsequential; it’s new knowledge. But in terms of revising your own policies, that “slightly” is critical. Note that, in other contexts, very large differences between populations are not viewed as good grounds for treating individuals differently.

So far I’ve mostly criticized the popular interpretation of the science, not the science itself. Where I do take issue with the authors is not in the text of their paper, but in Paul’s NYT article. There, we find the following:

“Upper-class people, in spite of all their advantages, suffer empathy deficits,” Dr. Keltner said. “And there are enormous consequences.”

Keltner may be right, but let’s be clear: This article is evidence for empathy deficits, not “enormous consequences.” His sense that there are “enormous consequences” is what led him to do the experiment. Contrast his claim in the NYT to what he says on this topic in PSYCHOLOGICAL SCIENCE (emphasis mine):

Empathic accuracy may mediate influences of class on relationship quality, commitment, and satisfaction. It is also interesting to speculate about the costs of heightened empathic accuracy for overall health and wellbeing, particularly because lower-class individuals tend to experience chronically elevated levels of negative emotion and negative mood disorders (e.g., Gallo & Matthews, 2003).

Maybe Keltner was quoted out of context. He might have other evidence for the claim he made in the NYT. But if he did, why didn’t he mention that evidence in the paper?

Anyway, this has been a rehearsal of fourth-grade science: Keep your eye on the y-axis. My experience teaching neuroscience at Princeton would suggest that this is harder than it sounds, but it pays off.

(Also, as a matter of random derision: While Paul is right that the paper doesn’t identify the university that provided the subjects, it’s almost certainly Berkeley — Kraus only left Berkeley for UCSF a couple of months ago, and the subjects in Experiments 2 and 3 are 50% Asian, so it’s not Toronto.)

Thanks to Adrian Arroyo for forwarding me the NYT article, and to Liz Fuller for the Jonah Lehrer article on the decline effect.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s