confidence equals knowledge; or, the subtleties of survey design

The Christian Science Monitor publicizes a Pew Forum survey of religious knowledge in the US. First, let me say that I scored 31 out of 32, crushing even my atheistic soul-brethren and leaving the mouth-breathers of Christianity and America in the dust:

On average, Americans got 16 of the 32 questions correct. Atheists and agnostics got an average of 20.9 correct answers. Jews (20.5) and Mormons (20.3). Protestants got 16 correct answers on average, while Catholics got 14.7 questions right.

Second, let me hazard a guess as to why.

The CSM’s quiz is a pretty good reflection of the Pew Forum’s methodology (418K PDF download), which is to say that the questions take substantially the same form. That form is mostly multiple choice, with most questions having four answers and a few six, and one of the options in the multiple choice being “I don’t know.” Now, the Pew Forum is not the Educational Testing Service; they do not penalize you a quarter point for each incorrect answer. Which forces you to decide: How do you answer when you don’t know?

Actually, it doesn’t even force you to decide; I didn’t even think about this issue as I was taking the test — but there was no case in which I wrote “I don’t know” in my handy little emacs file. I took my best guess, which was in at least one case a random guess, each time. I did this because I am hypereducated: When I see a multiple-choice test, my unquestioned goal is to maximize my score. The Pew people report that the strongest predictor of correct answers is “educational attainment.” Remember: Each degree comes with its own standardized test, and you practice more as the stakes get higher. (I practiced about half an hour for the SAT; the general GRE took dainty chunks out of my life for a couple of months, and the psychology GRE took gulps for the better part of a year. Christ only knows what people do for degrees where you make money.)

“That only matters if people are really professing ignorance.” They are. I’m not going to run a histogram, but that Pew PDF linked above puts the number at, depending on the question, somewhere between 4 (“Where, according to the Bible, was Jesus born?”) and 52 (“In which religion are Vishnu and Shiva central figures?”) percent. 52 is not a huge outlier — there are many other questions in the thirties and forties. Look for yourself. Lots of people are punking out when they don’t know, or think they don’t know.

“Fine, but how much could guessing help your score?” Don’t sell yourself long, stud. Let’s work an example: Americans are reported as knowing 16 of the 32 items. If you got a score of 16 without guessing — i.e. all your errors are from professing ignorance — you could have gotten 4-5 more questions right by guessing (16 questions, each with about three answers that aren’t “I don’t know,” fudged down for the three free response questions). Conversely, if you got a score of 16 with guessing, then at least 5-6 of your correct answers were due to guessing. These are back-of-the-frontal-lobe numbers, and it’s certainly fair to fudge them down a little further for the free response questions, but they’re in the ballpark. And that ballpark is about the size of the 4-to-5-question difference between Christians/Americans and Jews/Mormons/atheists.

Speaking for myself, it would be grossly unfair to say [spoiler alert] that I “knew” anything about Jonathan Edwards, Maimonides, or the Supreme Court’s attitudes on comparative religion courses and the Bible as Literature; and if the question on the first four books of the New Testament hadn’t had that helpful hint “(the four Gospels)”, I’d have merrily recited “Genesis, Exodus, Leviticus, Numbers” (for which you can thank Paul Epply-Schmidt’s eighth-grade unit on INHERIT THE WIND). Someone who knew as much as I know but felt less confident in it, or just had a non-Type-A attitude toward surveys coincidentally framed like the SAT, might have scored substantially lower.

Having said all this, I am not a survey methodologist; there may be overpowering reasons to include a “Don’t know” option in surveys (maybe people are stressed out by guessing and drop out, maybe a dozen other things). I can’t claim excessive expertise here. But the impact of guessing is bigger than you think, and a bit more deserves to be made of the fact that people’s attitudes toward guessing are allowed to vary uncontrolled. Perhaps my fellow Americans are not such mouth-breathers after all — they just don’t want to say a thing if they’re not sure.

One thought on “confidence equals knowledge; or, the subtleties of survey design

  1. Pingback: Tweets that mention confidence equals knowledge; or, the subtleties of survey design « the pulchrifex papers -- Topsy.com

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s