Dennis Haysbert, who played President David Palmer on 24, says that customers who switch from Geico to Allstate save an extra several hundred dollars.
Let’s grant his claim. Quick, true or false: Customers who switch from Allstate to Geico also save an extra several hundred dollars?
I have no idea, which is the point: The answer isn’t “false.” However, I can tell you one thing: Customers who switch from Allstate to Geico definitely save money. How do I come by this oracular certainty?
Because otherwise, they wouldn’t switch.
Most of you get where this is going, but if you’re not following, let’s do a thought experiment. Let’s pretend Geico and Allstate assign premiums at random with the same mean and variability — on average, customers are assigned the same premium, but any given customer will get two different premiums on one or the other side of that. Let’s say you’re a Geico customer. If your randomly assigned premium at Allstate is lower, you’ll switch, confirming President Allstate’s claim. If it’s higher, you won’t switch — which doesn’t have any effect on his claim at all. And if you’re a Geico customer considering an offer from Allstate, the only way you can affect his claim is by switching, against rationality, when the premium is higher.
I’m banging on this rather anodyne point because it’s actually astonishing how vacuous the claim is when you give it just a little thought. The only thing you’ve learned from President Allstate’s endorsement is that people who switch to Allstate from Geico usually don’t end up paying more money. Which amounts to saying that, on average, they’re rational.
What’s interesting to me about this claim is that it’s an instance of the non-independence error — selecting a subset of cases based on whether they show an effect, then calculating the effect size in those cases. This is, of course, a badly biased estimator of the effect size in the population; colleagues of mine have used it (not in published papers) to classify what word a subject was viewing based on the hemodynamic signal of a carefully chosen subset of the surrounding air (note that this sort of stunt is not quite in the spirit of identifying neural correlates of perspective-taking in dead fish, whose point is multiple comparisons correction, not non-independence.) Anyway, this is a problem that’s recently been made much of in the realms of neuroimaging and single-unit neurophysiology; Ed Vul’s web page provides links to many of the signal articles on the topic, most of which I haven’t read. So it was a bit of a shock to see it in a TV commercial. And it makes me wonder how much decision-making is happening on the basis of too-cleverly chosen samples.
Just as an illustration, here are a few things President Allstate could have said that I’d have viewed as better endorsements of his company:
“Geico customers switching to Allstate save more than Allstate customers switching to Geico.” (This at least implies that the expected value of switching one way is greater than the expected value of saving the other way, although there are some obvious scenarios in which that’s irrelevant to most people — maybe a few rare people save a lot and most people don’t save much. Also, if neither population saves much, maybe you don’t care that Allstate has the advantage.)
“Geico customers switching to Allstate are more likely to save money than Allstate customers switching to Geico.” (Again, this implies a genuine advantage, although not necessarily at a scale that you care about — imagine it’s 2% versus 1%.)
“Geico customers are more likely than not to save money by switching to Allstate.” (This is what he actually wants to say. Why didn’t he say it? It’s not just because the amount of money saved is quantifiable — so is the likelihood of saving.)