I’m linking this post on g by Cosma Shalizi because I’ve only read it once, I haven’t understood it properly, and I don’t want to lose it.
I spent a while on Wednesday getting screwed by floating point errors in R. Since it took me a bit of sifting through search results on “r number doesn’t equal itself” to find the problem, this post is basically a quick shot at raising the visibility of the solution.
At risk of turning this blog into “the fMRI graph theory analysis papers” (which would probably attract more readers) here are a couple of better renderings and/or conceptions of the default-mode and task-positive networks. I’ve included only edges that represent significant correlations across subjects — the first as quantified by t-test on Fisher-transformed correlations, the second as quantified by Wilcoxon rank-sum test on raw correlations. I’ve also used a layout scheme that tries to capture the proximity between nodes.
The parametric and nonparametric edge definitions yield pretty much exactly the same organization, with DMN and task-positive networks highly intraconnected (is that a word?) but sparsely interconnected. Both approaches also capture an isolated subnetwork in bilateral parahippocampal cortex and accurately ostracize the cerebellar ROI, which isn’t actually part of the DMN or task-positive networks — it was supposed to be posterior cingulate,but I messed it up.
Another slightly subtler feature captured by both approaches is the particular inter-network edges, with positive connectivity between the DMN superior frontal ROIs (R/LSF) and task-positive dorsolateral PFC (R/LDLP) on the same side, and between the DMN “parietal” ROIs and task-positive ipsilateral loci in the intraparietal sulcus. Edited after I realized I posted the same image twice: The parietal connectivity is consistent across correlation metrics but the prefrontal connectivity isn’t. It looks like there’s some symmetry in the medial prefrontal and inferotemporal connectivity within the DMN as well; the medial prefrontal connectivity is also symmetric in the parametric graph but not so much in the nonparametric graph.
Still need to work on rendering edge weights. But these graphs are much nicer.
For the last couple of weeks I’ve been working not quite as hard as I should have on a graph-theoretic analysis of some resting-state fMRI data. Thanks to Brian Avants and ANTS, I’ve generated the following average connectome for default-mode and task-positive networks:
The key thing here is that connections between nodes of the same color are overwhelmingly red (positive correlations, significant across subjects) and connections between nodes of different colors are overwhelmingly blue (negative correlations). So the default-mode network and the task-positive network are correlated with themselves and anticorrelated with one another. This is not a shocking result (see link), but it’s fun to verify in my own data with new technology. There’s something attractive about a graph theory approach to functional connectivity that more sophisticated super-data-driven approaches like ICA just don’t have — maybe because people actually have some vague sense of how to think about and analyze graphs. (For “people,” you can probably substitute “Matt” with no particular loss of accuracy.)
Next: nonparametric approaches to edge analysis, visualization tweaks, and (most importantly) between-groups analysis of the effects of electrical brain stimulation on the connectome…
NPR says: Pediatricians Need To Help ‘Sad Dads’.
The report says that 41% of depressed dads spank their kids, whereas only 13% of non-depressed dads do so. That’s a gigantic difference, right? Think of how many spankings we could prevent if we could just make depressed dads spank at the rate of non-depressed dads. Right now a total of 15% of dads spank. So if we cured all the depressed dads, the rate would plummet to…
… wait for it…
Right. The report doesn’t tell you that only 7% of dads are depressed. You need to look at the abstract for that. So instead of contributing somewhat more than 2% of total spankings, as they currently do, spanking-normalized depressed dads would contribute just under 1%.
You could tell me that any spanking is a bad spanking, and I wouldn’t really be able to contradict you. I don’t have any idea how bad corporal punishment is. However, I do know that reading to kids can provide a giant boost to their intellectual development — and the study (but not the report) shows that depressed dads are much less likely than non-depressed dads to read to their kids regularly.
I’m no fan of spanking, but I’m a lot more worried about kids becoming dumb.
(Yeah, I’m aware of the irony of invoking THIS AMERICAN LIFE in an attack on the scientific foundations of an NPR article. Here are some scholarly articles on the topic if you want to chase them down.)
ETA: I posted a similar but somewhat better written analysis at Partial Objects.
Groundhog Day was not long ago, and although I wasn’t paying attention, a friend on Amherst’s online community was; and she leveled the quite reasonable criticism, “You know his track record is 34%? That’s LESS THAN RANDOM, people.” And the only way a blog post gets written about this is to ask the question, “But is it?”
Of course, a fair coin will give you 50% on any binary decision (here, “early spring” versus “long winter,” or ES vs. LW), so in a sense this is true. But, of course, to the extent that it is true, you’d be well served to guess the opposite of what Phil guesses — if he gets it right only 34% of the time, then you’ll get it right 76% of the time. And, indeed, Phil’s record of 39% accuracy (not 34%) over 115 years is nonrandom by binomial test (p=0.025).
Does that mean there’s something about Phil that somehow knows how the weather will be, even though he guesses wrong? (To be fair, we may be the ones guessing wrong — we may just be mistaken about how Phil’s attitude toward his shadow relates to the coming weather. It’s not like he says “Spring’s a-comin’!” in his squeaky little groundhoggy voice.) Not necessarily. Phil might be more like a weighted die than a fair coin — he might have a bias to guess one outcome or another. And in fact, he does, an enormous one: Out of 115 predictions, he’s only guessed an early spring 15 times, a mite’s hair over 13%. Of course, if early spring and long winter are equally probable, biased Phil should still have a 50% record; of the 50% early springs, he’ll get only 13% right, or 6.5% of the total, but of the 50% late winters, he’ll get 87% right, or 43.5% of the total, so his right answers add up to 50%. Only if nature has a bias opposite Phil’s — that is, a bias to actually emit an early spring — could a bias on Phil’s part cause him to do worse than 50% “by chance.”
So is that it? Is Phil just a biased coin? Well, consider this: If Phil is biased toward predicting long winters, but every time he predicts an early spring he gets it right, wouldn’t you view that as evidence of some kind of information in his signal? To concretize it, take a 100-year hypothetical —
LW + PHIL LW: 27
LW + PHIL ES: 0
ES + PHIL LW: 60
ES + PHIL ES: 13
So nature is biased toward early spring (73%), and Phil is biased toward long winter (87%), and his total success rate is a sad 40% — but when he does predict early spring, he’s uncannily accurate. Remember, when you get Phil’s guess, you don’t know what’s going to happen. So if he guesses long winter, you’re probably better off doing the opposite, since he’s only right about that 31% of the time (27/27+67). But if he guesses early spring, he’s always right, so you’d better guess what he guesses. This more detailed look at the “data” affords a better picture of what’s going on; there are cases in which Phil is very accurate and cases in which he’s worse than chance. (And, in case you care, a Fisher’s exact test confirms that he’s distributing his guesses differently when it’s winter vs. when it’s spring, p=0.018.)
However, this analysis doesn’t really save Phil as a prognosticator, only as a subject of some meteorological or perhaps neuroscientific interest — because, according to the analysis, you should guess early spring regardless of what Phil guesses. The bias trumps all.
Just as an appendix, here’s another fake dataset showing a pattern of guesses that has similar characteristics to the one above, but gives clear evidence that he’s just guessing at random:
LW + PHIL LW: 24
LW + PHIL ES: 3
ES + PHIL LW: 63
ES + PHIL ES: 10
Phil is still biased toward guessing long winter (87%), and nature is still biased toward early spring (73%), and Phil’s success rate is now 34%, but you can see that he distributes his guesses the same way regardless of the actual outcome — as quantified by a nonsignificant Fisher’s test (p=1). In this case, again, always guessing early spring earns you 73% successes, and betting against Phil only earns you 66%, so again the bias trumps all. (However, it wouldn’t take much tweaking to generate a dataset that was qualitatively similar, but in which betting against Phil was the better policy than betting on early spring.)
Via Charlie Stross: Fantasy author Jim Hines has posted data from a survey of professionally published novelists; Steven Saus has posted additional analyses. Data of note [with code abbreviations in brackets] include (a) number of short fiction sales [SFS], (b) number of rejections [R], and (c) years of writing [YW] prior to selling a first novel, as well as (d) age at publishing a first novel [A]. I admit grouchiness as impetus for the following analyses — I’m grateful for Jim’s work in collecting the data, but his scatterplot of year of publication of first novel vs. SFS is basically unreadable because of the one mutant^H^H^H^H^H^Hluminary who sold 400 short stories before selling a novel. According to scientific custom, the “narrative” below is idealized, i.e. mostly non-chronological but perhaps conceptually coherent.
I first excised two data points with incoherent values — one author claimed a zero value on YW and one a zero value on A. My next step was to plot histograms and descriptives of a few pedestrian data transformations (PDF link from Seth Roberts). I looked at square root, log2, and reciprocal transformations of the four variables enumerated above. (Log2 and reciprocal weren’t applicable to SFS and R, which had many zero values.)
In all cases, the square root transformation seemed to bring the data closest to normality — it reliably reduced skewness and usually reduced kurtosis. So I used square root-transformed data in all subsequent analyses. The next step was to eliminate outliers like the one that made Jim’s original year-SFS plot so hard to read. I excised six more rows, each of which had a value at least three standard deviations away from the mean of at least one variable. Four had unusual values on SFS and two on R. That left me with 238 values to work with.
At this point, I did a bunch of plotting and backed away from parametric tests (specifically Pearson correlation). Probably because the sales data are so unalterably skewed, a nonparametric test (Spearman correlation) was still more sensitive than the Pearson correlation on transformed data. In what follows, I really should be plotting the ranks, but it seemed more grungy than it was worth to figure out how to do that properly in R after midnight, so I’m still plotting square root data. However, the one significant rank-order correlation had a similar magnitude when I did the Pearson correlation on the square root data, where it wouldn’t have on the raw data, so the square root data isn’t such a bad visualization.
When I first looked at the year-SFS scatterplot, there seemed to be a strong positive relationship between year of publication and number of short fiction sales, even though the correlation coefficient was very small. I knew the mode of SFS was zero, so I figured there were probably a lot of overlapping points sitting on top of one another. So I figured I’d size each point such that its area was proportional to the amount of data residing there. Here’s my version of Jim’s plot:
You can see that the correlation is zero, although the number of writers making a lot of short fiction sales before their first novel does seem to be rising by the year.
Next, I wanted to see whether prior sales were related to how often an author’s first novel was rejected (maybe more experienced writers are less liable to suffer rejection) or how old they were when their first novel was published (maybe writing more short fiction delays — or accelerates? — selling your first novel):
There’s enough data that the incredibly low correlation coefficients from this incredibly skewed data are actually not so far from significant. (Perhaps most saliently, there are a few isolated cases of people selling several dozen stories and still getting their novel rejected several dozen times.) What about years of writing? You could imagine that the more short stories you sell, the less time you have to spend writing before you publish your novel (or, alternatively, the better a writer you are, the more short stories you sell AND the less time you have to spend writing before you publish your novel).
Although this correlation coefficient is highly significant and the rho value is much bigger, it’s still a pretty small effect — SFS and YW share a little less than 5% of their variance, whereas SFS and R or A share a little more than 3%. What’s notable is the direction. Since the data is just correlational, it’s not clear whether writing more stories causes you to spend longer in your writing career before you publish a novel, or whether taking a long time to sell a novel just gives you more opportunity to publish stories before you do — but in any case, it looks like writing short stories doesn’t accelerate the publication of your first novel. If anything, it slows it down.
Anyway. Not much here — mostly an incrementally more sophisticated confirmation of Jim’s observation of null results. I didn’t feel like dealing with the categorical data earlier in the day, and it’s late, so I won’t now. But it was a nice opportunity to play around with a dataset that’s different from the kind I usually run into, and to teach myself a few new things in R.