one of the best academic paper titles
for those who don't speak academia: "according to our MRI machine, dead fish can recognise human emotions. this suggests we probably should look at the results of our MRI machine a bit more carefully"
I hope everyone realises how incredibly important this dead fish study is. This was SO fucking important.
I still don’t understand
So basically, in the psych and social science fields, researchers would (I don't know if they still do this, I've been out of science for awhile) sling around MRIs like microbiolosts sling around metagenomic analyses. MRIs can measure a lot but people would use them to measure 'activity' in the brain which is like... it's basically the machine doing a fuckload of statistics on brain images of your blood vessels while you do or think about stuff. So you throw a dude in the machine and take a scan, then give him a piece of chocolate cake and throw him back in and the pleasure centres light up. Bam! Eating chocolate makes you happy, proven with MRI! Simple!
These tests get used for all kinds of stuff, and they get used by a lot of people who don't actually know what they're doing, how to interpret the data, or whether there's any real link between what they're measuring and what they're claiming. It's why you see shit going around like "men think of women as objects because when they look at a woman, the same part of their brain is active as when they look at a tool!" and "if you play Mozart for your baby for twenty minutes then their imagination improves, we imaged the brain to prove it!" and "we found where God is in the brain! Christians have more brain activity in this region than atheists!"
There are numerous problems with this kind of science, but the most pressing issue is the validity of the scans themselves. As I said, there's a fair bit of stats to turn an MRI image into 'brain activity', and then you do even more stats on that to get your results. Bennett et. al.'s work ran one of these sorts of experiments, with one difference -- they used a dead salmon instead of living human subjects. And they got positive results. The same sort of experiment, the same methodology, the same results that people were bandying about as positive results. According to the methodology in common use, dead salmon can distinguish human facial expressions. Meaning one of two things:
- Dead salmon can recognise human facial expressions. OR
- Everyone else's results are garbage also, none of you have data for any of this junk.
I cannot overstate just how many papers were completely fucking destroyed by this experiment. Entire careers of particularly lazy scientists were built on these sorts of experiments. A decent chunk of modern experimental neuropsychology was resting on it. Which shows that science is like everything else -- the best advances are motivated by spite.
Ok so this is...not exactly accurate.
They ran the fMRI testing on the dead salmon to test the MRI machine before doing an experiment looking at humans' response to social stimuli. They wanted something "with good contrast, but also with several clearly defined and distinguishable types of tissue..." (1), so they got a fresh salmon from the grocery store. Having tested the machine, they laughed at the absurdity, set aside the results, and did their regular study. Years later, one of the authors was doing a seminar on analyzing fMRI data and used the salmon data they already had as a goofy example of improper analysis, and they found something interesting!
So when you do fMRI imaging, you get a lot of data. Like, 40-130k voxels (small cubes of your brain, like a 3D version of a pixel on a screen) showing changes in blood oxygen levels (which does seem to be a good indicator of mental activity, based on research like optogenetics looking at whether blood oxygen levels indicate brain activity*). (2, 3) From all over the brain, because the rest of your brain is still active and doing different things while the subject is answering your question or looking at the image you're showing them or doing whatever task you've asked them to do.
So now you have this fuckton of data, what now? Well, now you look at the information you got from each of the voxels from the duration of the scan, and you compare the data you got from each voxel at different times to see which ones are "activated," where those voxels are (are they clustered together or spread out randomly?), and what physical activity or stimuli it's correlated to (i.e. which question was asked or image was shown or task was done).
Figuring out what qualifies as a significant change in blood oxygen levels for each voxel is done using "a fair bit of stats" because it's a shit load of data, and picking out significant changes can come down to a fraction of a percent change. So now you're doing thousands of comparisons between voxels, and you can run into a well known issue called the "multiple comparisons problem," which basically means that if you run a lot of tests, some of them will come out positive even if they're not (i.e. false positives). They really fuck up research and are a problem when you're trying to figure out, say, "new MRI protocols to use with adolescents and adults" (4), so you have to correct for that when doing the "fair bit of stats" to make sure you aren't reporting on the false positives.
What "The Dead Salmon" study found was that among all the false positives (which is to be expected, as "just about any volume with 65,000 voxels is going to have some false positives with uncorrected statistics" (4)) there were 3 activated voxels that just happened to be clustered together right in the middle of the salmon's brain, which gave the appearance of brain activity in a dead fish because they hadn't used the proper corrections in their statistical model.
That's what led to it being an actual story instead of a fun anecdote, and it's why they submitted it to the conference. It was all to show support that the minority of researchers at the time who reported their statistics from fMRIs without correcting them to should stop doing that, but then neuroscience blogs discovered the poster and wrote about it, which led to it going viral (before the authors could publish their commentary on it) and being awarded an Ig Nobel Prize in 2012.
Ultimately, it "doesn’t add anything to the technical discussion of how multiple comparisons correction is performed, it is simply a salient reminder of why proper correction is always necessary" (5), and it didn't invalidate that many previous studies or the entire field of neuropsychology (although it going viral did seem to have the desired effect, since by the time they got the Ig Nobel Prize 3 years after the fact only 10% of papers in the field of fMRI reported no multiple comparisons correction where before it was closer to 25-40%). It provided a goofy example for why it's important that researchers do the proper statistics to analyze fMRI data that has been widely used since the 90s and which most of them were already doing. (6)
Since tumblr hates reblogs with links I will reblog again with sources and additional information.
*The difficulty with using brain imaging to make any kind of claim about the way humans think or act or interact with others is more in how the data is interpreted, since brains are complex and several parts of the brain can be involved in a single thought or action, and a whole host of other things that can complicate it that would be an entirely different post. Typically tho, the "men think of women as objects because the same part of the brain is activated by women as by tools" type of reporting comes from journalists who are trying to interpret scientific papers that might say something more like "images of women, tools, and elephants elicited activity in the same areas of the brain in men...this may indicate a commonality between these images, but determining what that commonality might be is beyond the scope of the current study and further research is needed," the journalists just picked out the part that would be most sensational so people would read their story.
Fun facts:
The original Dead Salmon Study was originally a poster at the Human Brain Mapping Conference in 2009, which was later written into a paper and published in the Journal of Serendipitous and Unexpected Results
Also, it doesn't appear that the activated voxel cluster in the dead salmon necessarily correlated with any significant part of the experiment, which they ran all the way through as if it were a human subject with the fish because they were also training their research assistants.
Sources:
(3) National Library of Medicine, Optogenetic fMRI sheds light on the neural basis of the BOLD signal
Additional reading:
Social Cognitive and Affective Neuroscience, The principled control of false positives in neuroimaging (a paper written by the original Dead Salmon Study authors on the same topic)
Prefrontal.org, Human Brain Mapping 2009 - Presentations (links to what Dr. Bennett presented at the 2009 conference along with the Dead Salmon Study poster)
Nature (scientific journal), Functional neuroimaging as a catalyst for integrated neuroscience (paper published in November 2023 talking about fMRI as a tool for measuring brain activity, its strengths/weaknesses, studies that have successfully used fMRI in different subdomains of neuroscience, and how to use it to integrate these subdomains to promote interdisciplinary cooperation--specifically mentions criticisms of fMRI and cites the Dead Salmon Study as an example of the need for caution and advances that have been made in the 15 years since that study was published)
Additional fun fact! They baked the Atlantic salmon and ate it for dinner, so were not reimbursed for it. In the blog post The Internet Found the Atlantic Salmon that Dr. Bennett made at the time, he also lists some of the funniest comments he'd seen about the whole thing, and they are delightful: