Alexey “Matthew Walker’s ‘Why We Sleep’ Is Riddled with Scientific and Factual Errors” Guzey writes:
I [Guzey] recently finished my 14-day sleep deprivation self experiment and I ended up analyzing the data I have only in the standard p Here’s the experiment.
One concern that I have is that my Psychomotor Vigilance Task data (as an example) is just not very good (which I note explicitly in the post), and I would be worried that if I try doing any fancy analysis on it, people would be led to believe that the data is more trustworthy than it really is, based on the fancy methods (when in reality it’s garbage in garbage out type of a situation).
Here’s the background (from the linked post):
I [Guzey] slept 4 hours a night for 14 days and didn’t find any effects on cognition (assessed via Psychomotor Vigilance Task, a custom first-person shooter scenario, and SAT). I’m a 22-year-old male and normally I sleep 7-8 hours. . . .
I did not measure my sleepiness. However, for the entire duration of the experiment I had to resist regular urges to sleep . . . This sleep schedule was extremely difficult to maintain.
Lack of effect on cognitive ability is surprising and may reflect true lack of cognitive impairment, my desire to demonstrate lack of cognitive impairment due to chronic sleep deprivation and lack of blinding biasing the measurements, lack of statistical power, and/or other factors.
I believe that this experiment provides strong evidence that I experienced no major cognitive impairment as a result of sleeping 4 hours per day for 12-14 days and that it provides weak suggestive evidence that there was no cognitive impairment at all.
I [Guzey] plan to follow this experiment up with an acute sleep deprivation experiment (75 hours without sleep) and longer partial sleep deprivation experiments (4 hours of sleep per day for (potentially) 30 and more days). . . .
His main finding is a null effect, in comparison with Van Dongen et al., 2003, who reported large and consistent declines in performance after sleep deprivation.
My quick answer to Guzey’s question (“I’d be curious if you think the data I have is worth analyzing in some more sophisticated manner”) is, No, I don’t think any fancy statistical analysis is needed here. Not given the data we see here. An essentially null effect is an essentially null effect, no matter how you look at it. Looking forward, yes, I think a multilevel Bayesian approach as described here and here) would make sense. One reason I say this is because I noticed this bit of confusion from Guzey’s description:
The more hypotheses I have, the more samples I need to collect for each hypothesis, in order to maintain the same false positive probability (https://en.wikipedia.org/wiki/Multiple_comparisons_problem). This is a n=1 study and I’m barely collecting enough samples to measure medium-to-large effects and will spend 10 hours performing PVT. I’m not in a position to test many hypotheses at once.
This is misguided. The goal should be to learn, not to test hypotheses, and the false positive probability has nothing to do with anything relevant. It would arise if your plan were to perform a bunch of hypothesis tests and then record the minimum p-value, but it would make no sense to do this, as p-values are super-noisy.
Guzey has a whole bunch of this alpha-level test stuff, and I can see why he’d do this, because that’s what it says to do in some textbooks and online tutorials, and it seems like a rigorous thing to do, but this sort of hypothesis testing is not actually rigorous, it’s just a way to add noise to your data.
Anyway, none of this is really an issue here because he’s sharing his raw data. That’s really all the preregistration you need. For his next study, I recommend that Guzey just preregister exactly what measurements to take, then commit to posting the data and making some graphs.
There’s not much to say about the data analysis because Guzey’s data don’t show much. It could be, though, that as Guzey says he’s particularly motivated to perform well so he can find that sleep deprivation isn’t so bad.
Why do we go short on sleep and why do we care?
God is in every leaf of every tree.
As is so often the case, we can think better about this problem by thinking harder about the details and losing a layer or two of abstraction. In this case, the abstraction we can lose is the idea of “the effect of sleep deprivation on performance.”
To unpack “the effect of sleep deprivation on performance,” we have to ask: What sleep deprivation? What performance?
There are lots of reasons for sleep deprivation. For example, maybe you work 2 jobs, or maybe you’re up all night caring for a child or some other family member, or maybe you have some medical condition so you keep waking up in the middle of the night, or maybe you stay up all night sometimes to finish your homework.
Similarly, there are different performances you might care about. If you’re short on sleep because you’re working 2 jobs, maybe you don’t want to crash your car driving home one morning. Or maybe you’re operating heavy machinery and would like to avoid cutting your arm off. Or, if you’re staying up all night for work, maybe you want to do a good job on that assignment.
Given all this, it’s hard for me to make sense of general claims about the impact, or lack of impact, of lack of sleep on performance. I have the same concerns about measuring cognitive ability, as ability depends a lot on motivation.
These concerns are not unique to Guzey’s experiment; they also arise in other research, such as the cited paper by Van Dongen et al.