Sleep is Weird
We all do it. And we spend a lot of time doing it. If we were to follow the guideline of 8 hours of sleep a night, we would spend a third of our entire lives asleep.
It’s such a strange nightly ritual: turning off the lights, lying down, closing our eyes, putting in earplugs — oh, you don’t do that part? — before, hopefully, slipping into unconsciousness. We then wake up the next morning and just go about our day, business as usual.
There is no doubt sleep is weird, but it’s also incredibly important. I won’t pretend to be a sleep expert and try to articulate the importance of sleep, but I can recommend two resources.
I think Matthew Walker does a great job of explaining the topic in words that a layman can understand in his book “Why We Sleep: The New Science of Dreams and Sleep”.
For a more academically rigorous source while writing this post, I consulted the “Sleep Physiology” chapter from “Sleep Disorders and Sleep Deprivation: An Unmet Public Health Problem” available here.
Tracking my Sleep
I started tracking my sleep late last year. I was tired of feeling tired, and I needed some data to talk about my sleep, or lack thereof, in a more intelligent manner.
So I went and bought an Apple Watch and started wearing it to sleep.
If you haven’t seen it before, this is what a night of sleep looks like on Apple’s native Health app.
This type of graph, plotting sleep stages over time, is called a hypnogram. Apple classifies sleep into three stages: REM, Core, and Deep. Core and Deep are the labels given to the different levels of non-REM (NREM) sleep.
You can find more details about how Apple estimates and classifies sleep stages here.
Getting my Sleep Data
The Apple Health hypnograms are cool, but I wanted to see the data for myself — it’s my own personal data after all.
There is no native desktop application for Apple Health and there is no way to easily see the raw data on the phone or tablet.
Fortunately, Apple does provide an option to “Export All Health Data”.
Using some standard Python libraries and pandas, I converted the exported XML file into my tabular format of choice, a DataFrame. This format will let me easily manipulate and view the data.
This image was AI generated. No real pandas were harmed in the making of this image or blog post. pandas is a Python library widely used for data analysis.
So now I had my Apple Health data in a DataFrame, ripe for wrangling, manipulation, and analysis. The subset of data I needed was conveniently labeled “Sleep Analysis”. I did some basic filtering and was off to the races.
The screenshot of the Apple hypnogram I showed earlier is actually made up of 28 data points, each having a value indicating what stage of sleep was logged (Awake, REM, Core, or Deep), as well as the start and end time of that logged stage.
Creating my own Hypnogram
With my data in hand, the first thing I wanted to do was create my own hypnogram. So I did just that. Here is my hypnogram for the same night of sleep shown earlier.
A little rough around the edges, but it’ll do. I noticed my plot shows a few moments of me being Awake, that don’t appear in Apple’s hypnogram.
Apple treats 30 second blips as noise and doesn’t display them. I will deal with the noise in my data in my own way, further along this post.
My Average Night of Sleep
Looking over a span of 7 months of data, I’m getting an average of 7.5 hours of sleep. I think measuring my sleep led me to prioritize getting better sleep, because an average of 7.5 hours is more than I would have guessed.
Here is my typical night of sleep shown as a donut chart, the hip cousin of the pie chart.
On average, I am getting around 100 minutes of REM sleep, but only an hour of Deep sleep.
My time in Deep sleep seems a bit low, but Apple themselves state that when validating their model, they found that “the most common misclassification overall was true Deep sleep being classified as Core sleep.” I’ll try not to lose sleep over my perceived low amount of Deep sleep.
The average amount of sleep a human adult needs is said to be between 7 and 9 hours. Considering I am still tired with an average of 7.5 hours of sleep, I might be someone that needs closer to 8 or 9 hours of sleep. There is no doubt that diet and exercise are also important factors.
Sleep Cycles
While reading about sleep, the concept of sleep cycles really caught my attention.
In a given night of sleep, we cycle through periods of NREM sleep and REM sleep multiple times, in roughly 90 minutes intervals.
I actually had first heard about 90 minute sleep cycles back in high school, when I first tried to “hack” my sleep.
If I was sleeping at 1am and needed to be up by 6am, I would set my alarm for 5:30am, aiming to get 4.5 hours or 3 cycles of sleep. The hope was that my alarm would go off at the end of a sleep cycle, making it easier to wake up refreshed.
It’s clear that I didn’t get anywhere near enough sleep in my teenage years and that I also misunderstood sleep cycles.
90 minutes is just a benchmark: an average, not a golden rule. The length of a cycle ranges from around 70 to 120 minutes depending on the person, and also changes over the course of the night for the same person.
So now, a little over a decade later, armed with some more knowledge and actual data, I wanted to better understand and see my own sleep cycles.
I wrote a function that would identify sleep cycles based on sleep stage data. There were a few underlying assumptions but the basic logic that I used was the following:
- A period of REM sleep following a period of NREM sleep would mark the end of a cycle.
- A significant duration of wakefulness would end the current cycle, and would exist outside of any sleep cycle.
- The last period of sleep wouldn’t count as a full cycle unless it were longer than 60 minutes, as it’s likely my alarm disrupted the ongoing sleep cycle.
Here is the output of my sleep cycle function overlayed on top of the hypnogram for the earlier night of sleep.
The cycle borders are indicated by vertical gray dash-dot lines. Let’s talk about what we see. Don’t mind the mess at the end of the night, as we’ll cover that later.
The first identified sleep cycle is 160minutes long. That is a bit longer than I would expect based on my understanding of our sleep architecture. I also note that I re-enter Deep sleep after a significant period of Core sleep, which is a bit suspicious for a single cycle.
Upon seeing this, I hypothesized that I may have had a brief period of REM sleep somewhere in the second block of Core sleep that my Apple Watch didn’t pick up on.
While not as bad as with Deep sleep, Apple does report misclassifying REM sleep as Core sleep. I also know that the amount of REM in the first sleep cycle can be as short as 1 to 5 minutes, providing just a short window to identify it, making this misclassification explanation fairly plausible.
For a second, let’s just assume that my hypothesis was true, and that there was a period of REM sleep misclassified as Core sleep. I modified the raw data by injecting 5 minutes of REM sleep into the second block of Core sleep to visualize what that would look like.
We now see 4 clear cycles of NREM and REM sleep. Each cycle is between 70 and 120 minutes, with the proportion of Deep sleep decreasing and REM sleep increasing with each subsequent cycle. This is aligned with my understanding of our sleep architecture.
But what’s that mess going on after the 4th cycle? Later on in the night, my sleep appears more fragmented, quickly moving between NREM and REM sleep. Each of these quick cycles is getting identified as a full new sleep cycle. Let’s smoothen the noisy data a bit.
Smoothing my Sleep Data
As I noted about Apple’s hypnogram, they don’t show 30 second blips of wakefulness. I wanted to take it a step further by merging short periods of sleep with the previously logged sleep stage. I did this assuming that the observed rapid cycling is due to noise as opposed to actually cycling every few minutes.
Okay that looks a lot cleaner. I’m tempted to make the last bit of Core sleep into REM sleep for a nice big block of REM in the 5th cycle, but I will contain myself.
Now, let’s look at the hypnogram for the next night.
We see that the first two cycles are practically the same lengths as in the earlier night of sleep (76mins vs 75 mins and 84 mins vs 88mins), but for this night, I didn’t inject any REM sleep. This would look like it supports my hypothesis about the missing REM sleep in the first example.
But now let me put a wrench in that. Here is the hypnogram for the night after that.
We’re back to a 160 minute long first cycle. Was there an undetected period of REM sleep somewhere here? Or was my first cycle actually that long? In this night, unlike the first night, it’s not clear where the theoretical missing REM stage would fit.
And yes, I actually fell asleep at exactly the same time, 12:58am, on these two consecutive nights!
Conclusion and Next Steps
Well there you have it. My first blog post. Ending on a cliff hanger.
There’s so much more I could do with this data, but unfortunately I only have a finite amount of time, and I didn’t create this blog to exclusively talk about sleep.
With this post, I had set out to take a look at the sleep data from my Apple Watch and lightly dip my toes back into the world of data science. I think I succeeded in that goal.
I’ll continue to passively collect my sleep data. I imagine I’ll write a follow up post on this topic in the future. I’d like to make my smoothing and cycle identification functions smarter. It would also be cool to create an app that can directly access the sleep data from Apple Health.
Thank you for reading until the end!