The Starlink app, whether on a mobile device, or in a web browser, will tell you in which direction the dish regularly finds something blocking its view of the satellites. I’ve had it in my head for a while that it should be able to do more than this. I think it should be able to give you a silhouette of any obstructions.

As a satellite passes through the sky above the dish, the “beam” connecting the two follows it, sweeping across the scene (Figure 0). The dish repeatedly pings the satellite as this happens, and records how many pings succeeded in each second. When the view is clear, all, or nearly all, pings succeed. When there’s something in the way all, or nearly all, pings fail. In theory, if the dish stays connected to the same satellite for the whole pass, we end up with a “scan line” N samples (= N seconds) long, that records a no-or-low ping drop rate when nothing is in the way, and a high-or-total ping drop rate when something is in the way.

One line isn’t going to paint much of a picture. But, the satellite is going to pass overhead every 91 to 108 minutes. The earth also rotates while this happens, so on the next pass, the satellite will be either lower in the western sky, or higher in the eastern sky. On that pass, we’ll get a scan of a different line.

But 91 minutes is a long time for the earth to rotate. That’s farther than one time zone’s width, nearly 23º of longitude. Since the beam is tight, we’ll have a wide band between the two scans in which we know nothing. However, each satellite shares an orbit with 20 or more other satellites. If they’re evenly spaced, that means the next satellite should start its pass only about 4-minutes after the previous one. That’s conveniently only about 1º of longitude. If the dish reconnects to the next satellite in an orbital reliably at a regular interval, we should get 20-ish scan lines before the first satellite comes around again.[1]

But are 1º longitude scanlines enough? Before we get into the math, let’s look at some data. I’ve created a few simple scripts to download, aggregate, and render the data that Starlink’s dish collects. With over 81 hours of data in hand – 293,183 samples – I can make Safari complain about how much memory my viewer is using … er, I mean I can poke around to see what Dishy sees.

In Figure 1, I’ve plotted ping drops attributed to obstructions at one second per 4×4-pixel rectangle. Solid red is 100% drop, and the lighter the shade the less was dropped, with white (or clear/black for those viewing with different backgrounds) being no drops. There are 600 samples, or 10 minutes, per line. It doesn’t look like much beyond noise, so let’s play around.

Figure 2 is the signal-to-noise ratio data instead. White/clear means signal was full (9), solid grey means signal was absent (0), with gradations in between. Still mostly noise, except for the obvious column effect. Those columns are 15 samples wide. So something happens every 15 seconds. It’s not clear what – it could just be an artifact of their sample recording strategy – but that’s as good of a place to start as any for a potential sync frequency.[2]

So let’s drop down to our guesstimated 4 minutes between satellite frequency. With 240 seconds per row (Figure 3) … mostly everything still looks like noise. Let’s start by guessing that the period between satellites is longer.

I clicked through one second increments for a quite a while, watching noise roll by. Then something started to coalesce. At 330 seconds (5.5 minutes) per row (Figure 4), I see two patterns. One is four wide, scattered, red stripes running from the upper right to the lower left. The other is many small red stripes crossing the wide stripes at right angles. Given that this persists over the whole time range, I don’t think it’s just me seeing form in randomness.

Advancing to 332 seconds per stripe (Figure 5) causes the small red stripes to pull together into small vertical stacks. Especially in the later data, some of these blobs seem to fill out quite a bit, encouraging me to see … something.

But here I’m fairly stuck. Doubling or halving the stripe size causes the blobs to reform into other blobs, as expected given their periodicity. But nothing pops out as obviously, “That’s a tree!” I experimented with viewing SNR data instead. It does “fill in” a bit more, but still doesn’t resolve into recognizable shapes.

It’s time to turn to math. I think there are two important questions:

1. How much sky is covered in a second? That is, what span does the width of a pixel cover?
2. How much sky is skipped between satellite passes? That is, how far apart should two pixels be vertically?

If I draw the situation to scale (Figure 6), with the diameter of the earth being 12742km, and the satellites being 340 to 1150km above that – giving them orbital diameters of 13082 to 13892km, there’s really not enough room to draw in my geometry! So I’ll have to zoom in.

We can start estimating how big our pixels are by comparing similar triangles. The satellites moving between 7.28 and 7.70 km/s. If we’re looking strait overhead, for our purposes at these relative distances (340 to 1150km), we can consider that 7km to be a straight line, even though it does have a very slight curve. In that case, we can just use scale the triangle formed by the line from us to the satellites T=0 position and the line from us to its T=1sec position, into our scene (Figure 7). If the scene objects are 20m (0.02km) away, then the width of one second at that object is `0.02km * 7.7km / 340km = 0.00045km`, or just under half a meter. Compared to the higher, slower orbit, it’s 0.00012km, or 12cm. At 12 to 45cm, we’re not going to see individual tree branches. Resolution will actually get a bit better when the satellite isn’t directly overhead, because it will be further away and so the perceived angle of change will be smaller. But for the moment, let’s assume we don’t do better than half that size.

On to estimating the distance between scan lines. Wikipedia states that there are 22 satellites per plane.[3] If these are evenly spaced around the orbit, we should see one every 4.14 to 4.91 minutes (248.18 to 294.55 seconds). If the earth rotates once every 23hr56m4s, then that’s 1.038º to 1.231º. At the equator, that’s 115.42 to 136.881km. I’m just above the 45th parallel, where the earth’s circumference is only 28337km, so the change in distance here is only 81.705km to 96.897km. If we change our frame of reference, and consider the satellite orbital to have moved instead of the earth, we can use the same math we did last time. To estimate, this distance (81km/satellite) is approximately one order of magnitude larger than the last ones (7km/s), so we can just multiply everything by ten. Thus, our scan lines should be 1.2m to 4.5m apart.

At 12 x 120cm per sample, we’re not going to be producing photographs. At 45 x 450cm, I doubt we’re going to recognize anything beyond, “Yes, there are things above the horizon in that direction.” Let’s see if anything at all compares.

What parameters should we use to generate our comparison scan? If we’re seeing satellites pass in 4.14 minute (91 minutes / 22 satellites) intervals, we should guess that a scan line will be about 248 seconds. If they’re passing every 4.91 minutes, we should guess about 295 seconds.[3] Given the aliasing that integer math will introduce, the fact that 4.14 and 4.91 are kind of the minimum and maximum, and that the satellites won’t sit at exactly those altitudes, it’s probably worth scanning from about 240sec to 300sec, to see what pops up. I see what look like interesting bands show up at 247, 252, 258, and 295 at least. Maybe I’m catching satellites at a band between the extremes?

But then why was 330-332 the sweet spot in our pre-math plot? Maybe I’m just indulging in numerology, but 330 = 22 * 15. Twenty-two is the number of satellites in an orbital, and 15 is the width of the columns we saw in the SNR plot. Could it be that satellites are not evenly spaced through 360º of an orbital, but are instead always 5.5 minutes (330 seconds) behind each other?[3] If that were the case, the orbital would “wrap” its tails past each other. That seems odd, because you’d end up with a relative “clump” of satellites in the overlap, so maybe there’s a better explanation for the coincidence.

In any case, I’m going to forge on with an example from the 332-sample stripe, because its blobs look the strongest of any to me. Let’s also redraw it with the boxes ten times as tall as they are wide, since that’s what I calculated to be the relationship between one satellite’s samples and the next satellite’s samples. If I overlay one of those clumps on the northward view I shared in my last post, does it line up at all?

I’ve stared at this for far too long now, and I have to say that this feels worse than the numerology I indulged in a moment ago. I’m starting to worry I’ve become the main character of the movie Pi, searching for patterns in the randomness. If there’s something here, it needs a lot more knowledge about satellite choice and position to make it work. Even if I adjusted the rendering to account for the correct curve of the satellite’s path and the camera’s perspective, the data is too rough to make it obvious where it lines up.

With some basic information like which satellite the dish was connected to for that sample, and the database of satellite positions, I’m pretty sure it would be possible to throw these rectangles into an augmented-reality scene. Would it be worth it? Probably not, except for the fun of doing it. The obstruction diagram in the Starlink app (Figure 9) divides the horizon into twelve segments. If it shows red in one 30º segment, it’s the tall thing you can see in that segment that is causing the obstruction. This additional data may be able to narrow within the segment, but if there are multiple tall things in that segment, they’re probably all obstructions.

So, while this was a fun experiment, this is probably where it stops for me. If you’d like to explore your own data, the code I used is in my starlink-ping-loss-viewer repo on github. The data used to to generate these visualizations is also available there, in the 1.0 release. Let me know if you find anything interesting!

… and just one more thing before I sign off. Following up on the topic of my past notes about short, frequent Starlink outages. Figure 10 is a rendering of my obstruction (red) and beta (blue) downtime over this data. I’ve limited rendering to only d=1 cases, where all pings were lost for the whole second, since this seems to be the metric that the Starlink app uses for labeling time down. One rectangle per second, 10 minutes per row. The top row begins in the early afternoon on February 9, and the bottom row ends just before midnight on February 12, US central time.

[1] Many thanks to u/softwaresaur, a moderator of the Starlink subreddit for pointing out that routing is far more complex, since active cells are covered by 2 to 6 planes of satellites, so it’s likely unrealistic to connect to several satellites in the same plane in a row.

[2] From the same source, routing information is planned on 15 second intervals. At the very least, this means that the antenna array likely finely readjusts its aim every 15 seconds, whether or not it changes the satellite it’s pointing at.

[3] Again from the same source, while 22 satellites per plane was the plan, 20 active satellites per plane was the reality, though this has now been adjusted to 18. That fits the cycle observation better, as 18 satellites at a 91-108 minute orbit is 5 to 6 minutes between satellites.

## Rural Internet: Starlink Outage Data

In my last post, I talked about how frequent, short outages prevent video calling from being comfortable on Starlink. If you were curious about exactly how short and how frequent I meant, this post is for you.

Starlink’s satellite dish exposes statistics that it keeps about its connection. The small “ping success” graphs I shared in the last post are visualizations provided by the Starlink app, which are driven by these statistics.

Thanks to starlink-grpc-tools assembled by sparky8512 and neurocis on Github, I have instructions and some scripts to extract and decode these statistics myself. I haven’t been great at collecting the data regularly, but I have six bundles of second-by-second stats, each covering 8-12 hours. (February 1 saw a couple of reboots, so the segments there are approximately 7.5 and 11 hours, instead of 12 for the other segments.)

The raw data exposes a per-second percentage of ping success. It’s somewhat common for a single ping’s reply to go missing. Several pings are sent per second, though, and one missing every once in a while is mostly no big deal. The script I’m using tallies the number of times /all/ of the pings within a given second went missing (percent lost = 100, or “d=1” in the data’s lingo). It also tracks “runs” of seconds where all of the pings in contiguous seconds went missing.

These first two graphs (Figure 1) explain what I mean by “frequent” and “short”. This histogram displays one bar per “run length” of all-pings-lost seconds. That is, the left-most bar tracks when all pings were lost for only one second, the next to the right bar tracks when all pings were lost for two consecutive seconds, the third bar tracks when all pings were lost for three consecutive seconds, and so on. The height of the bar represents the number of times an outage of that length was observed. The histogram is stacked, so that the outages on the morning of February 1 (green) begin where the outages on January 31 (blue) end.

Over the 66.5 hours for which I have data, we counted 739 1-second outages. That’s an average of just over eleven 1-second outages per hour, or just slightly more often than one every 6 minutes. The decay of this data is pretty nice: two second outages are approximately half as likely (344, averaging just over 5/hr, or just under every 12 min), three-second outages just a bit less than that, and so on. By the time we get to 8 seconds, we’re looking at only one per hour.

If we look at one 1s-8s outages, i.e. those that on average happen once per hour or more, we have a total of 2018. That’s an average of just over 30 disconnects per hour, or one every two minutes. For once, data proves the subjective experience correct. On a video call, it feels like you get something between a hiccup and a “they last thing I heard you say was…” every couple of minutes.

The right-hand graph is laid out in the same way, but the bars represent minute-long outages. You can just barely see a few counted as 1-minute and 2-minutes in length. Last Thursday, February 4 (red), was the first time we’ve had a significant Starlink outage, long enough for me to spend time poking around trying to figure out if it’s “just us or everyone.”

I’ve been mostly concerned with frequency – how often I can expect outages of each severity. The tool I’ve used to extract the statistics data exposes the outages differently. It is instead concerned with the total amount of downtime observed.

These graphs (Figure 2) are the data as the extraction tool provides it. Each bar represents outages of a certain length, as before. But now the height of the bar represents the total number of seconds of downtime they caused. The 1-second and 2-second bars are now about the same height because there were about half as many 2-second outages as 1-second outages, but they each lasted twice as long. The total amount of downtime they caused is about the same.

That giant red line that has appeared in the right hand graph is eye-catching. Thirty seven and a half minutes of downtime, caused by one 37-minute outage. That 1-minute outage stack is quite a bit taller too, accounting for ten minutes of total downtime itself. This is how the significant outage on Thursday appeared to us. There was a large chunk of time where we obviously had no connection to the internet (37 minutes), surrounded by quite a bit of time where we’d start getting something to download, but then it would stop (ten 1 and 2 minute outages).

The sum of all 1-second-or-longer downtime we experienced in this 66.5 hours of data is 14686 seconds, or just over 4 hours. That’s roughly 94% uptime.

We didn’t see the 37-minute outage in the earlier frequency graphs, because it has only happened once. If we zoom in on those graphs (Figure 3), so that most of the 1-13s bars are way off the chart, we can see a few more one-time-only outages. Each day has had some small hiccup in the “long tail” of over twenty seconds. I see hope in the fact that the grey color, which is the most recent data, from the day after the long outage, is nearly absent from the longer-run counts.

I’m curious about the sharp decline between 13 and 14 seconds. Is that a sweet spot for some fault recovery in Starlink’s system, or is it just an aberration in my data? I’ll have to keep collecting to see if it persists.

I’ve posted the summary data I used to generate these graphs in a gist on github.

At the end of my last post about the state of rural internet, I mentioned that we were about to try something new: Starlink by SpaceX. We’ve been using it as our primary internet connection for two weeks now, and TL;DR it would be tough to give it up, but it does have some limitations.

Download speed via Starlink is excellent. Samples I’ve taken via Speedtest.net over my wifi have never measured less than 30Mbps. Most samples are in the 60-80Mbps range. My highest measurement was 146Mbps. Upload speed via Starlink is also excellent. Speedtest measures them anywhere from 5 to 15Mbps. Ping latency bounces around a little bit, but is usually in the 40-50ms range.

Typical speeds I measured via fixed wireless were 20Mbps down, 3Mbps up. So Starlink, in beta, is already providing a pretty consistent 3-4x speed improvement. I no longer worry about downloading updates while trying to do anything else on the internet.

Unfortunately there is a “but”, because while the speed is great when it’s running, the connection drops for a second or five every few minutes. The dish’s statistics indicate that these interruptions are about half due to Starlink making updates (“beta downtime”) and half due to the trees blocking my dish’s view of the sky (“obstructions”). I’ll be working on the latter when the weather warms, and they’re constantly working on the former.

These short interruptions have almost no effect on browsing or streaming. Every once in a while, a page will pause loading for a moment, or a video will re-buffer very early on. I notice it only slightly more frequently than I remember cable internet hiccups.

But what these short interruptions do affect is video calling. Zoom, Facetime, etc. are frustrating. It /almost/ works. For two, three, five minutes everything is smooth, but then sound and video stop for five to ten seconds, and you have to figure out what the last thing everyone heard or said was. My wife participated in a virtual conference this past week, and she tried Starlink each morning, but switched back to fixed wireless after the second or third mid-presentation hiccup each day.

And yet, there’s also a silver lining to the outage story. One of our frustrations with our fixed wireless provider is that we’ve had several multi-hour outages over the last three months. On Thursday, we finally had a two-hour Starlink outage. Why is that a silver lining? When I loaded Starlink’s support page over my cellphone’s limited 4G connection (remember, my wife was video conferencing on fixed-wireless), they had a notice up that they knew about the outage in our area, and listed an expected time of resolution. That sort of communication is something we have never gotten from our fixed-wireless provider. It completely changes how I respond to an outage, and it gives me hope that Starlink better understands what people expect from internet service today.

If you’re curious whether data backs up my subjective review of Starlink connectivity, please continue to my next post, which includes the dish’s own statistics.

The comparative price of the two solutions is nearly a wash. Starlink hardware is \$500 plus shipping and handling (another \$50). Our fixed wireless installation was \$119, with the option to either buy the antenna for an additional \$199, or rent for \$7/mo. That makes Starlink at least \$200 more expensive up-front, without including any additional mounting considerations (brackets, tower, conduit, etc.). And don’t get me wrong, white the setup seemed simple to me, the value of professional installation and local, in-person troubleshooting should not be overlooked.

But once everything is in place, the monthly costs are the same: \$99. For fixed wireless, that gets me 25Mbps that handles video calls well, but goes out overnight. Starlink is currently a no-guarantees beta, marketed as “better than nothing” for people who can’t get even my alternatives. Even in this state, it’s providing 4x more speed for me, with better communication about downtime. I think they’ll have no trouble selling these to loads of people, and if they significantly improve the video-calling experience, they’ll put fixed-wireless out of business.