One reason I haven’t worked on fixing my obstructions before now is that winter makes the ground impenetrable, and the roofs treacherous. The second reason is that I didn’t want to attempt any solutions without having data to guide me.
“What data do you need? Just get the dish up as high as you can!” is the sentiment I’ve gotten from the Starlink subreddit. Forty-foot Rohn towers are the “put a bird on it” of that community. But the idea of just ordering $1000 or more of tower, and pouring a large cement pad for it, just to see if that fixes things, doesn’t sit right with me. What if all I needed was 20ft of tower at a different location? What if I actually needed something taller than 40ft?
Don’t even get me started on the, “Just cut down the trees,” crowd.
So once it got warm enough to be outside without thick mittens for more than a minute at a time, I did what any engineer would do: I hacked together a sextant, and mapped the positions and heights of my obstructions – trees, mostly.
Then, because my particular engineering specialty is computers, I created a 3D model of the property, and put Dishy in it.
A cone with a 100º peak, rotated northward until its edge is 25º of of horizontal represents Dishy’s field-of-view. I was lazy about tree modeling – they’re just cylinders rising to the measured height, at the correct location. I assume that if I can get the cone above the peaks, it will also be outside of the rest of the tree shape. Comparing this model to the obstruction view in the Starlink app, I think I got close. Each tree I see inside the app’s window, I see inside the model’s cone.
With hope that the model is correct, I started moving the Dishy cone around. My first question was whether I should plan to put Dishy on my roof. Unfortunately, there is a tree very close to the north side of my house, not to mention the east and west sides. The cone wasn’t clear until Dishy was ten feet above my roof. To confirm, when the ice had cleared, I climbed on the roof and pointed the app around. The view from the app made me think this might even be optimistic, as some branches of the tree reach over the roof, in a way that makes the simple tree-cylinder model too simplistic.
If the roof mount was going to require a tower anyway, how much tower would I need from the ground? I began moving Dishy’s cone up from its original position. When I reached twenty feet, moving it a little south and a little west seemed to almost entirely clear the cone. If I raise the aim or narrow the field at all, as is expected to happen when more satellites come online later this year, the cone is completely clear.
So I should have just ordered a 20ft Rohn and put it there, right? Maybe, but, I’m not quite that confident in my mapping. My hacked sextant gave very coarse readings, made worse by the fact that many readings were over 45º, where sine grows faster than cosine. My compass liked to swing 5-10º between eye level and ground level. I think the model is a decent start, but I’m not willing to risk the permanent installation of a steel tower on it yet.
But I do like building things, and even though lumber prices are higher than usual, it’s manageable for small projects. How close to 20ft can I get?
Pretty close, it turns out! A tripod made of 12ft 4x4s, with 7ft of a 10ft pipe sticking out the top, plus Dishy’s 16in stem comes to almost exactly 20ft. If something like this will work, it offers a few nice features:
The pole can be raised and lowered to make installing Dishy easier.
The pole or the legs can be extended, if just a little more height is required.
If ground anchors are enough to keep it from tipping, it’s possible to reposition the tower.
That last point is half of an important question: is this design strong enough? Luckily, radio operators have been mounting antennas on pipes for decades, so the engineering isn’t hard to find.
Dishy’s stem has a 1.5in. outer diameter. Schedule 40 1.5in. steel pipe has a 1.5in. inner diameter, so mounting Dishy on such a pipe wouldn’t even require the backordered adapter. Is it strong enough? I chose what seemed like the worst case scenario: Dishy’s flat face pointed straight into an 80mph wind. Our winds generally come from the west (parallel-ish to Dishy’s face), and average high gusts are 40mph, so this should give us a good margin of error. Luckily, even in the worst case I’ve specified, math says that 8ft. of 1.5in. S40 shouldn’t bend. Hooray!
What about those ground anchors? If I use the wind force already calculated, an calculate the leverage at the pivot point at ground level, I get almost 25,000in.lbs. I found ground anchors that say they provide 2250lbs. of holding power “in normal soil conditions.” Given that they will be 4.5ft from the pivot point, that comes out to over 120,000in.lbs. of leverage. That’s so much more than the wind leverage that even if my soil conditions are abnormal, I think I’m safe. (Yes, I have ignored wind torque on the wood tower as well. Given that it’s much lower to the ground, and the anchor torque is so much higher, I’m not concerned.)
If my well pump hadn’t failed just a couple of hours into construction, I might have had the tower up in two days. It took three. The tower was stable enough for me to lean my ladder against while mounting Dishy. We’ve only had light breezes so far, but Dishy doesn’t seem to wiggle too much.
So, the did-I-save-over-a-Rohn question: did I fix my obstructions? Almost. We had rainy days following installation, so obstructions have bounced around a bit. In the days before relocation, I was seeing over 35 minutes of obstruction per 12 hr period. Since relocation, I’ve seen as low as 2 minutes of obstruction per 12 hr period, and no higher than 12 minutes (during the thickest cloud cover). So it seems like I fixed at least 60%, and maybe almost 95%. Our leaves are finally starting to come in, so I’ll probably let this setup gather data for a bit before deciding how much farther to push it. Come back in a couple of weeks for my May outage data analysis to find out what effect this tower had on my connection statistics.
Every time I post pictures of a project I’ve completed, someone will ask if I have plans I can share. I never do. I have sketches with numbers near them, but I am confident that no one would be able to interpret them. If it has been too long since I made the project, I might have trouble interpreting them myself!
While I gather sketches and measurements of past projects, I think it’s also a good time to explain what tools I’ve used, and why. Most of them are new to me, so I’m hoping this post might generate some discussion on better ways to approach this.
What I’ve settled on for diagramming is OpenSCAD. It’s a 3D modeler, controlled by a programming language that supports basic shape manipulation. I chose a CAD system because I thought that, if I had a full model of the project, I could generate component and assembly diagrams from different, partially-completed views of that model.
I chose a 3D modeler that is programmable because … well, let’s be honest, a good deal of it is because programming is how I interact with computers. But the secondary reason is that I believe the model, itself, is not enough to explain how to build a project. Sure, someone could pull apart a model in whatever tool I used, and inspect it for themselves. But if the point of making the model is to explain the project’s construction, then the product of my process shouldn’t just be the model, but should also include descriptions of the model: diagrams of sizes and angles, and natural language telling a person how to make it.
The model isn’t going to generate natural language build instructions, but if the sizes and angles it uses are available in code, they can also be templated into English written along with the model. To accomplish the templating, I’ve chosen the Jekyll website generator. Via a small script, I can export variable names and values from the model, making them available to include in a templated webpage.
An additional benefit of programmability that I’m excited about is standard version control. That’s exciting because I can develop models iteratively, and improve things over time … and you can help me! The models, the diagrams, and the how-tos are all open-source on Github.
Since you can see my source code, that’s what I’d like to spend the rest of this post talking about. I started learning OpenSCAD only about six weeks ago. If you look through the code repository’s history, you’ll see how I’ve adapted my approach over time. Overall, it has been amazing how quickly I could get useful results from the tool.
There are also places I still feel like I’m fighting the tool. Most of these are places where I would really like the model’s code to somewhat read like a natural description of the creation of the project (start with a piece this size, cut this much off here, attach that other part there), but the details of making the tool render that clearly get in the way (rotate this around x and z, move it an infinitesimal amount to the side to prevent rendering conflicts, color this here so the cut is colored like so, by the way this can be animated). Finding the right abstractions are taking time.
Some abstractions are simple things, like getting used to expressing most things in vectors, instead of individual scalars along (or around) each axis. You can see that I learned that in the perfume display, and then forgot it when I started the toddler tower.
Other things aren’t so much abstractions as they are conventions. For example, which orientation should a component be described in? The way I would think about holding it while making it seems most natural in some ways, but the way it fits into the assembly seems most natural in others. I think the currently popularity of CNC and 3D printing means that most CAD models are described in the orientation that the machine will operate on them. Should I endeavor to describe my components such that they could potentially be made via CNC or 3D printing? Muddled in this decision are which way is up, and where should the origin point be?
Some abstractions seem like more complex concepts. Take, for example, these few notes:
Nodes in the scene cannot be referenced by variables.
No introspection can be done on nodes (size, position, color, etc. are all hidden to the language after creation).
Modules, which look a little like functions in some other programming languages, can create nodes in the scene, but cannot otherwise return values.
Nodes can’t be passed to modules, but there is a facility called “children”, which allows the effects of modules to be chained together.
Functions, which also look like functions in some other programming languages, can not create or alter nodes in the scene.
These notes have strong influence on composability. You can write a module that creates a cube of a certain size, and you can write a module that moves whatever its children are up and to the right, and you can chain them together so that you get a cube of a certain size that is moved up and to the right. But, the mover module can’t base the amount that it moves the children on anything about the children. You have to pass parameterization information like that as arguments to the mover module.
It seems like thinking about nodes in the scene similar to the way one would think about side-effects in other languages is the near the right model. My struggle with it is part of why you’ll see many modules and many functions in each model. Since I want to make diagrams showing each component at different stages of its completion, I need the ability to selectively apply each stage. The best I’ve found so far is to define each step of the creation as another module, so that I can apply them in different combinations. That solution came after being unsatisfied by parameterizing the modules with “do this step” or “don’t do that step” arguments. Functions and variables for every value help to make it possible to keep the many modules in sync without threading all of the information through arguments, though it does make for a lot of names to keep track of.
There are a hundred other little things I’ve learned and experimented with along the way, I’m sure, but I’ll save them for another spew session. The OpenSCAD code is only part of the repository. There’s fun things like Liquid templating and the Pure.CSS layout framework that made building the website relatively quick, which I may write about some day as well. For now, if you have time and interest to look around, read some of the code, and let me know what you think. Or better yet, if you have time, material, and interest, have a go at building one of the projects, and let me know what you think of the instructions!
The Starlink app, whether on a mobile device, or in a web browser, will tell you in which direction the dish regularly finds something blocking its view of the satellites. I’ve had it in my head for a while that it should be able to do more than this. I think it should be able to give you a silhouette of any obstructions.
As a satellite passes through the sky above the dish, the “beam” connecting the two follows it, sweeping across the scene (Figure 0). The dish repeatedly pings the satellite as this happens, and records how many pings succeeded in each second. When the view is clear, all, or nearly all, pings succeed. When there’s something in the way all, or nearly all, pings fail. In theory, if the dish stays connected to the same satellite for the whole pass, we end up with a “scan line” N samples (= N seconds) long, that records a no-or-low ping drop rate when nothing is in the way, and a high-or-total ping drop rate when something is in the way.
One line isn’t going to paint much of a picture. But, the satellite is going to pass overhead every 91 to 108 minutes. The earth also rotates while this happens, so on the next pass, the satellite will be either lower in the western sky, or higher in the eastern sky. On that pass, we’ll get a scan of a different line.
But 91 minutes is a long time for the earth to rotate. That’s farther than one time zone’s width, nearly 23º of longitude. Since the beam is tight, we’ll have a wide band between the two scans in which we know nothing. However, each satellite shares an orbit with 20 or more other satellites. If they’re evenly spaced, that means the next satellite should start its pass only about 4-minutes after the previous one. That’s conveniently only about 1º of longitude. If the dish reconnects to the next satellite in an orbital reliably at a regular interval, we should get 20-ish scan lines before the first satellite comes around again.
But are 1º longitude scanlines enough? Before we get into the math, let’s look at some data. I’ve created a few simple scripts to download, aggregate, and render the data that Starlink’s dish collects. With over 81 hours of data in hand – 293,183 samples – I can make Safari complain about how much memory my viewer is using … er, I mean I can poke around to see what Dishy sees.
In Figure 1, I’ve plotted ping drops attributed to obstructions at one second per 4×4-pixel rectangle. Solid red is 100% drop, and the lighter the shade the less was dropped, with white (or clear/black for those viewing with different backgrounds) being no drops. There are 600 samples, or 10 minutes, per line. It doesn’t look like much beyond noise, so let’s play around.
Figure 2 is the signal-to-noise ratio data instead. White/clear means signal was full (9), solid grey means signal was absent (0), with gradations in between. Still mostly noise, except for the obvious column effect. Those columns are 15 samples wide. So something happens every 15 seconds. It’s not clear what – it could just be an artifact of their sample recording strategy – but that’s as good of a place to start as any for a potential sync frequency.
So let’s drop down to our guesstimated 4 minutes between satellite frequency. With 240 seconds per row (Figure 3) … mostly everything still looks like noise. Let’s start by guessing that the period between satellites is longer.
I clicked through one second increments for a quite a while, watching noise roll by. Then something started to coalesce. At 330 seconds (5.5 minutes) per row (Figure 4), I see two patterns. One is four wide, scattered, red stripes running from the upper right to the lower left. The other is many small red stripes crossing the wide stripes at right angles. Given that this persists over the whole time range, I don’t think it’s just me seeing form in randomness.
Advancing to 332 seconds per stripe (Figure 5) causes the small red stripes to pull together into small vertical stacks. Especially in the later data, some of these blobs seem to fill out quite a bit, encouraging me to see … something.
But here I’m fairly stuck. Doubling or halving the stripe size causes the blobs to reform into other blobs, as expected given their periodicity. But nothing pops out as obviously, “That’s a tree!” I experimented with viewing SNR data instead. It does “fill in” a bit more, but still doesn’t resolve into recognizable shapes.
It’s time to turn to math. I think there are two important questions:
How much sky is covered in a second? That is, what span does the width of a pixel cover?
How much sky is skipped between satellite passes? That is, how far apart should two pixels be vertically?
If I draw the situation to scale (Figure 6), with the diameter of the earth being 12742km, and the satellites being 340 to 1150km above that – giving them orbital diameters of 13082 to 13892km, there’s really not enough room to draw in my geometry! So I’ll have to zoom in.
We can start estimating how big our pixels are by comparing similar triangles. The satellites moving between 7.28 and 7.70 km/s. If we’re looking strait overhead, for our purposes at these relative distances (340 to 1150km), we can consider that 7km to be a straight line, even though it does have a very slight curve. In that case, we can just use scale the triangle formed by the line from us to the satellites T=0 position and the line from us to its T=1sec position, into our scene (Figure 7). If the scene objects are 20m (0.02km) away, then the width of one second at that object is 0.02km * 7.7km / 340km = 0.00045km, or just under half a meter. Compared to the higher, slower orbit, it’s 0.00012km, or 12cm. At 12 to 45cm, we’re not going to see individual tree branches. Resolution will actually get a bit better when the satellite isn’t directly overhead, because it will be further away and so the perceived angle of change will be smaller. But for the moment, let’s assume we don’t do better than half that size.
On to estimating the distance between scan lines. Wikipedia states that there are 22 satellites per plane. If these are evenly spaced around the orbit, we should see one every 4.14 to 4.91 minutes (248.18 to 294.55 seconds). If the earth rotates once every 23hr56m4s, then that’s 1.038º to 1.231º. At the equator, that’s 115.42 to 136.881km. I’m just above the 45th parallel, where the earth’s circumference is only 28337km, so the change in distance here is only 81.705km to 96.897km. If we change our frame of reference, and consider the satellite orbital to have moved instead of the earth, we can use the same math we did last time. To estimate, this distance (81km/satellite) is approximately one order of magnitude larger than the last ones (7km/s), so we can just multiply everything by ten. Thus, our scan lines should be 1.2m to 4.5m apart.
At 12 x 120cm per sample, we’re not going to be producing photographs. At 45 x 450cm, I doubt we’re going to recognize anything beyond, “Yes, there are things above the horizon in that direction.” Let’s see if anything at all compares.
What parameters should we use to generate our comparison scan? If we’re seeing satellites pass in 4.14 minute (91 minutes / 22 satellites) intervals, we should guess that a scan line will be about 248 seconds. If they’re passing every 4.91 minutes, we should guess about 295 seconds. Given the aliasing that integer math will introduce, the fact that 4.14 and 4.91 are kind of the minimum and maximum, and that the satellites won’t sit at exactly those altitudes, it’s probably worth scanning from about 240sec to 300sec, to see what pops up. I see what look like interesting bands show up at 247, 252, 258, and 295 at least. Maybe I’m catching satellites at a band between the extremes?
But then why was 330-332 the sweet spot in our pre-math plot? Maybe I’m just indulging in numerology, but 330 = 22 * 15. Twenty-two is the number of satellites in an orbital, and 15 is the width of the columns we saw in the SNR plot. Could it be that satellites are not evenly spaced through 360º of an orbital, but are instead always 5.5 minutes (330 seconds) behind each other? If that were the case, the orbital would “wrap” its tails past each other. That seems odd, because you’d end up with a relative “clump” of satellites in the overlap, so maybe there’s a better explanation for the coincidence.
In any case, I’m going to forge on with an example from the 332-sample stripe, because its blobs look the strongest of any to me. Let’s also redraw it with the boxes ten times as tall as they are wide, since that’s what I calculated to be the relationship between one satellite’s samples and the next satellite’s samples. If I overlay one of those clumps on the northward view I shared in my last post, does it line up at all?
I’ve stared at this for far too long now, and I have to say that this feels worse than the numerology I indulged in a moment ago. I’m starting to worry I’ve become the main character of the movie Pi, searching for patterns in the randomness. If there’s something here, it needs a lot more knowledge about satellite choice and position to make it work. Even if I adjusted the rendering to account for the correct curve of the satellite’s path and the camera’s perspective, the data is too rough to make it obvious where it lines up.
With some basic information like which satellite the dish was connected to for that sample, and the database of satellite positions, I’m pretty sure it would be possible to throw these rectangles into an augmented-reality scene. Would it be worth it? Probably not, except for the fun of doing it. The obstruction diagram in the Starlink app (Figure 9) divides the horizon into twelve segments. If it shows red in one 30º segment, it’s the tall thing you can see in that segment that is causing the obstruction. This additional data may be able to narrow within the segment, but if there are multiple tall things in that segment, they’re probably all obstructions.
So, while this was a fun experiment, this is probably where it stops for me. If you’d like to explore your own data, the code I used is in my starlink-ping-loss-viewer repo on github. The data used to to generate these visualizations is also available there, in the 1.0 release. Let me know if you find anything interesting!
… and just one more thing before I sign off. Following up on the topic of my past notes about short, frequent Starlink outages. Figure 10 is a rendering of my obstruction (red) and beta (blue) downtime over this data. I’ve limited rendering to only d=1 cases, where all pings were lost for the whole second, since this seems to be the metric that the Starlink app uses for labeling time down. One rectangle per second, 10 minutes per row. The top row begins in the early afternoon on February 9, and the bottom row ends just before midnight on February 12, US central time.
 Many thanks to u/softwaresaur, a moderator of the Starlink subreddit for pointing out that routing is far more complex, since active cells are covered by 2 to 6 planes of satellites, so it’s likely unrealistic to connect to several satellites in the same plane in a row.
 From the same source, routing information is planned on 15 second intervals. At the very least, this means that the antenna array likely finely readjusts its aim every 15 seconds, whether or not it changes the satellite it’s pointing at.
 Again from the same source, while 22 satellites per plane was the plan, 20 active satellites per plane was the reality, though this has now been adjusted to 18. That fits the cycle observation better, as 18 satellites at a 91-108 minute orbit is 5 to 6 minutes between satellites.
But there’s the trouble. To get back the address of the originating account, you have to have both the signature and the original field values. If you supply different field values, you don’t get, “This signature doesn’t match,” you get, “This transaction came from a completely different account.”
Disclaimer: The views in this article are my own, and do not necessarily represent the views of my employer.
As part of the blockchain work I’ve been doing, I’ve been examining designs of the popular existing networks. Ethereum had my attention this week, and I was digging into its transaction authentication mechanisms when I found something confusing. I think it’s easiest to demonstrate with quick example.
Say I’m running a private network, and I submit a transaction to transfer value 1000 from one account to another. I can do that like this:
A key component of blockchains is that every transaction is signed by its issuer, and meddling with the details of the transaction will be obvious to all parties. Let’s verify that. Let’s try to resubmit that transaction, and have it transfer value 1001, which requires modifying just one bit:
Kind of. Value was deposited in the target account, but wasn’t debited from the source account. Where did it come from?
1646999999998999 // started with the same balance as 0xaf4b...
That’s not the account debited in the original transaction. So, I guess it’s true that we weren’t able to replay that transaction with modifications. But what about this other account? It didn’t sign this transaction.
This is where I had to learn the details of how transaction signing works in Ethereum. To submit a signed transaction, your client must encode a string containing: nonce, gas price, gas limit, destination address, value, contract data, and chain ID. The exact encoding is irrelevant here, but those are the components of the transaction (see this post and EIP 155 for the full details). A representation (hash) of those values is passed to an elliptic curve signing function, along with your account’s private key. That function produces what is called a “recoverable signature”. This recoverable signature is added to the end of the previous list, and the resulting string is a “signed raw transaction”.
A signed raw transaction can be submitted to any Ethereum node, without that node needing to know the private key of the account submitting the transaction. Notice that the list of fields included doesn’t include the address of the account submitting the transaction. Anyone can recover that address using the signature and the original list of signed fields.
But there’s the trouble. To get back the address of the originating account, you have to have both the signature and the original field values. If you supply different field values, you don’t get, “This signature doesn’t match,” you get, “This transaction came from a completely different account.”
This is what happened in the example above. If we recover the address using the original values and signature, we get the address we used to sign the original transaction:
Before you run off to tweet about this, let me say: trying to produce a set of field values to make some signature point to a particular account is not within the realm of your powers. I precalculated the account address that would match, and gave it value in my genesis block for the purposes of this demonstration. If I we try again with a value two greater, we get a completely different address that has no value:
Error: insufficient funds for gas * price + value
Trying to match a particular address is a process of mashing numbers hoping to accidentally hit one in 2160. Even if you just wanted to hit any in-use address, and each person alive on Earth had their own, you’d still be looking at one in 2127(=2160/233, 233 ≈ 8 billion).
So why do I care? Two reasons:
Being corrected for the wrong mistake makes the protocol harder to use. The error above for the “two greater value” mismatch points a debugger toward balances, not toward signatures.
Fixing this seems simple.
Number 2 is the naïve thing to say. If it’s so simple, why hasn’t it been done? It’s more likely that I just don’t understand the domain and/or design decisions made elsewhere. I’m going to trudge on with explaining anyway, and hope it leads to my education.
I think this can be fixed by including the originating address in the details that are signed. If what was signed was instead: nonce, gas price, gas limit, originating address, destination address, value, contract data, and chain ID; I think the problem would disappear entirely. We can try the same single-bit modification as last time:
With this scheme, to find a valid transaction, you’re forced to find a match for a specific address. So, as a side effect of improving usability, we also return to a collision probability of one in 2160.
Is it the case that EIPs 712 and 191 are attempting to address some of this situation, but not directly? It seems like “malleability” of ECDSA signatures, while perhaps slightly different than what is described above, is something that has caused trouble elsewhere.
Finally, thanks to the makers of two tools that helped me debug what was going on: Ethereumjs-tx, which includes a nice “from” recovery function, and Keythereum, which can extract a private key from a geth keystore.
Am I on track, or have I missed something? Let me know.
“What is NoSQL?” was an impossible question to answer. Beyond, “A database that doesn’t use SQL,” it meant something different to each person involved. “What is Blockchain?” is the same way.
Disclaimer: The views in this article are my own, and do not necessarily represent the views of my employer.
Blockchain is to 2018 what NoSQL was to 2009.
Large variety of implementations, each addressing a different part of the problem.
Vastly different interfaces, despite overlapping terminology.
Conferences where all of the major players are in attendance.
Discussion of decades-old research finally applicable to modern software.
A few key success stories.
A few gross misapplications.
And, of course, haters from all corners, ready to tell you that you don’t need a blockchain, just like there were so many ready to tell you that you didn’t need NoSQL.
The haters are right, as they always are. You could have used literally any database to do the sorts of storage, and storage guarantees, that NoSQL gave you. You can sign data and write updates immutably to get the sorts of auditability and trust that a blockchain gives you.
This is why I find it interesting that many NoSQL leaders are such vocal Blockchain haters. They know what it’s like to say, “No, not everyone needs NoSQL, but it can provide value to some,” or, “No, they’re not new pieces, but their combination is greater than the sum of their parts.” Yet they’re fully willing to claim that Blockchain is less that worthless.
Maybe part of the problem is terminology, again. “What is NoSQL?” was an impossible question to answer. Beyond, “A database that doesn’t use SQL,” it meant something different to each person involved. “What is Blockchain?” is the same way. Is it the specific structure of “blocks of data, chained by cryptographic hashes”? Is it decentralized trust? If is “proof of $something” consensus? Is it cryptocurrency?
Both of these communities are personal to me. I was part of the NoSQL movement while working on Riak at Basho, and on CloudKit at Apple. I’m writing about Blockchain today, because for the last few months, I’ve been part of the Blockchain movement, working on a project at VMware. The NoSQL argument is over, so I won’t bother rehashing it. But, I would like to discuss here why I’ve found the Blockchain domain worth my time.
A large portion of my interest in Blockchain comes down to the specific project I’ve chosen to work on. My team at VMware understands that, just as NoSQL was a term used to describe tools for enabling scalable databases, “Blockchain” is a term used to describe tools for enabling distributed trust. The core tool often discussed, though it is really the combination of several technologies, is byzantine fault tolerant state machine replication.
Yes, BFTSMR is “just” replication with consensus and cryptography, and you could assemble a number of existing technologies to make your own. But, just like Dynamo was “just” a distributed hash table with logical clocks and gossip, the exact combination provides some rather different capabilities. Each of the components has seen major developments since the classic texts explored them:
Consensus algorithms: with distributed systems becoming ever more common, we’ve had a renaissance of better understanding and implementation.
Cryptographic signatures: hardware has many useful primitives built in to make this efficient, and things like threshold signatures have enabled new optimizations.
State machine abstraction: it is easier than ever to embed an efficient interpreter in your system, to allow quick extensibility and experimentation.
Networking: RDMA, public cloud, edge networking, and other new topologies beg for new communication patterns to harness their potential.
Storage: flash is cheap, and any data model you want is readily available.
BFTSMR has been ignored for years (decades!) because it has been impractical for real systems. With the latest developments, even though it hasn’t reached the speed and efficiency of optimized central databases, it is starting to become a reasonable choice for some applications. If the abstractions commonly provided by Blockchain (like transactions recorded in an immutable log) allow people to take advantage of these classic good ideas and advancements, I am excited to help enable that.
There are some very real things to worry about in association with Blockchain. Cryptocurrencies are not investment strategies, and are not going to have any of the promised power-distribution and -equalization effects they peddle while their value and cost fluctuate so wildly. Control of existing networks is misunderstood. Proof of work is criminally wasteful. Current anonymization is so difficult that most attempts are trivially unmasked. But just as we didn’t let misunderstandings about consistency, early bugs with data loss, or questionable performance numbers bury the very real improvements brought by NoSQL, let’s not write off all of Blockchain just yet.
I’m very excited about the real business improvements that will be enabled by the technical innovations my team is making at VMware. If you would like to join us, please reach out either to oneofus directly or through our careers portal.
 It’s scary how close you can come to an acronym: Byzantine fauLt tOlerant state maCHine replicAtIoN. Maybe that’s why Ethereum is a staCK machine.
 Replication should imply consensus, but we saw how that went in NoSQL.
 Apologies for using “cryptography” to mean all things that may have cryptographic properties: hashes, signatures, obfuscation, …
That graph was based on data that was pre-smoothed using a windowed average provided by the Helium API. The window is one hour, which produced 190 points. Zooming in just a little bit takes us to a 30 minute window, at 303 points:
Unlike the other graphs I plotted in that series, I didn’t plot the maximum and minimum on this one. Here is what the raw data looks like:
That is 11282 points. There are spikes several times taller than what looks like the “typical” variation. The question, then, is did the windowed average portray this data accurately? Here are 303 points overlayed on the 11282 (I cut off the peaks to give us a little more detail):
I think it seems reasonable. A little noisy, but visually following the mid-point of the band. Can ASAP do better? Here are the 304 points it gives when asked for 303 from this dataset:
If you look closely, you can see a few blue edges from the windowed average peaking out from under the ASAP plot, but they’re largely the same. ASAP hasn’t surfaced any additional features in this data that the plain windowed average hid. I think that’s not surprising. The process that was being measured was a slow, continuous change, and not something where there should have been sudden changes in behavior.
That was using window-averaged data as well. Let’s plot it against raw data and ASAP like before:
That’s strange. The blue window-averaged line follows the raw data pretty well, at the 303 points I asked for. The ASAP line looks weirdly off, though. The smooth function only returned 279 points, and it’s caused by a gap in the middle. That straight line around the anomaly contains 30 points. The fact that the curve to the left of that line looks like it’s shifted in time makes me suspicious, but there seem to be no errors generated. Even if I bump up the resolution to the full 890 pixels of this SVG, the ASAP curve looks like this, and produces 31 fewer points than the windowed-average.
Strike up the band, it’s time for the eighth, and final, installment of this fermentation instrumentation series. In part four, I placed several different sensors in several different carboys of beer beginning fermentation. In parts six and seven, I analyzed a week of data from two of the sensors. This post will cover the third sensor, a floating accelerometer.
The ADXL345 provides three readings for each sample: one for each axis in 3D space. I’m using the chip in “2g 10-bit” mode, which means each axis will report a number from -512 (2g negative acceleration) to +512 (2g positive acceleration). In this case, the only acceleration I want to measure is gravity, so I should see only values between -256 (1g negative acceleration; “this axis is pointing straight down”) and 256 (1g positive acceleration; “this axis is pointing straight up”). Using a bit of trigonometry, I should be able to figure out the angle at which the sensor is tilted.
My float is sort of a rounded rectangular prism. I’ve oriented the sensor such that the y-axis is in line with the long axis of the prism, with the positive end pointing toward the end I expect to float. The x-axis is horizontal across the short axis of the prism, and the z-axis is pointed “up”. The expectation is: y will start about zero, or slightly positive, as high buoyancy keeps the float “flat”; x will start about zero as well, because any dip should be along y; and z will start near max, almost straight up. As the beer ferments, reduced buoyancy should cause one end to dip, causing y to increase (because it will point upward more steeply), and z to decrease (because it will move off of straight upward), with x staying the same (because the rotation should be around that axis). So what actually happened?
Pictures may be worth a thousand words, but I’m pretty sure this one just says, “Not that.” We have both x and z increasing, and y is doing … I’m not even sure. Let’s see if there is anything to salvage.
Let’s check an assumption first. I’m expecting to only see acceleration from gravity here, so the total acceleration should always be 1g (plus or minus some measurement noise). We can check that with a bit of Pythagorus: the square root of the sum of the squares of the readings should be a constant 256.
Except for the sudden change in the middle, it’s a variance of about 1, which is 0.0039g for this chip. It’s interesting that it’s only 252 max, and I have no idea what that sudden shift is (it seems correlated with a sudden shift on the z axis, but nothing on the other axes), but it does look like we’re measuring approximately a constant 0.97-0.98g force.
The increasing x may a bit of a red-herring. It just means that the tube is “rolling” (turning around its long axis). This is why z is increasing as well: x returning to horizontal around the unchanging y axis means that z is returning to vertical. There is a chance that the float is rolling instead of tipping as buoyancy changes. This might be worth returning to later, but let’s see if we can save y first.
Despite the fact that our sanity check showed that we’re reading constant gravity as we expect, and therefor all axes agree, we could use Pythagorus again to compute what the value of the y reading should have been, given x, z, and our expected force:
The synthesized y reading is in blue, while the actual y reading is in red. This graph used 256 for the expected gravity. Let’s instead use the 253/250 mix we saw before, which will also account for that unexplained shift in z:
Many features are similar between these plots, but we appear to have exaggerated a somewhat steady descent in y during the period that x and z where steadily climbing (Feb 19 through 21). I expected y to start around zero and become more negative over time. Starting above zero, and decreasing anyway just means that the sensor was tilted away from the expected sinking angle to start. Interestingly, y moving from slightly up to closer to horizontal will also have the effect of bringing z up closer to vertical, just like x moving from negative toward zero did.
If the rolling is not the result of buoyancy change, then this change in y alone leaves us with a change from early Feb 19 mid afternoon Feb 21 of either 33-22 (computed, blue line) or 10-5 (observed, red line). asin(33/256) = 7.41º, asin(22/256) = 4.93º; asin(10/256) = 2.24º, asign(5/256) = 1.12º. Using 252 instead of 256 only alters the result by 0.1º. So, a change of at most 2.5º, and at least 1º. A bit of a narrow bad, if you ask me.
If the sensor shifted during placement, the x axis might be measuring pitch instead of roll. But if even if not, what if the fermentation primarily produced rolling instead if pitching? The x reading swings from -115 to -100. That’s 26.7º to 23º, meaning a change of 4.3º. That’s more, is about all that can be said about that.
If we take both roll and pitch together, we can just consider the change in z, but we also have to ignore the sudden shift near the end of the time range we were looking at. That gives 223 to 231, or 60.59º to 64.46º. Still just 4º change.
If I return to the design of the float, its weight of the float is 1.8oz. So, to float in fresh water, it will have to displace 1.8oz, or 3.24 cubic inches. The float is 4 inches long, by 1 3/16 inches wide, by 7/8 inches tall. A rough estimate places that at 4.15625 cubic inches. The rounded edges are tricky, though. Measuring via displacement shows it’s actually about 3.5 cubic inches. So, what I have is a float that is only just barely floating in water. That was the aim, and what was observed, but good to see the math line up.
If the float has to displace 3.24 cu.in. of water at specific gravity 1.000, then at our starting gravity of 1.040 it only has to displace 3.12 cu.in. of unfermented beer, and 3.22 cu.in. of beer at the finishing gravity of 1.0075. So we’re looking at a change of 0.1 cubic inches of displacement.
I found the center of mass to be about 3/8 inch closer to the end that is expected to sink. That’s not a huge margin of influence, but since the float will be almost entirely submerged anyway, it’s probably enough. The heavy end also happens to have less volume, due to the curvature, so it should have to sink more to displace the same amount.
Calculus is probably the correct way to solve this problem. 18.01 was a long time ago, though, so I’m hoping I can fudge it. Instead of trying to figure out how far this float should have tipped, let’s figure out if a 4º pitch could have changed the displacement 0.1 cu.in.
If we’re already mostly-submerged, we’re looking near an edge that is not 1 3/16 inches wide, but instead closer to 0.75 in (due to curvature). If the float were flat (which the y and z axis readings mostly suggest), 0.75 * 3.75 in would be above water 0.036 in. If we pitched that 4º, we would lose 0.12 cubic inches into the water:
Height lost at 4º over 3.75″: (sin(4/180)*3.75) = 0.08332647479 inches
Volume of 0.083 x 3.75 x 0.75 ” triangular prism: 0.75 * (0.083 * 3.75)/2 = 0.117 cubic inches
So 4º could actually the correct change for these parameters! This does require that the x and z axes were reading pitch, and not roll, though. A 4º roll with these parameters is only a change of about 0.02 cubic inches.
A final check: how well does the shape of this data match the shape of the BeerBug’s?
Comparing the Z axis, it looks like the story is similar to the pressure sensor: the change plateaus at about the same point as the BeerBug, signaling the end of primary fermentation. If the 4º change was measuring the correct thing, then math would have told me the correct final gravity, but it would have been much easier to have developed a calibration table with known angles of specific gravities beforehand.
For the BeerBug, it may be worth continuing to use their service. The device works when the service works, as shown in this data. If anything, I think I’d work on snooping its communication, so I could tee it off to my own storage, in case their service goes down again.
For the pressure sensor, it’s mostly about a new housing, and then calibration to that housing. It needs something that is both heavy enough to sink, and also flexible enough to compress. If both of those are taken care of, it seems like calibrating to known specific gravities may actually provide decent data.
For the tilt sensor, it’s also about a new housing. The weight distribution needs to be far more unbalanced, to ensure a larger change in angle. Something narrower, so that more sinking is required to balance displacement, would work to. If those can be taken care of, then calibration may make this as good as other options.
Both the pressure sensor and the tilt sensor would also benefit from getting the Helium Atom and battery onboard. Current results are probably affected by the cable running out of the carboy. For now, this would require fermenting in something with a wider neck, since the development board is too wide to fit in the carboy. That’s easy to do, and would avoid me having to design my own printed circuitry.
What was really amazing to me is how easy this sort of thing has become. A week after I got hardware, I put it into service. I2C is a nice standard communication protocol. Lua is a quick language to pick up. The Helium chip, library, and service work very smoothly. Between the dev kit and the sensors, I’m over $100, but less than $200 into this exploration. I can see why people are excited about IoT these days – it’s easy to get started, and fun to participate.
But for now, there are two cases of beer to sample in a couple of weeks, and they’re stacked under earlier brews, so I won’t have any more fermentation to measure for a while. I’m setting up one of my Atoms to monitor the temperature in the conditioning closet. I wonder what I should start measuring with the other.
Keeping on the hop, it’s time for part seven of this fermentation instrumentation series. In part four, I placed a few different sensors in some actively fermenting beer to gather data. In the previous post, I looked at data from a commercial sensor. Now it’s time to examine the data from my experimental pressure sensor.
I have two atmospheric pressure sensors collecting data while this beer ferments. One is outside the carboy, while the other is in a non-rigid (i.e. squeezable) container near the bottom of the inside of the carboy. The idea is that as the beer ferments, it will become less dense (because alcohol is less dense than sugar), and thus the same volume of liquid will put less pressure on the sensor.
Let’s start with predictions. I took the long way around and made a table that says that if my sensor is four inches below the surface of the water (it is), it should see about nine millibars of pressure more than just sitting in the air, regardless of specific gravity. This was the long way around, because it turns out “water-inch” is a known pressure unit (equal to 2.49mbar).
Here is what my sensors measured:
Unfortunately, there are two problems, relative to my predictions. The first is that the curves are not 9mbar apart. The second is that the red one is the internal sensor, consistently reading lower than the external sensor. Before I put the sensor in the carboy, I saw about the same difference. It’s possible that the missing 9 mbar is due to the fact that the sensor housing is not laying on the bottom of the carboy, as in the picture at the top of this post, but is instead resting with one end higher than the other. That means the pressure on it is not uniform, and I could be losing all of the additional pressure to the top end (which also happens to be the most flexible end). This might be enough to declare the experiment invalid, but let’s continue looking at what I have anyway.
What is more interesting from the earlier table is the difference we’re supposed to see as the beer ferments. I measured an original gravity (OG) of 1.040 when I pitched the yeast, and a final gravity (FG) of 1.00075 when I bottled (there are some sugars the yeast won’t eat, so we don’t reach 1.000). The predicted difference in pressure is just over 0.3 mbar. The pressure varies by over 15 mbar just due to the weather, though, so how can we tell? By subtracting the external, weather-only pressure from the internal, weather+beer pressure:
Hooray! We do see relative pressure change in the carboy. It’s noisy, but I think we have to compare the top-ish of the hump with the resting level of the plateau on the right. Why not compare the start point at the left with the resting point at the right? The climb on the left is likely one of two things: something similar to what the BeerBug sees, as discussed in part six (i.e. initial oxygen consumption, carbon dioxide production, or yeast proliferation), or the sensor moving. In either case, it does take the yeast 12-24 hours to really get working, and that’s about where the climb levels off, so that’s where the conversion of the sugar really starts.
Drawing some lines across the difference graph, I find the “max” pressure to be about -1.65, and the “end” pressure about -2.05, a difference of 0.4 mbar. That’s 30% off of the hypothesis. This doesn’t seem like a bad error (for a first attempt), but I’m skeptical that we saw this much pressure change without seeing the entire pressure difference (the missing 9 mbar).
So, let’s see if we can answer some unknowns. Before removing the sensor from the carboy, I attempted to figure out if the pressure leveled off higher due to pressure from the airlock. There is about a half inch of water that has to be moved out of the way, which would be 1.2 mbar. That’s nearly double the difference between the start pressure and the resting pressure, but to check, I let the gas out a couple of times between 5pm and 7pm in this graph:
There is no dip in pressure, just added noise from me jostling the cable. The resting pressure change is not from pressurizing the airlock. Follow-up experiments will be necessary to determine if it has anything to do with the specific mixture of absorbed gases, or the presence of yeast cells, I think for now the most likely answer is that the sensor moved.
I also tried to find the missing 9 mbar. After removing the sensor from the carboy, I put it in another container under about 4 inches of water, fully horizontal this time.
That graph starts with the sensor in the open air. It looks like I found about 6 mbar once I got the sensor truly in the bottom of the container. I think the last 3 mbar can probably be attributed to the rigidity of the container – it probably couldn’t deform further.
Perhaps most importantly, how does this compare to the BeerBug’s specific gravity curve? If I do just a little scaling and shifting, I can lay the curves on top of each other:
The two carboys do contain different strains of yeast (the original reason for using three separate carboys), and the BeerBug’s carboy started noticeably faster. So, the different in the start of the curves is to be expected. The head in both, and the bubbling out of the airlocks of both did seem to reduce at about the same time, so the simultaneous arrival at finishing plateau is expected as well.
Overall results are, unfortunately, inconclusive. It looks like the end of fermentation was signaled correctly. I would not have been able to predict the finishing gravity, though. There is enough here to warrant future experiments, I think. This was something of an opportunistic test. I was brewing these three batches anyway, so why not try the sensors? Something with more control (i.e. taking fermentation out of the equation) should be illuminating.
Welcome back for part six of the fermentation instrumentation series. In part four, I placed a few different sensors in some actively fermenting beer to gather data. A week has now passed, and I’ve bottled the beer. Time to look at the data. Let’s start with the device we know – the BeerBug.
We’re lucky this time. New owners have just taken over BeerBug operations, and they’re relaunching the product. Unfortunately, that means they’re going through a bumpy transition period. While I was brewing, I could see the latest reading from my beer, but none of the history. But, after a lengthy email exchange, they have pulled through, and I have the data for this batch.
As before, I’ve uploaded the data to Helium’s servers. This is mostly so I can use the same tools for processing the data for all three of the batches in this experiment. So, with out further ado, this is how the BeerBug thought the specific gravity of my English Mild changed over the week:
This is pretty typical. All the way at the left, we have the gravity that I specified as my starting point, what I read from my glass hydrometer: 1.039. The phenomenon that has been observed for every beer, but is as yet unexplained, comes next: the climb to a higher gravity. This probably has something to do with the initial yeast activity, as they rapidly reproduce throughout the beer, consuming the dissolved oxygen, and beginning to produce carbon dioxide. The gas exchange or cell proliferation may change the buoyancy observed by the BeerBug’s float.
After the initial climb late Saturday, we dive right into the expected steady decline in gravity over the next few days. By night time on Tuesday, the gravity has nearly leveled off. A much slower decline continues as the few yeast cells that haven’t starved continue to find some sugar to eat. By the time I bottled on Sunday, the BeerBug read 1.006. My regular glass hydrometer agreed ±0.001. That’s pretty impressive.
The other thing that seems impressive is that there is far less noise in this data than there was in the BeerBug data from part three of this series. I think the explanation for this begins with the fact that there are fewer points in this dataset. In part three, there was a reading every minute. In this dataset, there is sometimes a reading every minute, but sometimes a reading only every 3, 5, or 10 minutes. This might represent a new strategy in the BeerBug firmware – if the measurement variation was white noise, averaging over longer periods should reduce it. Or, it could be just missing data, which would make the error band (the light blue) close in on the average (the dark blue), because the average *is* the data if you remove enough.
The BeerBug also has a temperature sensor in the housing that sits above (outside) of the carboy. Here is its data, in blue, with the temperature data we looked at from one of the Helium boards in part five of this series, in red:
The readings begin only a degree and a half or so off, but the drop into Sunday morning is deeper for the BeerBug. Its readings also stay consistently nearly 3ºC cooler. This was unexpected, given the placements of these sensors. The Helium sensor was closer to an external door, and the BeerBug was just a few inches above active yeast. I’ll chalk it up to simple differences in the characteristics of the sensors, for now.
I’ve uploaded this data to a gist in CSV format, if you would like to examine it yourself. In the next post, we’ll look at the data from the pressure sensor, and see if we can find a shape similar to the BeerBug’s.
Welcome back for part five of the fermentation instrumentation series. In part four, I placed a few different sensors in some actively fermenting beer to gather data. I now have a few days of pressures and force vectors to analyze …
… but I’m not quite ready to share it all yet. There are some things that look promising, but mainly still a fair bit of confusing. I think there are a couple of quick tests I can run after emptying the carboys that will move some things out of the confusing pile and toward either confirmation or rejection. So, I’m going to delay writing those posts until I can do less handwaving.
To tide you all over until then, I thought I’d share some quick insights from the sensor data that I do not expect to be closely tied to specific gravity: temperature. I have two temp sensors collecting readings: one on a Helium Atom outside the carboys, and one packaged with the pressure sensor submerged in beer at the bottom of a carboy. Let’s start with the one outside the carboy:
This graph tracks the air temperature a few inches from the carboy. It’s basically the air temperature of my kitchen/dining-room. And from it, you can nearly read my life. The temperature drops initially as my kitchen cools after brewing. It rises in the morning as we make brunch, and again in the evening as we make dinner. The spike at 8am Tuesday morning is not breakfast. That is the residual heat from my hand as I held the Atom to connect USB power. The cooling into Wednesday morning is the clouds breaking and the weather temperature dropping.
But there’s something even more fun going on here: the light region around the dark line marks the min/max of the readings. Why is the max so much higher? Enhance.
Where did this sawtooth come from? Clue 1: there are exactly six teeth per hour. Clue 2: I queue up readings for ten minutes, and then send them to the cloud all at once. My bet is that I’m picking up residual heat from that extra work. Looking at my code, I forgot to power down the sensors until after I sent all the data to the cloud. Let’s fix that, and then recheck:
The sawtooth until 11am is what we saw earlier. The jump between 11 and 12 is heat from my hand as I plug in the USB cable again. And then … hmm, same sawtooth. Maybe this is heat from the radio instead. It’s a tenth of a degree Celcius, nothing to worry about, but an interesting artifact.
So, what about the temp sensor in the beer?
Ah, yes, that would be the effect of being surrounded by sixteen pounds of water. It doesn’t change temperature quickly. This works out in the beer’s favor: yeast really don’t like quick temperature changes. Giving them time to adapt keeps them healthy and fermenting.
Here are both temperatures overlaid, so you can compare directly (with bonus 24+ hours on the end):
My apologies for starting with the data you’re all less interested in. It’s too interesting not to share something, but there are too many questions about the other samples to tell a coherent story yet. The data you’re really interested in will be up after bottling, and I’ll share the raw data at that time as well, so you can do your own analysis.
Update: the first set of data, from the BeerBug, is now up in part six.