Beer IoT (Part 8)

Strike up the band, it’s time for the eighth, and final, installment of this fermentation instrumentation series. In part four, I placed several different sensors in several different carboys of beer beginning fermentation. In parts six and seven, I analyzed a week of data from two of the sensors. This post will cover the third sensor, a floating accelerometer.


The ADXL345 provides three readings for each sample: one for each axis in 3D space. I’m using the chip in “2g 10-bit” mode, which means each axis will report a number from -512 (2g negative acceleration) to +512 (2g positive acceleration). In this case, the only acceleration I want to measure is gravity, so I should see only values between -256 (1g negative acceleration; “this axis is pointing straight down”) and 256 (1g positive acceleration; “this axis is pointing straight up”). Using a bit of trigonometry, I should be able to figure out the angle at which the sensor is tilted.

My float is sort of a rounded rectangular prism. I’ve oriented the sensor such that the y-axis is in line with the long axis of the prism, with the positive end pointing toward the end I expect to float. The x-axis is horizontal across the short axis of the prism, and the z-axis is pointed “up”. The expectation is: y will start about zero, or slightly positive, as high buoyancy keeps the float “flat”; x will start about zero as well, because any dip should be along y; and z will start near max, almost straight up. As the beer ferments, reduced buoyancy should cause one end to dip, causing y to increase (because it will point upward more steeply), and z to decrease (because it will move off of straight upward), with x staying the same (because the rotation should be around that axis). So what actually happened?


Pictures may be worth a thousand words, but I’m pretty sure this one just says, “Not that.” We have both x and z increasing, and y is doing … I’m not even sure. Let’s see if there is anything to salvage.

Let’s check an assumption first. I’m expecting to only see acceleration from gravity here, so the total acceleration should always be 1g (plus or minus some measurement noise). We can check that with a bit of Pythagorus: the square root of the sum of the squares of the readings should be a constant 256.


Except for the sudden change in the middle, it’s a variance of about 1, which is 0.0039g for this chip. It’s interesting that it’s only 252 max, and I have no idea what that sudden shift is (it seems correlated with a sudden shift on the z axis, but nothing on the other axes), but it does look like we’re measuring approximately a constant 0.97-0.98g force.

The increasing x may a bit of a red-herring. It just means that the tube is “rolling” (turning around its long axis). This is why z is increasing as well: x returning to horizontal around the unchanging y axis means that z is returning to vertical. There is a chance that the float is rolling instead of tipping as buoyancy changes. This might be worth returning to later, but let’s see if we can save y first.

Despite the fact that our sanity check showed that we’re reading constant gravity as we expect, and therefor all axes agree, we could use Pythagorus again to compute what the value of the y reading should have been, given x, z, and our expected force:


The synthesized y reading is in blue, while the actual y reading is in red. This graph used 256 for the expected gravity. Let’s instead use the 253/250 mix we saw before, which will also account for that unexplained shift in z:


Many features are similar between these plots, but we appear to have exaggerated a somewhat steady descent in y during the period that x and z where steadily climbing (Feb 19 through 21). I expected y to start around zero and become more negative over time. Starting above zero, and decreasing anyway just means that the sensor was tilted away from the expected sinking angle to start. Interestingly, y moving from slightly up to closer to horizontal will also have the effect of bringing z up closer to vertical, just like x moving from negative toward zero did.

If the rolling is not the result of buoyancy change, then this change in y alone leaves us with a change from early Feb 19 mid afternoon Feb 21 of either 33-22 (computed, blue line) or 10-5 (observed, red line). asin(33/256) = 7.41º, asin(22/256) = 4.93º; asin(10/256) = 2.24º, asign(5/256) = 1.12º. Using 252 instead of 256 only alters the result by 0.1º. So, a change of at most 2.5º, and at least 1º. A bit of a narrow bad, if you ask me.

If the sensor shifted during placement, the x axis might be measuring pitch instead of roll. But if even if not, what if the fermentation primarily produced rolling instead if pitching? The x reading swings from -115 to -100. That’s 26.7º to 23º, meaning a change of 4.3º. That’s more, is about all that can be said about that.

If we take both roll and pitch together, we can just consider the change in z, but we also have to ignore the sudden shift near the end of the time range we were looking at. That gives 223 to 231, or 60.59º to 64.46º. Still just 4º change.


If I return to the design of the float, its weight of the float is 1.8oz. So, to float in fresh water, it will have to displace 1.8oz, or 3.24 cubic inches. The float is 4 inches long, by 1 3/16 inches wide, by 7/8 inches tall. A rough estimate places that at 4.15625 cubic inches. The rounded edges are tricky, though. Measuring via displacement shows it’s actually about 3.5 cubic inches. So, what I have is a float that is only just barely floating in water. That was the aim, and what was observed, but good to see the math line up.

If the float has to displace 3.24 of water at specific gravity 1.000, then at our starting gravity of 1.040 it only has to displace 3.12 of unfermented beer, and 3.22 of beer at the finishing gravity of 1.0075. So we’re looking at a change of 0.1 cubic inches of displacement.

I found the center of mass to be about 3/8 inch closer to the end that is expected to sink. That’s not a huge margin of influence, but since the float will be almost entirely submerged anyway, it’s probably enough. The heavy end also happens to have less volume, due to the curvature, so it should have to sink more to displace the same amount.

Calculus is probably the correct way to solve this problem. 18.01 was a long time ago, though, so I’m hoping I can fudge it. Instead of trying to figure out how far this float should have tipped, let’s figure out if a 4º pitch could have changed the displacement 0.1

If we’re already mostly-submerged, we’re looking near an edge that is not 1 3/16 inches wide, but instead closer to 0.75 in (due to curvature). If the float were flat (which the y and z axis readings mostly suggest), 0.75 * 3.75 in would be above water 0.036 in. If we pitched that 4º, we would lose 0.12 cubic inches into the water:

Height lost at 4º over 3.75″: (sin(4/180)*3.75) = 0.08332647479 inches

Volume of 0.083 x 3.75 x 0.75 ” triangular prism: 0.75 * (0.083 * 3.75)/2 = 0.117 cubic inches

So 4º could actually the correct change for these parameters! This does require that the x and z axes were reading pitch, and not roll, though. A 4º roll with these parameters is only a change of about 0.02 cubic inches.

A final check: how well does the shape of this data match the shape of the BeerBug’s?


Comparing the Z axis, it looks like the story is similar to the pressure sensor: the change plateaus at about the same point as the BeerBug, signaling the end of primary fermentation. If the 4º change was measuring the correct thing, then math would have told me the correct final gravity, but it would have been much easier to have developed a calibration table with known angles of specific gravities beforehand.

That wraps up this experiment. I’ve added this data to the gist containing the other sensor data. What’s next for this beer sensing story … ?

For the BeerBug, it may be worth continuing to use their service. The device works when the service works, as shown in this data. If anything, I think I’d work on snooping its communication, so I could tee it off to my own storage, in case their service goes down again.

For the pressure sensor, it’s mostly about a new housing, and then calibration to that housing. It needs something that is both heavy enough to sink, and also flexible enough to compress. If both of those are taken care of, it seems like calibrating to known specific gravities may actually provide decent data.

For the tilt sensor, it’s also about a new housing. The weight distribution needs to be far more unbalanced, to ensure a larger change in angle. Something narrower, so that more sinking is required to balance displacement, would work to. If those can be taken care of, then calibration may make this as good as other options.

Both the pressure sensor and the tilt sensor would also benefit from getting the Helium Atom and battery onboard. Current results are probably affected by the cable running out of the carboy. For now, this would require fermenting in something with a wider neck, since the development board is too wide to fit in the carboy. That’s easy to do, and would avoid me having to design my own printed circuitry.

What was really amazing to me is how easy this sort of thing has become. A week after I got hardware, I put it into service. I2C is a nice standard communication protocol. Lua is a quick language to pick up. The Helium chip, library, and service work very smoothly. Between the dev kit and the sensors, I’m over $100, but less than $200 into this exploration. I can see why people are excited about IoT these days – it’s easy to get started, and fun to participate.

But for now, there are two cases of beer to sample in a couple of weeks, and they’re stacked under earlier brews, so I won’t have any more fermentation to measure for a while. I’m setting up one of my Atoms to monitor the temperature in the conditioning closet. I wonder what I should start measuring with the other.

Beer IoT (Part 7)

Keeping on the hop, it’s time for part seven of this fermentation instrumentation series. In part four, I placed a few different sensors in some actively fermenting beer to gather data. In the previous post, I looked at data from a commercial sensor. Now it’s time to examine the data from my experimental pressure sensor.


I have two atmospheric pressure sensors collecting data while this beer ferments. One is outside the carboy, while the other is in a non-rigid (i.e. squeezable) container near the bottom of the inside of the carboy. The idea is that as the beer ferments, it will become less dense (because alcohol is less dense than sugar), and thus the same volume of liquid will put less pressure on the sensor.

Let’s start with predictions. I took the long way around and made a table that says that if my sensor is four inches below the surface of the water (it is), it should see about nine millibars of pressure more than just sitting in the air, regardless of specific gravity. This was the long way around, because it turns out “water-inch” is a known pressure unit (equal to 2.49mbar).

Screen Shot 2017-02-27 at 9.17.30 PM.png

Here is what my sensors measured:

Blue: External Pressure, Red: Internal Pressure

Unfortunately, there are two problems, relative to my predictions. The first is that the curves are not 9mbar apart. The second is that the red one is the internal sensor, consistently reading lower than the external sensor. Before I put the sensor in the carboy, I saw about the same difference. It’s possible that the missing 9 mbar is due to the fact that the sensor housing is not laying on the bottom of the carboy, as in the picture at the top of this post, but is instead resting with one end higher than the other. That means the pressure on it is not uniform, and I could be losing all of the additional pressure to the top end (which also happens to be the most flexible end). This might be enough to declare the experiment invalid, but let’s continue looking at what I have anyway.

What is more interesting from the earlier table is the difference we’re supposed to see as the beer ferments. I measured an original gravity (OG) of 1.040 when I pitched the yeast, and a final gravity (FG) of 1.00075 when I bottled (there are some sugars the yeast won’t eat, so we don’t reach 1.000). The predicted difference in pressure is just over 0.3 mbar. The pressure varies by over 15 mbar just due to the weather, though, so how can we tell? By subtracting the external, weather-only pressure from the internal, weather+beer pressure:


Hooray! We do see relative pressure change in the carboy. It’s noisy, but I think we have to compare the top-ish of the hump with the resting level of the plateau on the right. Why not compare the start point at the left with the resting point at the right? The climb on the left is likely one of two things: something similar to what the BeerBug sees, as discussed in part six (i.e. initial oxygen consumption, carbon dioxide production, or yeast proliferation), or the sensor moving. In either case, it does take the yeast 12-24 hours to really get working, and that’s about where the climb levels off, so that’s where the conversion of the sugar really starts.

Drawing some lines across the difference graph, I find the “max” pressure to be about -1.65, and the “end” pressure about -2.05, a difference of 0.4 mbar. That’s 30% off of the hypothesis. This doesn’t seem like a bad error (for a first attempt), but I’m skeptical that we saw this much pressure change without seeing the entire pressure difference (the missing 9 mbar).

So, let’s see if we can answer some unknowns. Before removing the sensor from the carboy, I attempted to figure out if the pressure leveled off higher due to pressure from the airlock. There is about a half inch of water that has to be moved out of the way, which would be 1.2 mbar. That’s nearly double the difference between the start pressure and the resting pressure, but to check, I let the gas out a couple of times between 5pm and 7pm in this graph:

Screen Shot 2017-02-27 at 8.48.28 PM.png

There is no dip in pressure, just added noise from me jostling the cable. The resting pressure change is not from pressurizing the airlock. Follow-up experiments will be necessary to determine if it has anything to do with the specific mixture of absorbed gases, or the presence of yeast cells, I think for now the most likely answer is that the sensor moved.

I also tried to find the missing 9 mbar. After removing the sensor from the carboy, I put it in another container under about 4 inches of water, fully horizontal this time.


That graph starts with the sensor in the open air. It looks like I found about 6 mbar once I got the sensor truly in the bottom of the container. I think the last 3 mbar can probably be attributed to the rigidity of the container – it probably couldn’t deform further.

Perhaps most importantly, how does this compare to the BeerBug’s specific gravity curve? If I do just a little scaling and shifting, I can lay the curves on top of each other:

Blue: Differential Pressure, scaled and shifted; Red: Specific Gravity as measured by BeerBug

The two carboys do contain different strains of yeast (the original reason for using three separate carboys), and the BeerBug’s carboy started noticeably faster. So, the different in the start of the curves is to be expected. The head in both, and the bubbling out of the airlocks of both did seem to reduce at about the same time, so the simultaneous arrival at finishing plateau is expected as well.

Overall results are, unfortunately, inconclusive. It looks like the end of fermentation was signaled correctly. I would not have been able to predict the finishing gravity, though. There is enough here to warrant future experiments, I think. This was something of an opportunistic test. I was brewing these three batches anyway, so why not try the sensors? Something with more control (i.e. taking fermentation out of the equation) should be illuminating.

If you want to explore this data yourself, I’ve posted the data in CSV format in a gist.

Stay tuned for the final episode analyzing the accelerometer data soon!

Update: accelerometer data is live in part eight.

Beer IoT (Part 6)

Welcome back for part six of the fermentation instrumentation series. In part four, I placed a few different sensors in some actively fermenting beer to gather data. A week has now passed, and I’ve bottled the beer. Time to look at the data. Let’s start with the device we know – the BeerBug.


We’re lucky this time. New owners have just taken over BeerBug operations, and they’re relaunching the product. Unfortunately, that means they’re going through a bumpy transition period. While I was brewing, I could see the latest reading from my beer, but none of the history. But, after a lengthy email exchange, they have pulled through, and I have the data for this batch.

As before, I’ve uploaded the data to Helium’s servers. This is mostly so I can use the same tools for processing the data for all three of the batches in this experiment. So, with out further ado, this is how the BeerBug thought the specific gravity of my English Mild changed over the week:


This is pretty typical. All the way at the left, we have the gravity that I specified as my starting point, what I read from my glass hydrometer: 1.039. The phenomenon that has been observed for every beer, but is as yet unexplained, comes next: the climb to a higher gravity. This probably has something to do with the initial yeast activity, as they rapidly reproduce throughout the beer, consuming the dissolved oxygen, and beginning to produce carbon dioxide. The gas exchange or cell proliferation may change the buoyancy observed by the BeerBug’s float.

After the initial climb late Saturday, we dive right into the expected steady decline in gravity over the next few days. By night time on Tuesday, the gravity has nearly leveled off. A much slower decline continues as the few yeast cells that haven’t starved continue to find some sugar to eat. By the time I bottled on Sunday, the BeerBug read 1.006. My regular glass hydrometer agreed ±0.001. That’s pretty impressive.

The other thing that seems impressive is that there is far less noise in this data than there was in the BeerBug data from part three of this series. I think the explanation for this begins with the fact that there are fewer points in this dataset. In part three, there was a reading every minute. In this dataset, there is sometimes a reading every minute, but sometimes a reading only every 3, 5, or 10 minutes. This might represent a new strategy in the BeerBug firmware – if the measurement variation was white noise, averaging over longer periods should reduce it. Or, it could be just missing data, which would make the error band (the light blue) close in on the average (the dark blue), because the average *is* the data if you remove enough.

The BeerBug also has a temperature sensor in the housing that sits above (outside) of the carboy. Here is its data, in blue, with the temperature data we looked at from one of the Helium boards in part five of this series, in red:



The readings begin only a degree and a half or so off, but the drop into Sunday morning is deeper for the BeerBug. Its readings also stay consistently nearly 3ºC cooler. This was unexpected, given the placements of these sensors. The Helium sensor was closer to an external door, and the BeerBug was just a few inches above active yeast. I’ll chalk it up to simple differences in the characteristics of the sensors, for now.

I’ve uploaded this data to a gist in CSV format, if you would like to examine it yourself. In the next post, we’ll look at the data from the pressure sensor, and see if we can find a shape similar to the BeerBug’s.

Update: part seven is live with pressure data.

Beer IoT (Part 5)

Welcome back for part five of the fermentation instrumentation series. In part four, I placed a few different sensors in some actively fermenting beer to gather data. I now have a few days of pressures and force vectors to analyze …

… but I’m not quite ready to share it all yet. There are some things that look promising, but mainly still a fair bit of confusing. I think there are a couple of quick tests I can run after emptying the carboys that will move some things out of the confusing pile and toward either confirmation or rejection. So, I’m going to delay writing those posts until I can do less handwaving.

To tide you all over until then, I thought I’d share some quick insights from the sensor data that I do not expect to be closely tied to specific gravity: temperature. I have two temp sensors collecting readings: one on a Helium Atom outside the carboys, and one packaged with the pressure sensor submerged in beer at the bottom of a carboy. Let’s start with the one outside the carboy:



This graph tracks the air temperature a few inches from the carboy. It’s basically the air temperature of my kitchen/dining-room. And from it, you can nearly read my life. The temperature drops initially as my kitchen cools after brewing. It rises in the morning as we make brunch, and again in the evening as we make dinner. The spike at 8am Tuesday morning is not breakfast. That is the residual heat from my hand as I held the Atom to connect USB power. The cooling into Wednesday morning is the clouds breaking and the weather temperature dropping.

But there’s something even more fun going on here: the light region around the dark line marks the min/max of the readings. Why is the max so much higher? Enhance.


Where did this sawtooth come from? Clue 1: there are exactly six teeth per hour. Clue 2: I queue up readings for ten minutes, and then send them to the cloud all at once. My bet is that I’m picking up residual heat from that extra work. Looking at my code, I forgot to power down the sensors until after I sent all the data to the cloud. Let’s fix that, and then recheck:


The sawtooth until 11am is what we saw earlier. The jump between 11 and 12 is heat from my hand as I plug in the USB cable again. And then … hmm, same sawtooth. Maybe this is heat from the radio instead.  It’s a tenth of a degree Celcius, nothing to worry about, but an interesting artifact.

So, what about the temp sensor in the beer?


Ah, yes, that would be the effect of being surrounded by sixteen pounds of water. It doesn’t change temperature quickly. This works out in the beer’s favor: yeast really don’t like quick temperature changes. Giving them time to adapt keeps them healthy and fermenting.

Here are both temperatures overlaid, so you can compare directly (with bonus 24+ hours on the end):


My apologies for starting with the data you’re all less interested in. It’s too interesting not to share something, but there are too many questions about the other samples to tell a coherent story yet. The data you’re really interested in will be up after bottling, and I’ll share the raw data at that time as well, so you can do your own analysis.

Update: the first set of data, from the BeerBug, is now up in part six.

Beer IoT (Part 4)

This is part four of a series on monitoring homebrew fermentation. In parts one, two, and three, I experimented with data I downloaded from one platform and uploaded to another. In this part, I create some new sensors to try.

I have hardware!

Helium Atom connected to an ADXL345

And it’s pretty slick. Using any I2C device with Helium’s wrappers is some of the easiest hardware hacking I’ve ever done. This is my first time using Lua, but while it made some different choices than other common languages, it has been very easy to learn.

Maybe an example will prove my point. This is how you take a reading from an ADXL345 accelerometer (he is Helium’s built-in library):

While building such a script, you can fill it with print statements and run it whenever you like by connecting the Atom to your computer via USB cable. This all makes it super easy to learn how a new sensor works.

When you’ve acquired the measurements you want to save, you send them to Helium’s cloud platform like this:

Once you have posted data, you can use Helium’s dashboard to check it out:


This system is so smooth that in just a week (of evenings) I’ve been able to write scripts to take readings from two different sensors. Those sensors are now sitting in the bottoms of two carboys monitoring the fermentation of an English Mild. Yes, the first thing I did with my new electronics was submerge them in infected sugar water. I tested the water tightness of their containers … oh, at least several times.

Foreground: carboy with “tilt” sensor, carboy with “sink” sensor, carboy with BeerBug; Background: Helium Atoms in red container, airlock/blowoff in green container

What monitoring a fermentation amounts to is measuring the density of the liquid. Water with sugar in it is denser than water on its own, or water with alcohol in it. As the yeast convert the sugar to alcohol, the liquid becomes less dense.

Most tools test the density of the liquid indirectly, by instead testing the buoyancy of a known float. The standard hydrometer is a float with a scale attached, so you can read how high it’s floating by looking at it.

The device I’m looking to replace, the BeerBug, reads this float-height by suspending the float from a flexible metal tongue, which is also connected to a magnet, whose position is read by a hall-effect sensor. As the float floats higher, the magnet nears the sensor, producing a stronger reading. It requires that you measure the gravity of your liquid with a hydrometer first, but once the initial reading is calibrated, the change in buoyancy can be measured (the magnet moves farther from the sensor as the beer ferments).

BeerBug operation – left: pre-ferment, right: post-ferment

I wasn’t able to obtain a hall-effect sensor as quickly as I wanted, so my devices take different approaches. The first is based on someone else’s design. By making the float very buoyant on one end, and just barely not able to float on pure water on the other, the angle at which the float floats will change with the density of the liquid. So the float should start close to horizontal when the unfermented beer is very sugary, and end up more steeply tilted as the sugar is converted to alcohol. The sensor in this float is thus the ADXL345 accelerometer that the above code demonstrates using. By measuring the direction of the force of gravity, we can figure out what angle the sensor is floating at.

Tilt operation – left: pre-ferment, right: post-ferment

The idea behind the second experimental sensor is to directly measure the increased pressure from the denser liquid, instead of measuring its effect on buoyancy. I’ve place an atmospheric presure sensor in a non-rigid housing, which should allow the liquid to squeeze the air around the sensor, raising the pressure around it. As the liquid becomes less dense, the pressure should reduce. The sensor has been placed at the bottom of the carboy, to get as much liquid above it to provide pressure as possible. I’m also taking readings from the pressure sensor on the Atom, which is sitting in the open air outside the carboy, so I can compensate for weather-related pressure changes.

Sink operation (percent pressure as compared with pure water): left: pre-ferment specific gravity of 1.040; right: post-ferment sg of 1.010

So far, I’m just collecting raw data: pressure readings in the latter case, and force readings in the former. It’s going to take some analysis to figure out what they mean. Unfortunately, the BeerBug site is currently only serving the most recent reading, and not history, so direct comparison of data will not be possible for now. The Helium site is running smoothly, though – and in addition to their dashboard, as shown above, I can also use the graphing code from my earlier experiments:

I’ve shared the code I’m using for these experiments on Github. Please feel free to download and use the code yourself, or to suggest ways I can improve my Lua! Check back soon for analysis of how the measurement and fermentation went.

Update: the first bit of analysis, from the temperature sensors is up in part five.

Beer IoT (Part 3)

My code is ugly, but it works, so it’s time to post part three of this series. In part one, I downloaded data captured by my BeerBug. In part two, I uploaded it to the Helium platform. In this entry, I’ll read use Helium’s API to query and graph the data.

If I were dealing with a currently-active data source, Helium’s dashboard would allow me to view what was happening. That is a fantastic resource for developers, because it takes one step of uncertainty out of the equation by allowing inspection in the middle of the pipeline. But, “currently-active” is limited to 90 days in the dashboard, and my data is about a year old, so I need something else.

What I have built are a few simple D3 graphs:


Each graphs the average value for a time slice as a dark line, with a lighter band around it marking the range from minimum to maximum. It’s crude, but it gets the point across. You can move earlier and later in the range by dragging left and right. Zoom in by holding shift while dragging to select a region. Zoom out by holding alt while dragging to select a region.

As I said before, it’s ugly, but I’ve put the code in a gist, if you’re looking for examples to follow (it’s neither well-organized nor well-documented, but if you’re also working with the Helium API, you may pick up on a clue of what you’re looking for).

Some things that made this graphing easy:

  • Helium supports CORS, so I didn’t even have to set up a proxy webservice. Loading graph.html from a file:// URL still allowed me to make requests to Helium to for the data.
  • D3 has a wide variety of basic example graphs. What I started with was a basic mash-up of the Line Chart and Bitvariate Area Chart examples.
  • Helium’s API will give you the latest data for your sensor (note: no 90-day window here), if you don’t provide an end filter, and also include a “previous” link in the response to get the next-latest data.

Some things that made this graphing hard (or at least tricky):

  • D3 defaults to local time, but Helium is all in UTC. Forgetting to translate leads to confusing debugging about why offset calculations are wrong.
  • Helium’s API will always give you the latest data for your sensor, if you don’t provide an end filter. That is, you can really only follow “previous” links backward through time. Once you follow a “previous” link, you’ll get a “next” link, but you should already have the data that link would give you. You can’t begin with a start filter and expect to follow “next” links to the latest data.

I’m posting this simple viewer now instead of waiting until I’ve had time to clean it up more, because the next step is probably a rewrite. As expected, Helium’s API works really well for supporting a simple dashboard: if you’re concerned with recent updates, and then scrolling back in time from there, the API makes it easy. But, what I learned during a Helium presentation at a meetup this week is that the real purpose of this API is to allow Helium’s servers to act as a transport between your sensors and your own servers. The expectation is that you’ll grab data from Helium, store it in your own database, and serve your app from your own storage.

Helium-as-transport is an interesting bet. It’s focusing on exactly the problem I’ve had with my BeerBug: I have to rely on their site for the tool to be useful. If Helium can keep the path from device to my analysis up more reliably, they will succeed in their goal of making sensor IoT more available to people that want to focus on the sensing and the analysis, whtout worrying about the infrastructure in between (i.e. bascially everyone).

Update: Part 4 is up – hardware on display!

FSMs Make Instrumentation Easy

This piece originally appeared on the blog as part of a series on instrumentation.

There is a way to structure programs that makes inclusion of instrumentation straightforward and automatic, and it’s one that every hardware and software engineer should be completely familiar with: finite state machines. You have seen them time and again as illustration of how a system works:

Turnstile state machine

What makes FSM instrumentation straightforward is that the place to expose information is obvious: along the edges, when the state of the system is changing. What makes it automatic is that some generic actor is usually driving a host of specific FSMs. You only need to instrument the actor (“entering state Q with message P”, “leaving state S with result R”), and every FSM it runs will be instrumented for free.

I learned how easy FSMs are to instrument while working on Webmachine, the webserver that is known for implementing the “HTTP Flowchart”.

HTTP Diagram

Each Webmachine resource (a module handling a request) is composed of a set of decision functions. The functions are named for the points in the flowchart where decisions have to be made about which branch to follow. This is just alternate terminology, though: the flowchart and resource describe an FSM, in which the decision points (and terminals) are states.

Driving the execution of a Webmachine resource is a module called webmachine_decision_core. This is where the logic lives for which function to call, and which branch to take based on the result. It triggers each function evaluation by calling a generic webmachine_resource:resource_call function, with the name of the decision.

resource_call(F, ReqData,
                }) ->
    case R_Trace of
        false -> nop;
        _ -> log_call(R_Trace, attempt, R_Mod, F, [ReqData, R_ModState])
    Result = try
        apply(R_Mod, F, [ReqData, R_ModState])
    catch C:R ->
            Reason = {C, R, trim_trace(erlang:get_stacktrace())},
            {{error, Reason}, ReqData, R_ModState}
    case R_Trace of
        false -> nop;
        _ -> log_call(R_Trace, result, R_Mod, F, Result)

This is where the ease of instrumenting an FSM is obvious. The entirety of the hooks needed to support tracing and visual debugging of every Webmachine resource are those two log_call lines. They record the entrance and exit of each state of the FSM without requiring any code to complicate the implementation of the resource module itself. For example, a simple resource:



init([]) ->
    {{trace, "/tmp"}, undefined}.

content_types_provided(ReqData, State) ->
    {[{"text/html", to_html}], ReqData, State}.

to_html(ReqData, State) ->
    {"<html><body>Hello, new world</body></html>", ReqData, State}.

This resource does no logging of its own (as you can see), but for each request it receives, a file is created in /tmp that can be rendered with the Webmachine visual debugger. For example, the processing for a request that specifies Accept: text/html looks like this (live example):


It’s easy to see that the request made it all the way to the 200 OK result at grid location N18. Along the way, it passed through many decisions where the default behavior was chosen (grey-outlined diamonds), and a few where the resource’s own implementation was called (purple-outlined diamonds). Clicking on any decision will display more information about what happened there.

In contrast, the processing for a request that specifies Accept: application/json looks like this (live example):


Now it’s easy to see that the request stopped at the 406 Not Acceptable result at grid location C7 instead. For no more code than specifying where to put the log output, we’ve gotten the complete story of how each request was handled. In case you prefer the original text to this visual styling, I’ve also archived the raw trace files.

This sort of regular, simple instrumentation may seem naive, but the regularity and simplicity offer some benefits. For example, all of the instrumentation points have obvious names: they are the same as the states of the FSM. This alone continues to help beginners bootstrap their understanding of Webmachine. When they’re confused about why something happened, they can go straight to the trace or debugger, and either search for the name of the decision they expected to turn differently, or find the name of the decision that did go differently, and know exactly where to return to in their code. Resource implementors add no code, but get well-labeled tracing for free.

Finite state machines can be found under many other names: flowcharts, chains, pipelines, decision trees, and more. Any staged-processing workflow benefits from a basic “stage X began work W”, “stage X finished work W”, which is completely independent of what the stage is doing, and is equivalent to the stage entering and exiting the “working” state. See Hadoop’s job statistics for an example: generically generated start/stop information that an operator can use to get a basic idea of progress without needing the job implementor to add their own instrumentation. I sometimes even consider the basic request/response logging of multi-service systems as a form of this: sending a request is equivalent to entering a waiting state, etc.

To speak more broadly, the important points to instrument are those when application state is changing. This is how I track down where a process diverged from its expected path, or how long it took to make the change. Finite state machines help by making those points more obvious. Instrumenting state transitions reduces the burden on the implementor, by naturally answering the question of where instrumentation belongs and what it’s called. It also reduces the burden on the user of learning what the implementor decided. Inspection of the system becomes easier because the state transitions are always instrumented, and instrumented in a way that maps directly to the system’s operation.

Thanks to Julia and Charity for organizing the instrumentation series.