Erlang at Basho

Hmm…so, six-ish months ago, I posted that I had just spent a month working at a new job, where I got to code Erlang all day, and that I had a crazy month coming up, which would mean less attention here. That ‘month’ extended itself through the summer, and is going to continue through the fall, but I wanted to give you all an update.

Of course, anyone who really wanted to dig has already found that my new all-Erlang-all-the-time job is at Basho Technologies, but here’s an announcement in a little more obvious place. We’re creating a web-based system for salespersons and sales organizations to track, analyze, and improve their process. We are not just another CRM, and do not intend to become one. We’re providing real tracking, guidance, and analysis, not just an ‘online rolodex’.

But, enough about the business, you all want to know about the tech we’re using, right? Okay, so it’s not quite all-Erlang-all-the-time. Depending on the week, I spend anywhere from 40-90% of my time in Erlang, but because we’re providing a web service, I spend the rest of the day in HTML (or our template/rendering language of choice), Javascript, or CSS. There may be a time at which I’ll have to add Java and/or Flash to that mix as well, but for now, they’re staying down the hall.

Erlang provides the logic for basically all of our server-side code. The webserver, mochiweb with a nice logic layer overtop code-named “webmachine” (which you’ll probably hear more about in the coming months), is entirely coded in Erlang, and accepts Erlang modules as “resource providers.” The storage system uses components from the distributerl project to provide a distributed storage solution. Even our template/rendering language is interpreted by Erlang code.

At this point, I’m sure there are a few of you with your jaws hanging open, minds full of protests and questions. “What about the lack of strings?” “What about the syntax?” I tell you now: these are not problems we’ve stumbled over.

“Lack of strings” is a complete misnomer. There are lists, iolists, and binaries. Through combinations of the three, we get everything we need. We do what coders have been doing for decades: we wrap tricky, repetitive, often-needed code in libraries and move on. Honestly, we’re having more trouble with the standard datetime tuple (no standard for timezone marking, confusing timezone expectations of conversion functions).

To truly get an idea of how much trouble the syntax is, I recommend you try this excercise: code in Erlang-only for a few weeks, then try coding in something else, like Javascript. I think you’ll realize that each language has its own bunch of syntax quirks.

We were able to run the exercise above in our first few weeks of coding. Many of us had never touched Erlang, and some had never touched Javascript. Each language took a couple of days for the learner to be able to produce something useful, and another week or so for the same learner to be able to do it on their own in a reasonable amount of time. We do have complaints about the way each language is written now, but they share little with the complaints we had when starting out.

Case in point: records. Everyone’s favorite structure to beat up (after strings, of course). At first they seem like overly-verbose, half-baked forms of structures from other languages. Then comes the epiphany: they’re not designed to take the place of those structures. The best way to think about records is to envision C-structs full of void pointers. Then you can realize that their real power is in pattern matching – function guards, selective extraction, format assertions.

But just describing the ways in which Erlang hasn’t hindered us isn’t very impressive – how has Erlang actually helped us? Well, we’ve been able to implement our own home-grown distributed storage system in just a couple of months. We’ve also been able to create an entirely new design of webserver that makes writing truly RESTful webservices simple and quick. We’ve been able to troubleshoot problems by quickly opening shell connections to the running webservers and learning exactly what is going on. We’ve written lots of extra self-contained services that can export, import, analyze, and otherwise munge live data easily. In short, we’ve been able to iterate quickly over lots of proposed solutions to problems and choose based on experience, rather than weak assumptions and hearsay.

Or, maybe it’s just that I work with an amazing team that’s able to roll with punches quickly, and usually end up with a better solution in the end than the one we designed in the beginning. Could be – I know this team is the most talented I’ve met since college. :)

Sitting here almost eight months since starting Erlang full-time, and almost two years since picking it up for personal projects, I’d say that I’m here to stay for a while yet. I doubt neither that there will be another language in the future that will steal my attention, nor that there are other languages I’d choose for other types of projects. But, for distributed web-based services today, Erlang has my vote.

About these ads

7 comments so far

  1. Zamous on

    Is your distributed storage system implemented with mnesia by chance? If so can you explain more about it?

    Thanks

  2. Bryan on

    Hey, Zamous. At the moment, yes, mnesia is involved, but only at the node-local storage level. All of the internode communication, redundancy, etc. is handled by other code we’ve written. In fact, mnesia is only the latest mechanism we’ve used for disk storage – we’ve been through flat files and BDB before it. Mnesia’s convenient because it “speaks native” in Erlang, but it’s not a required part of this system.

  3. Robert Virding on

    Thank you, thank you. Someone who is not complaing about the syntax. :-)

  4. Abhijith on

    Thank you for sharing this. I’ve been hearing a lot about Mochi Web. Could you give me an insight into how erlyweb differs from mochiweb and also, why you chose MochiWeb over Erlyweb.

  5. Bryan on

    Hi, Abhijith. Mochiweb and Erlyweb are pretty difficult to compare, actually. While Erlyweb is something of a webapp framework, Mochiweb is more tools for building a webapp framework. In fact, Erlyweb could really be reworked to run *on top of* Mochiweb, and such work has been attempted once or twice.

    We’ve chosen Mochiweb instead of Erlyweb mainly because we’ve made choices about technology we’d like to use that are in disagreement with Erlyweb’s defaults. We use a different templating engine (not ErlTL), a different storage system (not ErlyDB/ErlSQL.), and a different application structure (not URI-chunks == module and function names). There were reasons for each of those decisions as well, but once those decisions were made, it didn’t make sense to try to bend Erlyweb toward them.

  6. Chris Williams on

    I just saw that you posted webmachine to google code, I was actually in the process of building out a similar framework which we were calling Dissident http://github.com/voodootikigod/dissident/ I had not gotten too far on the backend, which from what I can tell was going to be similar if not exact to webmachine. I was wondering if you wouldn’t mind us using webmachine as the backend and putting our AJAX head design concept on top of webmachine?

  7. Bryan on

    Hey, Chris. I’d have to say that Dissident sounds very cool, and webmachine may be a great match for its backend. Webmachine has just been released under the Apache License v2.0, so you should be able to “do what you want with it”, for the most part. I see you’ve already found Justin’s post about it (on blog.therestfulway.com) – he’d be the right person to help sort things out, if needed.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: