Riak Demo: Stickynotes

Warning: This blog post is woefully out of date:

  1. Instead of building Riak from source, you should download a pre-compiled release from downloads.basho.com (though the source is still available, if you want).
  2. Starting Riak is now done with the bin/riak start command, not start-fresh.sh Stopping is also easily done with bin/riak stop
  3. Jiak is no longer the preferred HTTP interface. There is now one that accepts any content type, not just JSON.
  4. Without Jiak, back-linking of notes will need to be done with post-commit hooks.

There are probably other things I’ve overlooked. Really, you should just head over to the Riak wiki to get the intro.


Basho released Riak to the world just over a week ago. Some think that the docs are good enough that further explanation is not necessary. But, if you’re looking for a little more introduction, read on.

The short version of this post: Riak is as simple as downloading it and hitting the HTTP interface:

1$ hg clone http://bitbucket.org/basho/riak/
2$ cd riak
3$ make
4$ ./start-fresh.sh config/riak.erlenv

5$ curl -X PUT -H "Content-type: application/json" \
5>   http://localhost:8098/jiak/foo \
5>   -d "{\"schema\":{\"allowed_fields\":[\"bar\"],\"required_fields\":[\"bar\"],\"read_mask\":[\"bar\"],\"write_mask\":[\"bar\"]}}"

6$ curl -X PUT -H "Content-type: application/json" \
6>   http://localhost:8098/jiak/foo/baz \
6>   -d "{\"bucket\":\"foo\",\"key\":\"baz\",\"object\":{\"bar\":\"Hello World\"},\"links\":[]}"

7$ curl http://localhost:8098/jiak/foo
{"schema":{"allowed_fields":["bar"],"required_fields":["bar"],"read_mask":["bar"],"write_mask":["bar"]},"keys":["baz"]}

8$ curl http://localhost:8098/jiak/foo/baz
{"object":{"bar":"Hello World"},"vclock":"MzIwsDSwMDQyMjAyNjXQLcpMzHYwNDLXMwBCQ3SusYGZoZGxvqG+mbGJobmxkbmFsQEA","lastmod":"Wed, 12 Aug 2009 20:23:50 GMT","vtag":"6DxaqiRCDBevf03tzPZpzl","bucket":"foo","key":"baz","links":[]}

Commands 1-4 download and start Riak. Command 5 creates the bucket foo, and tells Jiak, Riak’s HTTP interface, that objects in bucket foo must have a bar field that is readable and writable. Command 6 creates the object baz in bucket foo. Command 7 gets the listing of bucket foo. Command 8 retrieves object baz in bucket foo.

Aside: Some have found it difficult to stop Riak, so I’ll throw the tip out here: killall heart. You may also have to kill the erl command afterward, but if you don’t kill heart first, Riak will just come right back.

That’s all it takes to use Riak. You can stop reading now and happily speak REST to it for the rest of your application’s lifetime.

If you’re still hanging around, though, maybe you’d be interested in a few more features that Jiak has to offer, like link-walking and field validation. To demonstrate those, I’ll spend the rest of this post describing the creation of an application on Riak.

Let’s say that I wanted to create a note-taking system, something like the fine Erlang/Mochiweb demo put together by the guys at Beebole.

There will be two kinds of objects in my system: notes and groups. The properties of a note will be the text of the note, the color, the position on the screen, and the stacking order. The properties of a group will be the name of the group and the list of notes in the group.

I’ll start by whipping up two modules to manage those buckets for me. These modules tell jiak_resource how to validate the structure of objects in these buckets (jiak_resource is the Webmachine resource that runs Riak’s HTTP interface). They’ll follow the basic structure of jiak_example.erl, which ships with Riak.

-module(groups).
-export([init/2, auth_ok/3, bucket_listable/0, allowed_fields/0,
         required_fields/0, read_mask/0, write_mask/0,
         expires_in_seconds/3, check_write/4, effect_write/4,
         after_write/4, merge_siblings/1]).

init(_Key, Context) -> {ok, Context}.

auth_ok(_Key, ReqData, Context) -> {true, ReqData, Context}.

bucket_listable() -> true.

allowed_fields()  -> [<<"name">>].
required_fields() -> allowed_fields().
read_mask()       -> allowed_fields().
write_mask()      -> allowed_fields().

expires_in_seconds(_Key, ReqData, Context) ->
    {600, ReqData, Context}.

check_write({_PutType, _Key}, JiakObject, ReqData, Context) ->
    {ObjDiffs,_} = Context:diff(),
    case lists:foldl(fun check_diff/2, [], ObjDiffs) of
        [] ->
            {{ok, JiakObject}, ReqData, Context};
        Errors ->
            {{error, list_to_binary(string:join(Errors, ", "))},
             ReqData, Context}
    end.

check_diff({<<"name">>, _, Value}, ErrorAcc) ->
    if is_binary(Value) -> ErrorAcc;
       true             -> ["name field must be a string"|ErrorAcc]
    end.

effect_write(_Key, JiakObject, ReqData, Context) ->
    {{ok, JiakObject}, ReqData, Context}.

after_write(_Key, _JiakObject, ReqData, Context) ->
    {ok, ReqData, Context}.

merge_siblings(Siblings) ->
    jiak:standard_sibling_merge(Siblings).

The groups module ensures that every group has a name field (through implementation of allowed_fields/0, required_fields/0, read_fields/0, and write_fields/0), and that the value of that field is a string (using check_write/4 and check_diff/2).

-module(notes).
-export([init/2, auth_ok/3, bucket_listable/0, allowed_fields/0,
         required_fields/0, read_mask/0, write_mask/0,
         expires_in_seconds/3, check_write/4, effect_write/4,
         after_write/4, merge_siblings/1]).

init(_Key, Context) -> {ok, Context}.

auth_ok(_Key, ReqData, Context) -> {true, ReqData, Context}.

bucket_listable() -> true.

allowed_fields() ->
    [<<"text">>, <<"x">>, <<"y">>, <<"z">>, <<"color">>].

required_fields() -> allowed_fields().
read_mask()       -> allowed_fields().
write_mask()      -> allowed_fields().

expires_in_seconds(_Key, ReqData, Context) ->
    {600, ReqData, Context}.

check_write({_PutType, Key}, JiakObject, ReqData, Context) ->
    {ObjDiffs,_} = Context:diff(),
    case lists:foldl(fun check_diff/2, [], ObjDiffs) of
        [] ->
            {{ok, JiakObject}, ReqData, Context:set_prop(key, Key)};
        Errors ->
            {{error, list_to_binary(string:join(Errors, ", "))},
             ReqData, Context}
    end.

-define(COLORS, [<<"yellow">>, <<"pink">>, <<"green">>, <<"blue">>]).

check_diff({<<"text">>, _, Value}, ErrorAcc) ->
    if is_binary(Value) -> ErrorAcc;
       true             -> ["text field must be a string"|ErrorAcc]
    end;
check_diff({Coord, _, Value}, ErrorAcc)
  when Coord==<<"x">>;Coord==<<"y">>;Coord==<<"z">> ->
    if is_integer(Value) -> ErrorAcc;
       true ->
            [io_lib:format("~s field must be an integer", [Coord])
             |ErrorAcc]
    end;
check_diff({<<"color">>, _, Value}, ErrorAcc) ->
    case lists:member(Value, ?COLORS) of
        true -> ErrorAcc;
        false ->
            [io_lib:format("color field must be one of (~s)",
                           [string:join([binary_to_list(C)||C<-?COLORS],
                                        ",")])
             |ErrorAcc]
    end.

effect_write(_Key, JiakObject, ReqData, Context) ->
    {{ok, JiakObject}, ReqData, Context}.

after_write(_Key, JiakObject, ReqData, Context) ->
    spawn(fun() ->
                  [[_, GroupKey, _]] = jiak_object:links(JiakObject, groups),
                  {ok, C} = jiak:local_client(),
                  {ok, G} = C:get(groups, GroupKey, 2),
                  Key = Context:get_prop(key),
                  C:put(jiak_object:add_link(G, notes, Key, <<"note">>), 2)
          end),
    {ok, ReqData, Context}.

merge_siblings(Siblings) ->
    jiak:standard_sibling_merge(Siblings).

The notes module ensures that all notes have a text field that is a string; x, y, and z fields that are integers; and a color field that is one of a specific list of strings (using the same functions as the groups module used).

One more interesting thing happens in the notes module. If you look at after_write/4, you’ll see that it fetches the groups object that the note links to, and adds the note to that group’s links. Jiak calls after_write/4 after the note has been stored in Riak, so what I’ve written here is effectively an automatic back-link monitor. We’ll return to the concept of links in a moment.

If I put the notes and groups modules in place, I can fire up Riak and immediately being sending HTTP requests to its Jiak interface to create and modify groups and notes. For example:

$ curl -X PUT -H "Content-type: application/json" \
>   http://127.0.0.1:8098/jiak/groups/todos \
>   -d "{\"bucket\":\"groups\",\"key\":\"todos\",\"object\":{\"name\":\"todo\"}\"links\":[]}"

$ curl -X PUT -H "Content-type: application/json" \
>   http://127.0.0.1:8098/jiak/notes/blog \
>   -d "{\"bucket\":\"notes\",\"key\":\"blog\",\"object\":{\"text\":\"finish blog post\",\"x\":0,\"y\":0,\"z\":0,\"color\":\"green\"},\"links\":[[\"groups\",\"todos\",\"open\"]]"

These two lines would create a group named todo with a note labeled finish blog post.

Now, about those links. See the ["groups","todos","open"] item in the links field of that notes object? That’s a link to the groups object named todos, and I’ve tagged it open. The affect_write/4 function in the notes module will add a link to the groups object of the form ["notes","blog","note"].

What does this get me? More than just record-keeping: I can now use the Jiak utility jaywalker to get all of the notes in the todo group with a single query:

$ curl http://127.0.0.1:8098/jiak/groups/todos/notes,_,_

You’ll recognize the first part, through /jiak/groups/todos/. The segment after that is a link query. This ones says "all notes objects, with any tag." The links are structured as {bucket},{tag},{accumulate} segments, with underscore meaning "any."

The example query will return an object with a results field that is a list of lists of results. That is, if I were to store the data returned from that query in a variable called data, my list of notes would be at data.results[0].

I’m not limited to one hop, either. If there were also person objects in my system, linked from notes objects as authors I might get all of the authors of the notes in a group with:

$ curl http://127.0.0.1:8098/jiak/groups/todos/notes,_,_/person,author,_

More information on link-walking can be found in the jaywalker_resource documentation.

The curl commands and the HTTP requests they represent are pretty simple, but I’m going to be hitting these resources from Javascript running in a browser. So, I’ll wrap it all up in a nice utility class:

function JiakClient(BaseUrl, Opts) {
    this.baseurl = BaseUrl;
    if (!(this.baseurl.slice(-1) == '/'))
        this.baseurl += '/';

    this.opts = Opts||{};
}

JiakClient.prototype.store = function(Object, Callback, NoReturnBody) {
    var req = {
        contentType: "application/json",
        dataType: "json"
    };

    if (this.opts.alwaysPost || !Object.key)
        req.type = 'POST';
    else
        req.type = 'PUT';
    
    req.url = this.baseurl+Object.bucket+'/';
    if (Object.key) req.url += Object.key;
    
    if (!(this.opts.noReturnBody || NoReturnBody))
        req.url += '?returnbody=true';

    if (typeof Callback == 'function')
        req.success = Callback;

    req.data = JSON.stringify(Object);

    return $.ajax(req);
}

JiakClient.prototype.fetch = function(Bucket, Key, Callback) {
    return $.ajax({
        url:      this.baseurl+Bucket+'/'+Key,
        dataType: "json",
        success:  Callback
    });
}

JiakClient.prototype.remove = function(Bucket, Key, Callback) {
    return $.ajax({
        type:    'DELETE',
        url:     this.baseurl+Bucket+'/'+Key,
        success: Callback
    });
}

JiakClient.prototype.walk = function(Start, Spec, Callback) {
    var req = {
        dataType: "json",
        success: Callback
    };

    if ('bucket' in Start)
        req.url = this.baseurl+Start.bucket+'/'+Start.key+'/';
    else
        req.url = this.baseurl+Start[0]+'/'+Start[1]+'/';

    for (i in Spec) {
        req.url += (Spec[i].bucket||'_')+','+
            (Spec[i].tag||'_')+','+
            ((Spec[i].acc || i == Spec.length-1) ? '1' : '_')+'/';
    }

    return $.ajax(req);
}

Now I can create the same effect as the curl commands with this code:

var J = new JiakClient("/jiak/");
J.store({
  bucket:"groups",
  key:"todos"
  object:{"name":"todo"}
  links:[]
},
function(group) {
  J.store({
    bucket:"notes",
    key:"blog",
    object:{
    },
    links:[["groups",group.key,"open"]]
  });
});

With the request for "all notes in the todo group" looking like:

J.walk({bucket:"groups", key:"todos"},
       [{bucket:"notes"}],
       function(data) {
         var notes = data.results[0];
         //do stuff...
       });

Okay, now I have my backend and frontend – I’d better fill in the middle. Any intermediary webserver could work, but because Riak comes with Webmachine included, I’ll just setup a quick Webmachine app.

$ cd $RIAK_HOME
$ deps/webmachine/scripts/new_webmachine.erl stickynotes ..
$ cp config/riak.erlenv ../stickynotes/riak-config.erlenv

I copied the Riak config (riak.erlenv) because there are two customizations I want to add to it:

{add_paths, ["../stickynotes/ebin"]}.
{jiak_buckets, [notes, groups]}.

The former puts the stickynotes ebin in Riak’s code path, so Jiak can reach the notes and groups modules I just wrote. The latter does nothing but force the atoms 'notes' and 'groups' into the Riak node so Jiak can use list_to_existing_atom/1.

To my new Webmachine app, I’ll add four things:

  1. the notes and groups modules I just wrote
  2. the static files from Beebole’s example applcation (HTML, CSS, JS) … modified a bit to load and use the jiak.js utility from above
  3. a simple static resource server
  4. a simple proxy resource to pass requests through to jiak (the couchdb_proxy from my wmexamples repo will do fine)

Once I have everything aligned (including dispatch setup correctly), I just start Riak and my Webmachine node:

$ cd $RIAK_HOME
$ ./start-fresh.sh ../stickynotes/riak-config.erlenv
$ cd ../stickynotes
$ ./start.sh

…then point my browser at http://localhost:8000/, and I see the app’s UI with an empty group ready to store some notes. I recommend opening Firebug to watch the requests fly by.

I realize that I’ve glossed over a bit of the Webmachine application stuff, but that’s because it’s mostly rehash of older posts. The better way to cover all of that material is for me to tell you to open up the demo/stickynotes directory in the Riak repo you just cloned, and read the code written there yourself. :)

About these ads

21 comments so far

  1. B. Factor on

    Wow! Riak has an amazingly clever and careful design. Congratulations to the Riak team! I’m surprised this hasn’t made more of a splash.

  2. Bryan on

    Hey – thanks, B. We’re pretty happy with Riak. There’s a list of improvements we’re anxious to get to, but it works now, and we’re excited to show it off.

    We’re seeing quite a bit of splash privately. Lots of people have discussed their Riak experiments with us, but those conversations haven’t bubbled into public channels yet. Being the Rage of Reddit isn’t Riak’s primary goal at the moment anyway. ;)

  3. Dan on

    Hi Bryan,

    Great post. Riak looks like a great tool. Any thoughts on how a two way link would be maintained? For example, if you updated a note with new groups the old groups would still point to the note. Is there a way to find out the links from the old note and update them accordingly?

    Thanks,
    Dan

    • Bryan on

      Hi, Dan. Two-way links can be tricky, partially for the reason you brought up: grooming the other side.

      For your use case, yes, there is a way to handle this. If you look at notes:check_write/4, as implemented in stickynotes, the first thing it does is:

      {ObjDiffs,_} = Context:diff()

      The second half of that tuple, the bit ignored by the underscore, is actually a links diff list. That diff will tell you which links were added, and which were removed. Using this diff list, it would be possible to extend notes:after_write/4 to comb through the list of removed links, and alter the links in each group that the note has just been unlinked from (that is, “remove the stale backlinks”).

  4. Serge on

    Bryan,
    the line 4 did not work for me as is
    until I changed it to

    ./start-fresh.sh config/riak-demo.erlenv

    And in the config/riak-demo.erlenv the $RIAK was replaced by the actual path to Riak.

    And the config/riak.erlenv has two places to take care of:
    riak_dets_backend and riak_heart_command.
    After I created a new store folder for the riak_dets_backend I was able to re-produce your steps 5 to 8.
    Now it gets interesting (to me at least): dets means persistence unlike ets in the riak-demo.erlenv. So I did:

    9. killall heart

    10. epmd -kill
    > Killed

    11. ps -A |grep beam
    > …empty…

    No beam running! We double-checking anyway:

    12. curl http://localhost:8098/jiak/foo
    > curl: (7) couldn’t connect to host

    Cool – Riak is really down!
    Now is the very fun:

    13. ./start-fresh.sh config/riak.erlenv

    And trying again:
    14. curl http://localhost:8098/jiak/foo
    > 503 Service UnavailableService UnavailableThe server is currently unable to handle the request due to a temporary overloading or maintenance of the server.mochiweb+webmachine web server

    HUH??? Let’s help it a bit: repeat your step $5:

    15. url http://localhosqauser@ubuntu:~/src/riak$ curl -X PUT -H “Content-type: application/json” http://localhost:8098/jiak/foo -d “{\”schema\”:{\”allowed_fields\”:[\”bar\”],\”required_fields\”:[\”bar\”],\”read_mask\”:[\”bar\”],\”write_mask\”:[\”bar\”]}}”

    and ask again for foo:

    16. curl http://localhost:8098/jiak/foo
    > {“schema”:{“allowed_fields”:[“bar”],”required_fields”:[“bar”],”read_mask”:[“bar”],”write_mask”:[“bar”]},”keys”:[“baz”]}

    We do see our stored “baz” now – we did not enter it after restart! Let us check on it again: your step $8:

    17. curl http://localhost:8098/jiak/foo/baz
    > {“object”:{“bar”:”Hello World”},”vclock”:
    “a85hYGBgzGDKBVLGRgYGloYGhiZGxiYWhha6RZmJ2Q6GRuZ6BkBoiM41MjEwMrXMYEpkzGNlcLrvdYQvCwA=”,”lastmod”:”Wed, 14 Oct 2009 23:48:18 GMT”,”vtag”:”5g8vRMWGZyEggoZtcRxz35″,
    “bucket”:”foo”,”key”:”baz”,”links”:[]}

    So the persistence worked. What happened on my step 14 right after restart? Are there smarter ways for me to avoid 503s?

    • Bryan on

      Serge:

      A) This blog post is out of date in some spots now (hooray for fast iterations). You did the right thing by switching to config/riak-demo.erlenv, or editing config/riak.erlenv.

      B) The reason your line 14 didn’t work, but line 17 did has to do with how you restarted riak on line 13. In order to retrieve the foo/baz document, Riak needed to look up the schema for the foo bucket. Bucket schemas are stored in the bucket properties, which are stored in the ring state. Using start-fresh.sh creates a fresh, empty ring state. What you wanted to do was “./start-restart.sh config/riak.erlenv”. start-restart.sh would have loaded the ring state left on disk by the riak node you killed earlier.

      • Serge on

        Bryan,

        what I was thinking? this start-restart is the part of the heart I saw and edited many times! That is why I shouldn’t work late :)
        So, the line 13 now is

        13. ./start-restart.sh config/riak.erlenv

        and we won’t be needing that line 15 with PUT in it.

        Cool!

  5. Serge on

    [EDIT]Brian – Please remove my previous comment – the blog guard removed my binaries looking like HTML. In this one – please assume that “groceries” and “mine” are binaries.[/EDIT]

    Done. –Bryan

    Bryan –
    after deleting one key, the list of keys still gets reported the same for several seconds, like the deletion did not happen. If I will try to pull the non-existing value during that period, I will get an error. How will I be “in control” here?

    Taken out of your “doc/basic-client.txt”:
    … I have that key / value for “mine”:
    (cli@127.0.0.1)30>Cli:list_keys(“groceries”). {ok,[“mine”]}

    … all fine, let me delete “mine”:
    (cli@127.0.0.1)31> Cli:delete(“groceries”,”mine”,1).
    ok
    … “ok”! I like it so far. Now let us repeat the same instruction, list_keys, continuously:

    (cli@127.0.0.1)32> Cli:list_keys(“groceries”).
    {ok,[“mine”]}
    (cli@127.0.0.1)33> Cli:list_keys(“groceries”).
    {ok,[“mine”]}

    … 10 (some) seconds later:
    (cli@127.0.0.1)48> Cli:list_keys(“groceries”).
    {ok,[]}
    … now it seems deleted.
    I am guessing about the gossip interval etc, but in the “normal” DB you will see the desired result right after that deletion “ok”. Any hints?

    • Bryan on

      Serge – a few things:

      This delete lag was standard behavior in an earlier version of Riak, but it was tightened considerably in a recent release. Pull the latest, and you will see this 10-second lag disappear.

      However, list_keys is a “special” operation. It’s not supported by most distributed key-value stores, and it’s something that takes a little finesse to make work with them. Riak’s list_keys skirts basically all the logic around vclocks and such to provide something in the form of a key-list, at the expense of 100% consistency.

      And yet, there is one way you can guarantee the behavior you desire: set the RW parameter for the delete/3 function call to the N-value for the bucket you’re deleting from. This will ensure that all N nodes storing replicas of the object have removed that object before delete/3 returns success. In a non-degraded cluster, on a backend that conforms to spec, the key should not appear in a list_keys after such a successful delete.

      Where you have used RW=1 in your example delete call, only one node need have responded with success by the time your delete/3 call returned. The other two nodes may still respond with the “deleted” key in a list_keys call immediately afterward.

  6. Serge on

    Bryan –

    Thank you for the update – the release 0.6 is certainly different: I am back “in control” for my deleted key no longer appears in the list.
    About N and RW.
    I’ve got a feeling that I am working on a bit too low level here and that initial N should be used all over the place for consistency. If I hit it with HTTP RESTful requests, the resources you have written are supposed to take care of such things.
    Also I miss the “delete_bucket” or something like stickydb:reset type of function to “start over” if I am on the dets-based storage. Maybe I just did not dig deep enough for that.

    • Bryan on

      Being able to control N/R/W is one of the powerful benefits of Riak, and you should work understand them before making arbitrary decisions about them.

      For example, there may be cases where you want to demand full consistency by always using N=R=W. However, you need to be aware that doing so is accepting a tradeoff in availability. Reads fail if R nodes don’t respond successfully. Specifying R=N means that you demand all nodes storing replicas of an object to be alive, reachable, and ready.

      The restful HTTP interface exposes R/W through query parameters (?r=R&w=W). By default this interface uses R=W=2 with N=3 to provide a somewhat standard CAP tradeoff (read-your-writes consistency, allow one node down, …), but since no single N/R/W choice is correct for all application, it’s not possible to just “take care of such things.”

      There is no “delete_bucket” function for Riak. To clear out a bucket, delete each of the keys in the bucket, or stop Riak, clear out the disk storage directory, and start a new cluster.

  7. Serge on

    Yes, I have read about R<N great feature of "always available", "consistency – may be later" in the original. I have seen yours and Karger et al references on it, that is the starting point of all things here.

    My question was rather about that later moment. When all nodes are back up again or connection gets restored – I strongly believe the consistensy will be back for deleted keys as well. The test (experiment) will certainly demonstrate it.
    And you are right – the reset won't need to delete the bucket, clearing all keys will suffice for my purposes.

    The last (is there such thing?) question: is dets_backend capable to hold more than that notorious 2Gb? This one seems critical in the persistent storage selection and I could not find any definite answer yet.

    Thank you.

    • Bryan on

      dets_backend is, in fact, subject to all of the limitations of dets.

      BUT, you must realize that Riak, by its distributed nature, mostly skirts the space limitations of dets.

      Each Riak vnode using the dets_backend will open its own table. The number of vnodes is determined by the ring_creation_size parameter in the configuration file. This is often set to something like 16 in developer setups, or 1024 or more in production deployments. 16*2GB or 1024*2GB is a different ballgame.

      Furthermore, it is *not* a requirement of Riak that the entire dataset fit on one node. In fact, data stored with a N-value less than the number of nodes in your cluster will not be stored on all nodes. Therefore, the “maximum capacity” of a given cluster configuration is not as simple as the “maximum capacity” of any given node or backend.

      • Serge on

        I was close to that in my assumptions, but not that definite, of course. To summarize that (please correct me if any), the “small” DB for _corporate_ use can be set up as
        – one box with
        – one node on it
        – with 16 (say) vnodes assigned to it
        will be able to hold 16×2 = _lotsa_ textual data, provided the BLOBs will be placed directly into the file system and the static links will go into the dets instead.

        The last straw of fear before the thing is that under improbable conditions one of vnodes gets overflown way earlier than the others. Are there any triggers implemented or in mind to face that? Or it is totally unlikely?

      • Bryan on

        One vnode overflowing before all others is very unlikely, unless you happen to store some single object big enough to fill an entire vnode on its own. Otherwise, the consistent hashing function and ring partition claim strategies are well-distributed enough that all vnodes should fill approximately evenly.

        There is currently nothing in the dets_backend watching for a potential overflow, but limit-checking code could be added.

        …or you could choose among the other available backends.

      • Serge on

        Terrific – I think that auto-generated riak_util:unique_id_62 will never fall into one vnode. Or event e-mail addresses for that matter.

        I still miss those ordinary 1,2,3… ids like the known @@identity or SEQUENCE, but I admit that they might be non-adequate to the volatile networking multi-node environment with no master node in it.

  8. J on

    I have ran the following curl commands:

    $ sudo ./start-fresh.sh config/riak-demo.erlenv
    $ curl -X PUT -H “Content-type: application/json” http://127.0.0.1:8098/jiak/groups/todos -d “{\”bucket\”:\”groups\”,\”key\”:\”todos\”,\”object\”:{\”name\”:\”todo\”},\”links\”:[]}”
    $ curl -X PUT -H “Content-type: application/json” http://127.0.0.1:8098/jiak/notes/blog -d “{\”bucket\”:\”notes\”,\”key\”:\”blog\”,\”object\”:{\”text\”:\”finish blog post\”,\”x\”:0,\”y\”:0,\”z\”:0,\”color\”:\”green\”},\”links\”:[[\”groups\”,\”todos\”,\”open\”]]}”
    $ curl http://127.0.0.1:8098/jiak/groups/todos/notes,_,_
    {“results”:[[]]}
    $ curl http://127.0.0.1:8098/jiak/notes/blog/groups,_,_
    {“results”:[[{“object”:{“name”:”todo”},”vclock”:”a85hYGBgzGDKBVIsjGWr2jOYEhnzWBk4EqKO8GUBAA==”,”lastmod”:”Mon, 26 Oct 2009 18:01:44 GMT”,”vtag”:”23zAcKJCpRQzn8HKxdy5jh”,”bucket”:”groups”,”key”:”todos”,”links”:[]}]]}

    Based on your instructions and from my understanding of links the results for the 4th and 5th commands seem to be switched. Did I miss something?

    • Bryan on

      Yep. Sounds like you understood it backward.

      Your first curl command creates the “todos” item in the “groups” bucket.

      Your second curl command creates the “blog” item in the “notes” bucket. You’ve added a link in the “notes/blog” item that points to the “groups/todos” item.

      Your third curl command asks Riak to start at the “groups/todos” item and follow all “notes” links it has. The “groups/todos” item has no links at all, so Riak finds nothing to give you.

      Your final curl command asks Riak to start at the “notes/blog” item and follow all “groups” links it has. It has one link (the one you added in the second curl command), so it follows that one, finds the “groups/todos” item and returns it to you.

      Make sense?

      • J on

        Sorry, I wasn’t very clear in stating my confusion. Ignoring my final curl command, I did everything according to the blog instructions including the following:

        “…I can now use the Jiak utility jaywalker to get all of the notes in the todo group with a single query:
        $ curl http://127.0.0.1:8098/jiak/groups/todos/notes,_,_

        But at this point, I received an empty “results” list, as I’ve shown, instead of receiving “all of the notes” as it is stated. Did I misunderstand this sentence?

      • Bryan on

        Aha! The important paragraph was the one just before the bit you quoted – the part where it talks about the affect_write/4 function in the notes module.

        When using the stickynotes app, instead of curl, the notes bucket is marked as being handled by the ‘notes’ module. When objects are stored in the notes bucket, functions from this module are called, and it is one of those functions that adds the reverse link to the groups object.

        If you wanted to do the same with just curl, you’d need to execute something like:

        > curl -X PUT -H “content-type: application/json” http://127.0.0.1:8098/jiak/notes –data “{\”bucket_mod\”:\”notes\”}”

        Assuming you’re using the stickynotes riak config file, and therefore have the stickynotes ebin in your code path, and the notes and groups atoms exist, riak will make the calls you expect, and notes:affect_write/4 will add that reverse link for you.

        HOWEVER, the curl command described required that you pull the latest riak (as of this evening). I just patched a bug while writing this reply. If it’s inconvenient for you to update riak, you can get the same effect by typing the following two lines at the erlang shell running riak (assuming you used the debug-fresh.sh script):

        > {ok, C} = riak:local_client().
        > C:set_bucket(<>, [{bucket_mod, notes}]).

      • J on

        Thanks Bryan! That clarifies things :)


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: