• How to make mruby command line applications

    Jon is our resident DevOps Lead Detective. Today's guest post explores mruby, makefiles, C, and investigative DevOpsing.

    What if server management was like solving a murder mystery... Presenting Detectivefile the only film-noir server configuration mangement file.

    # someone stole the system ruby
    investigate! :name => %q{another ruby tuesday} do
      # i bet they are over here somewhere
      stake_out? %q{ruby -v | grep 'ruby 1.8'} do
        usual_suspects! :apt => "pkg=ruby1.8 state=absent"
      # or probably in the repo, yea thats the ticket
      stake_out! %q{ruby -v | grep 'ruby 1.9'} do
        usual_suspects! :apt => "pkg=ruby1.9.3 state=latest"

    But first we need a tool to parse this nonsense. I bet if we wrote a quick Makefile to help us build the project, we could figure out a quick way to include a parser.

    # makefile basics, variable assignment and substitutions
    product=detective-agency # what our utility is called
    build=/tmp/$(product)-build # a directory for our output
    target=$(build)/$(product) # the final goal
    objects = $(patsubst %,$(build)/%, $(patsubst %.c,%.o, $(wildcard *.c)))
    $(target): $(build) $(objects)
        $(CC) $(LDFLAGS) -o $@ $(objects)
        touch $(build) && rm -R $(build)
        mkdir -p $(build)
    $(build)/%.o: %.c
        $(CC) $(CFLAGS) -c $< -o $@

    And a simple C program to read our Detectivefile

    // primary entry point for `detective-file`
    // we need the basics for our application, like file handling
    #include <stdio.h>
    #include <stdlib.h>
    int main(int argc, char** argv) {
      // quickly open the file, exiting with failure code if it is not found
      FILE *f = 0;
      f = fopen("Detectivefile", "r");
      if (0 == f) {
        return 1;
      // perform basic clean, and exit succesfully
      return 0;

    This is mostly un-interesting, but, it highlights how to start a simple command line application. In order to parse our Detectivefile we will need some sort of interpreter. This interpreter needs to have these properties:

    • lightweight
    • easily embeddable, and configurable static library
    • dynamic, reflective language with ability to be meta programed

    It just so happpens that this project exists, and it has a name ... mruby

    But, how does it work? Let us start by adding it as a submodule and including the basic skeleton of mruby integration for our application, main.c

    // then we include the mruby headers
    #include <mruby.h>
    #include <mruby/array.h>
    #include <mruby/irep.h>
    #include <mruby/compile.h>
    #include <mruby/string.h>
    #include <mruby/error.h>
    int main(int argc, char** argv) {
      // ... existing fopen calls
      // and add the implentation for mruby integration
      mrb_state *mrb;
      mrb_value ret;
      // initialize mruby
      if (!(mrb = mrb_open())) {
        fprintf(stderr,"%s: could not initialize mruby\n",argv[0]);
        return -1;
      mrb_value args = mrb_ary_new(mrb);
      int i;
      // convert argv into mruby strings
      for (i=1; i<argc; i++) {
         mrb_ary_push(mrb, args, mrb_str_new_cstr(mrb,argv[i]));
      mrb_define_global_const(mrb, "ARGV", args);
      // cleanup mruby
      // ... existing cleanup and exit code

    And adding the required updates to the Makefile

    # add additional make targets for the library
    $(target): $(build) $(objects) $(mruby_static_lib)
        $(CC) $(LDFLAGS) -o $@ $(objects) $(mruby_static_lib)
        cd mruby && MRUBY_CONFIG=../config/mruby.rb make

    You will notice we have a checkout of mruby in our current directory, and it has a config file mruby.rb

    # in config/mruby.rb, specify the toolchain and debug settings
    MRuby::Build.new do |conf|
      # load specific toolchain settings
      toolchain :gcc

    This of course only gets us started with mruby, next we have to do something constructive with the interpreter, perhaps we can bind a calling context to the Detectivefile handle, and parse it like it might be ruby code?

    // include mruby context and eval() code from opened Detectivefile handle
    int main(int argc, char** argv) {
      // ... existing fopen calls
      // ... existing mrb calls
      mrbc_context *detective_file = mrbc_context_new(mrb);
      mrbc_filename(mrb, detective_file, "Detectivefile");
      ret = mrb_load_file_cxt(mrb, f, detective_file);
      mrbc_context_free(mrb, detective_file);
      if_exception_error_and_exit(mrb, "Exception in Detectivefile\n");
      // ... existing close and exit succesfully

    And as a quick error handler, let us implement

    // NOTE: only a single execution is present at any point in time
    static void if_exception_error_and_exit(mrb_state* mrb, char *msg) {
      // check for exception, only one can exist at any point in time
      if (mrb->exc) {
        fprintf(stderr, "%s", msg);

    And now we should probably add a simple test case to our Makefile so we can track the progress of features in our utility

    # more targets for testing
    test: $(build)/test.yml
        ansible-playbook --list-tasks -v -i 'localhost,' -c local $(build)/test.yml
    $(build)/test.yml: $(target) Detectivefile
        $(target) > $@

    And now we can run make test

    $ make test
    cc  -o /tmp/detective-agency-build/detective-agency /tmp/detective-agency-build/main.o mruby/build/host/lib/libmruby.a
    /tmp/detective-agency-build/detective-agency > /tmp/detective-agency-build/test.yml
    Exception in Detectivefile
            [0] Detectivefile:11
    Detectivefile:11: undefined method 'investigate!' for main (NoMethodError)
    make: *** [/tmp/detective-agency-build/test.yml] Error 2

    Now we need to figure out how to include our Detectivefile implementation, perferring to do most of the grunt work of this application in mruby let us figure out a way to include a directory of custom .rb files

    First we update the mruby build configuration file config/mruby.rb

    # additional declarations to build bin/mrbc
    MRuby::Build.new do |conf|
      # ... existing config
      conf.bins = ["mrbc"]

    Then we update the Makefile

    # continue adding targets for lib/*rb
    static_ruby_headers = $(patsubst %,$(build)/%, $(patsubst lib/%.rb,%.h, $(wildcard lib/*.rb)))
    .SECONDARY: $(static_ruby_headers) $(objects) # this is so we don't lose the auto-generated ruby
    CFLAGS=-Imruby/include -I$(build) # we are now including build output as C source
    $(build)/%.o: %.c $(static_ruby_headers) # all C code requires the build bytecode headers
        $(CC) $(CFLAGS) -c $< -o $@
    $(mrbc): $(mruby_static_lib) # we need the mruby/bin/mrbc app
    $(build)/%.h: lib/%.rb $(mrbc)
        mruby/bin/mrbc -g -B $(patsubst $(build)/%.h,%, $@) -o $@ $<

    Then we need to parse these bytecode header files in our main.c application

    // also add headers for variable argument handling
    #include <stdarg.h>
    #include "detective.h" // detective.rb bytecode
    // allow variable length list of static libs
    static void eval_static_libs(mrb_state* mrb, ...) {
      va_list argp;
      va_start(argp, mrb);
      int end_of_static_libs = 0;
      uint8_t const *p;
      while(!end_of_static_libs) {
        p = va_arg(argp, uint8_t const*);
        if (NULL == p) {
          end_of_static_libs = 1;
        } else {
          mrb_load_irep(mrb, p);
          if_exception_error_and_exit(mrb, "Exception in bundled ruby\n");
    int main(int argc, char** argv) {
      // ... existing init ...
      // ... existing define ARGV constant
      eval_static_libs(mrb, detective, NULL);
      // ... existing shutdown

    Now we re-run make test

    $ make test
    cc -Imruby/include -I/tmp/detective-agency-build -c main.c -o /tmp/detective-agency-build/main.o
    cc  -o /tmp/detective-agency-build/detective-agency /tmp/detective-agency-build/main.o mruby/build/host/lib/libmruby.a
    /tmp/detective-agency-build/detective-agency > /tmp/detective-agency-build/test.yml
    ansible-playbook --list-tasks -v -i 'localhost,' -c local /tmp/detective-agency-build/test.yml
    ERROR: parse error: playbooks must be formatted as a YAML list, got <type 'NoneType'>
    make: *** [test] Error 1
    $ ls -lh /tmp/detective-agency-build/test.yml
    -rw-r--r--  1 mavenlink  wheel     0B Sep  9 01:01 /tmp/detective-agency-build/test.yml

    It would appear that we are not emitting any YAML for ansible to consume...

    Let's take another look at the lib/detective.rb file, maybe we can try to puts something...

    def investigate!(*args)
      puts "---\n- name: stuff"

    And then give it a quick test...

    $ make test
    mruby/bin/mrbc -g -B detective -o /tmp/detective-agency-build/detective.h lib/detective.rb
    cc -Imruby/include -I/tmp/detective-agency-build -c main.c -o /tmp/detective-agency-build/main.o
    cc  -o /tmp/detective-agency-build/detective-agency /tmp/detective-agency-build/main.o mruby/build/host/lib/libmruby.a
    /tmp/detective-agency-build/detective-agency > /tmp/detective-agency-build/test.yml
    Exception in Detectivefile
            [1] lib/detective.rb:4:in Object.investigate!
            [0] Detectivefile:11
    lib/detective.rb:4: undefined method 'puts' for main (NoMethodError)
    make: *** [/tmp/detective-agency-build/test.yml] Error 2

    Bananas, it appears that with mruby we can no longer rely on simple things like the puts function, it is beginning to look like we need to investigate further. Let's figure out how to call back into C with a value and printf from there...

    // lets implement some business logic that simply echos retun of the yields
    static mrb_value business(mrb_state* mrb, mrb_value obj)
      mrb_value ret;
      mrb_value block;
      mrb_get_args(mrb, "&", &block);
      ret = mrb_yield_argv(mrb, block, 0, NULL); // we yield back into ruby expecting a string result 
      if_exception_error_and_exit(mrb, "Exception in business yield\n");
      char *tasks = mrb_str_to_cstr(mrb, ret);
      fprintf(stdout, "%s", tasks);
      return mrb_nil_value();
    // and then include this business into the main application
    int main(int argc, char** argv) {
      // ... existing init ...
      // ... existing define ARGV constant
      mrb_define_method(mrb, mrb->object_class, "business", business, MRB_ARGS_BLOCK());
      eval_static_libs(mrb, detective, investigation, NULL);
      // ... existing shutdown

    And some more implementation

    $ /tmp/detective-agency-build/detective-agency                                                                                                      
    - name: stuff

    This is starting to look like YAML ... but it will be tedious to have to always return a hand-built string literal, we should include a libyaml library, and investigate mrgem, we return to config/mruby.rb to declare a gem dependency

    # don't forget to include -I settings for build
    MRuby::Build.new do |conf|
      # ... existing conf
      conf.gem :git => 'git@github.com:AndrewBelt/mruby-yaml.git', :branch => 'master'
      conf.cc do |cc|
        cc.flags = [ENV['CFLAGS'], "-I#{root}/../yaml/include"].join(" ")

    And now lets add the libyaml code in our Makefile

    # ... more static libs
    # ... and existing object detection
    objects += $(mruby_static_lib) $(yaml_static_lib)
    # ... and more targets to the existing Malefile
    $(target): $(build) $(objects)
        $(CC) $(LDFLAGS) -o $@ $(objects)
        cd yaml && ./configure CFLAGS="-DYAML_DECLARE_STATIC" --enable-static --disable-shared && make
        cd mruby && make clean
        cd yaml && make clean
        touch $(build) && rm -R $(build)

    and more ruby in lib/investigation.rb

    def tasks
        YAML.dump([{:name => "stuff"}])

    And now we are generating more correct YAML (note: the additional ...)

    $ /tmp/detective-agency-build/detective-agency
    - name: stuff

    But this program only emits some unknown form of YAML, it is in fact emitting a complete ansible playbook, if only there was some way we could bridge into that command...

    In conclusion, it is clear that embedding mruby is not only easy, but can add a whole new dimension of fun and extensibility to your applications. There are also other benchmarks to consider as well, take for example a rudimentary implementation of a Detectivefile interpreter, written in pure.rb:

    #!/usr/bin/env ruby
    require 'yaml'
    require './lib/detective.rb'
    require './lib/investigation.rb'
    def business(*args)
        puts yield

    If we compare the number of system calls of detective-agency to the number of system calls generated from the system ruby using dtruss, it is clear that not only is detective-agency lightweight in implementation, but it is lightweight in its execution

    $ sudo dtruss ruby pure.rb 2>&1 | wc -l
    $ sudo dtruss /tmp/detective-agency-build/detective-agency 2>&1 | wc -l
    $ time ruby pure.rb > /dev/null
    real    0m0.048s
    user    0m0.036s
    sys     0m0.010s
    $ time /tmp/detective-agency-build/detective-agency > /dev/null
    real    0m0.010s
    user    0m0.004s
    sys     0m0.003s

    The complete source, and installable debian packages, are available at the detective-agency project homepage. Also for your consideration is the following links to more mruby resources, including articles on how to integrate mruby into your GO or rust apps!

  • Exceptions are Part of Your API

    The adapter pattern is great for wrapping gems that communicate with an external service. For instance, we use the databasedotcom gem to integrate with Salesforce. It's always a good idea to wrap something like this in our own interface. That way, if the gem changes - or we want to change gems - we can make our changes in one place, instead of all over our code.

    When we do this properly, our app shouldn't know anything about how the gem works, and just use the interface. It's easy to forget about exceptions, though, and end up with your app having to rescue Databasedotcom::SalesForceError all over the place.

    This is a leaky abstraction, because details of the implementation 'leak' through to our app. In this case, the leak is the exception.

    Preventing this is simple. First, let's implement a custom error that the rest of our app will be aware of.

    class OurOwnCustomError < StandardError

    Next, catch the gem's errors in our interface, and either handle them or throw our custom error.

    def authenticate_salesforce(options)
      rescue Databasedotcom::SalesForceError => e
        raise OurOwnCustomError, "Could not authenticate with salesforce"

    Now, our app only has to know about the error we implement. Should the gem change, or we change to a different one, we don't have to change code all over our app.

  • Slow is Smooth

    Everybody knows that you have to work fast as a company to survive. Time is not infinite. Money is not infinite. And your customers will go somewhere else to get what they need. It's critical, though, to consider your long term speed, or ability to produce results quickly.

    An old Army saying sums this up: "Slow is smooth and smooth is fast." Activities must be performed with the proper amount of both urgency and diligence. Going too fast and making mistakes really slows you down over the long run. Rush while putting on your gas-mask, and it'll take you twice as long to finish. Trust me, you really don't want that.

    When working on a new feature, you absolutely have to ship it, even if it isn't perfect. However, taking an extra week to make sure that your code is clean and easily understood, that your tests are well-written, and that you're using good practices will save you time later. Write good code - not perfect code - the first time, and you'll deliver much faster overall.

    Finding that balance is hard. Really hard. While I'm way too averse to putting out code that's not perfect, it's also way too easy to justify putting out sub-par work in the name of "moving fast." We have to consider the future time-cost of "breaking things."

  • Searching for Love... on the Client and Server

    In a web app, searching through a list of things is very common. Backbone makes it really easy to filter a collection on the client, or fetch data from the server. When letting a user search through their list of widgets, which do you use?

    Searching on the client is fast and snappy, when there aren't many records. Get a couple thousand widgets, though, and the client side rendering will slow way down. This forces you to search on the server.

    You could always search on the server to avoid this, but why make the server round-trip when there aren't many records?

    Conditionally do both. If there are less than some arbitrary number of records, such as 200, load everything onto the client and do your searching / filtering there. Otherwise, make requests to the server.

    We've implemented this by creating separate client-side and server-side search objects which implement the same api and return a promise. That way, whatever view needs to filter your collection doesn't need to know nor care where the data is coming from.

    Rig up a factory function to return the correct searcher, and you're good to go:

    searcherFactory = (collection) ->
      if collection.state.totalRecords > 200
        new ServerSearcher(collection)
        new ClientSearcher(collection)

    Then, just use it:

    @searcher = searcherFactory(@saxophonists)
    @searcher.search("George Michael").then (filteredSaxophonists) ->
      # Do whatever with your filtered list.
  • Waiting to Capybara

    We recently upgraded to Capybara 2 for our integration tests (I know, I know, only two years late). Doing this upgrade forced us to refactor a lot of our tests so we wouldn’t have to rely on timing as much. However, one issue gave us a lot of problems, but was thankfully pretty simple to take care of once we figured it out.

    The issue we encountered was around testing to see if certain elements get removed from the page in response to an ajax action. Capybara will automatically wait for elements to appear, but it's not as obvious how to wait for something to disappear.

    The code was something like this:

    page.find(".list").should_not have_content(@title)

    As you can see, we're performing a search which fetches its results from the server, and then asserting that a particular title is not on the page. The @title starts off on the page, but is removed after the search.

    Turns out, when Capybara got to that assertion, the ajax request had not yet returned. The title was still on the page, even though everything was working. Seems that should_not have_content was not waiting for the title to be removed.

    So how do we do that?

    Well, it's really easy, although it took me an embarrassingly long time to figure out.

    Change the assertion to:

    page.find(".list").should have_no_content(@second_title)

    The have_no_content matcher will wait for the content to be removed from the page, which is exactly what we want. Taking this a step further, if we need Capybara to wait for something to happen, we should always use positive assertions - should have_no_whatever instead of should_not have_whatever.

subscribe via RSS