How to Test Rack Attack With Rspec

Why Rack Attack

When your application starts growing and getting more popular you will start getting hit with security scanners, kiddie scripts and hackers trying to access authenticated area of your application.

There are many solutions depending on your application architecture, but today I am going to focus on a middleware solution named Rack Attack.

Rack Attack has recently gained some momemtum with their presentation at the RailsConf 2014. It allows whitelisting, blacklisting, throttling, and tracking based on arbitrary properties of the request.

Although it does not protect the application from a DDOS, it does drastically reduce the efficiency of brute force attacks.

This post will focus on an implementation of the throttling capability of Rack Attack.

Bumps In The Road

The README along with the example configuration gives a great starting point.

Memcache

The throttle state is stored in a configurable cache defaulted to Rails.cache. We are currently using memcache but the integration with Rack Attack did not work as smoothly as we hoped for. We quickly realized that Rack Attack was not setting any keys in memcache despite trying to retrieve them.

We had to update our application to use the gem Dalli to replace memcache-client, and as a side effect to change config.cache_store in each environment configuration file to :dalli_store

Custom response

The syntax provided in the example configuration to customize the response simply didn’t work. The response returned was still having the default ‘Retry Later’ message in the body.

throttled_response = lambda do |env|
    [ 503,  # status
    {},     # headers
    ['']]   # body
end

had to be updated to

def self.throttled_response
    lambda do |env|
        [ 429, { 'Retry-After' => env['rack.attack.match_data'][:period] }, ['Too many requests. Please try again later.']]
    end
end

Rack Attack and Rspec

Rack defines a standard for interaction between web servers and Ruby web applications.

However I could not find much materials about testing Rack middleware with Rspecs. Most tutorials will use a ‘hello world’ example that simply hijack the response request by overwriting it directly. For our purpose we need the request to be processed by our Rails app and the response to be modified if the request needs to be throttled.

Using rack_test is an easy way to test your Rack middleware; just do the following:

  • include Rack::Test::Methods
  • define an instance method app returning your Rails app; you can find the name of your application in config.ru, likely in the format YOURAPP::Application

For example:

describe "throttling urls" do
    include Rack::Test::Methods     

    def app
        YourApp::Application
    end
end

Testing throttling directives

For this example, imagine you would like to throttle your connection for the login action, available in your route as a POST request on ‘/login’.

You would probably define this rule in rack attack initializer:

# Throttle POST requests to /login by IP address
throttle('logins/ip', :limit => 15, :period => 60.seconds) do |req|
    req.ip if req.path == '/login' && req.post?
end

As a conscientious Rails developer you want to make sure to write a test making sure this directive is working as intended and in effect. We should test:

  • The response is not be modified if the number of request is below the limit (15)
  • The response has a 429 status code (or any custom status code defined) if the number is above the number of requests allowed (15)
  • The response is not modified if the HTTP method is not POST (in this case we wouldn’t want to block the login page for a user to enter his credentials)

This translate as:

describe "login" do
    let(:limit) { 15 }

    context "when the number of requests is lower than the limit" do
      it "the response status should not be altered" do
        limit.times do
            post '/login', {}, 'REMOTE_ADDR' => 1.2.3.4
            last_response.status.should_not be(429)
        end
      end
    end

    context "when the number of requests is higher than the limit" do
      it "the response status should be 429" do
        (limit*2).times do |i|
            post '/login', {}, 'REMOTE_ADDR' => 1.2.3.5
            last_response.status.should be(429) if i > limit
        end
      end
    end

    it "does not alter response for GET method" do
      (limit*2).times do
        get '/login', {}, 'REMOTE_ADDR' => 1.2.3.6
        last_response.status.should_not be(429)
        end
    end
end

Note: ideally we would write a test making sure the limit is reset after the period (60 seconds). This could drastically slow down your testing procedure depending on the number of rules you have deployed.

Testing custom response

There are a lot of reasons to customize your response if the request as being throttle; you might want to control the request rate for your API or might have read that returning a 503 to mislead brute force attacker is a good idea.

Imagine you defined this rule:

def self.throttled_response
    lambda do |env|
        [ 503, { 'Retry-After' => env['rack.attack.match_data'][:period] }, ['Too many requests. Please try again later.']]
    end
end

We should then test that when the request has been throttled:

  • The response code is 503
  • The response body is ‘Too many requests. Please try again later.’
  • The response headers include the header ‘Retry-After’ with the period attribute

And that if the request has not been throttled then the status code, body and headers haven’t been modified.

This translate as:

describe "custom response" do
    let(:limit) { 15 }

    context "when the request has been throttled" do
        let(:period) { 60 }

        before do
            (limit*2).times do
                post '/login', {}, 'REMOTE_ADDR' => '1.2.3.4'
            end
        end

        it "returns the custom body message" do
            last_response.body.should == 'Too many requests. Please try again later.'
        end

        it "includes the Reply-After header" do
            last_response.header['Retry-After'].should == 60
        end

        it "returns a 503 status code" do
            last_response.status.should == 503
        end
    end

    context "when the request has NOT been throttled" do
        before do
            post '/login', {}, 'REMOTE_ADDR' => '1.2.3.5'
        end

        it "returns the custom body message" do
            last_response.body.should_not == 'Too many requests. Please try again later.'
        end

        it "includes the Reply-After header" do
            last_response.header['Retry-After'].should be_nil
        end

        it "returns a 503 status code" do
            last_response.status.should_not == 503
        end
    end
end

Caveats

This testing approach is not without issues; if the ratio between the limit and the period is too high it is possible that the testing would not throttle the request. For example it is unlikely that the testing suite can perform 100 requests in 1 second.

However Kickstarter recommendation to set the limit and period to an higher multiple is mitigating this possbility. Instead of limit: 1, period: 1 (1 req/s), do limit: 10, period: 10. The long-term average still can’t exceed 1req/s.

Interestings

  • In Rails, ar_obj.presence is equivalent to ar_object.present? ? object : nil. This is great if you want to replace an empty string with a default value:
1
2
3
4
5
# You can replace
foo.present? ? foo : "default value"

# with
foo.presence || "default value"
  • Github pull requests now have labels as part of their launch of redesigned conversations. Although these are not visible in the PR list view, by flagging our open pull requests with tags such as ‘WIP’, or ‘Reviewable’ to help communicate to potential code reviewers the state of the feature.
  • Elastic search will treat namespaces fields flattened if A) the query doesn’t specify the filed by its namespace and B) the field is not defined at the top level
  • Backbone.js uses $.delegate to delegate focus/blur events to focusin/focusout transparently. In tests, if you try to trigger focus/blur handlers manually you’re gonna have a bad time.
  • Rails Polymorphic associations with an STI model always reference the parent class, not the subclass.

Seamless Elasticsearch Reindexing

Elasticsearch is an excellent search system with fairly effortless scaling and operational semantics. We moved last year from Solr to Elasticsearch and haven’t looked backed. One technique we use with Elasticsearch that we’re very pleased about is zero-downtime full-index rebuilds. This allows us to make zero-downtime index changes and rebuild, even as the search system is being updated with new user-generated content. To achieve this, we use an Elasticsearch Alias to track both the active indices, and any upcoming indices that are currently being built.

As a boiled down example, here’s how we could reindex our Users. (Our real indexing code is a bit more complicated, taking into account background processes, permissions, and other requirements.)

When a user is updated, we look up the users_write alias and use it to write to all tracked indices, which will include both the currently active ones and any that are being built in the background. At the moment we use Tire as a DSL to Elasticsearch.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
def self.index_for_search(user_id)
  Timeout::timeout(5) do
    user = User.find_by_id(user_id)
    write_alias = Tire::Alias.find("users_write")
    if write_alias
      write_alias.indices.each do |index_name|
        index = Tire::Index.new(index_name)
        if user
          index.store user
        else
          index.remove 'user', user_id
        end
      end
    else
      raise "Cannot index without existence of 'users_write' alias."
    end
  end
end

Now, when we want to do a full index rebuild (or initial index creation), we add a new index, add it to the alias, and start building it, knowing that any active users will be adding their data to both indices simultaneously.

We continue to read from the old index until the new one is built, then we switch the read alias.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
def self.reindex_users_index(options = {})
  finished = false
  read_alias_name = "users"
  write_alias_name = "users_write"
  new_index_name = "#{read_alias_name}_#{Time.now.to_i}"

  # Make new index for re-indexing.
  index = Tire::Index.new(new_index_name)
  index.create :settings => analyzer_configuration,
               :mappings => { :user => user_mapping }
  index.refresh

  # Add the new index to the write alias so that any system changes while we're re-indexing will be reflected.
  SearchHelper.set_alias_to_index(write_alias_name, new_index_name, false)

  # Reindex all users.
  User.find_in_batches do |batch|
    index.import batch.map { |m| m.to_elasticsearch_json }
  end
  index.refresh
  finished = true

  # Update the read and write aliases to only point at the newly re-indexed data.
  SearchHelper.set_alias_to_index read_alias_name, new_index_name
  SearchHelper.set_alias_to_index write_alias_name, new_index_name
ensure
  index.delete if defined?(index) && !finished
end

def self.set_alias_to_index(alias_name, index_name, clear_aliases = true)
  tire_alias = Tire::Alias.find(alias_name)
  if tire_alias
    tire_alias.indices.clear if clear_aliases
    tire_alias.indices.add index_name
  else
    tire_alias = Tire::Alias.new(:name => alias_name)
    tire_alias.index index_name
  end

  tire_alias.save
end

Brainstem

A few months ago we released Brainstem, a Ruby library designed to power rich APIs in Rails. The Brainstem gem provides a presenter library that handles converting ActiveRecord objects into structured JSON and a set of API abstractions that allow users to request sorts, filters, and association loads, allowing for simpler implementations, fewer requests, and smaller responses.

Why Brainstem?

  • Separate business and presentation logic with Presenters.
  • Version your Presenters for consistency as your API evolves.
  • Expose end-user selectable filters and sorts.
  • Whitelist your existing scopes to act as API filters for your users.
  • Allow users to side-load multiple objects, with their associations, in a single request, reducing the number of requests needed to get the job * done. This is especially helpful for building speedy mobile applications.
  • Prevent data duplication by pulling associations into top-level hashes, easily indexable by ID.
  • Easy integration with Backbone.js. “It’s like Ember Data for Backbone.js!”

Our talk from RailsConf 2013

Introducing the Mavenlink API

The Mavenlink engineering team is proud to announce the release of a completely new API for interacting with Mavenlink. The new API is dramatically more powerful than the previous API, and is already used by Mavenlink Mobile and other forthcoming in-house projects. We will continue to expand the new API over time, allowing developers from any company to incorporate Mavenlink’s features and user base in order to create valuable and compelling applications.

As part of the new API, Mavenlink customers can now easily approve applications via the OAuth 2 protocol. OAuth 2 allows application developers to request permissions and interact securly with Mavenlink on behalf of Mavenlink customers.

To begin using the new Mavenlink API, simply register your application and then head over to the Mavenlink developer site for the full API documentation. While the API that we are launching today has been tested in production over the past several months, we will continue to expand the functionality available through this API. Please give us your feedback, report any bugs, let us know if you have any questions, and tell us what features you’d like to see in our API to empower your own applications.

A Tale of Two Ys

Today marks a momentous occasion: today, scheduled Do Not Disturb mode on iOS starts working again! While the exact reasons that the feature stopped working between January 1 and January 7 are unlikely to ever be explained, investigating date bugs during the new year led me to discover a fascinating (and horrifying) aspect of dates on modern computers: the ISO 8601 Week Date.

The best (or worst, really, if you’re a programmer) part of the Week Date is the Week Year — it’s a four-digit year that looks identical to the regular year. Except between December 28th and January 4th, when it might be one year later or one year earlier than the regular year. Awesome, right?

In Ruby, or C, or any language that uses strftime() to format dates, it’s not that hard to make sure that you get the regular year instead of the week year. According to the IEEE standard, %Y and %y are both the regular year (in four and two digits, respectively), while %G and %g are the week year. Where it gets tricky is in other languages, like Objective-C, that use Unicode date format patterns. In those strings, mm represents the minute, and MM represents the month. DD will print the two-digit day of the year, while dd will print the two-digit day of the month. So, to get the month and day, you use a string like MM-dd. With that precedent, you might think that adding the year would mean using YYYY-MM-dd. Unfortunately, in a case of horrible ambiguity that is extremely hard to catch in advance, YYYY is the week year, while yyyy is the regular year.

At that point, the main way to find out that you’re using the wrong capitalization in your pattern is to notice that your application thinks it’s a different year for a few days around the new year. And that seems bad. So everybody that uses date format patterns: write a script that will complain about uses of YYYY. Seriously. You’ll probably need it later.

Notes on “Exceptional Ruby” by Avdi Grimm

I read “Exceptional Ruby” by Avdi Grimm this weekend. An exceptional book, so to speak, and worth a read.

Some of the highlights for me, representing either things I didn’t know or things that deserve a reminder:

  • raise and fail are completely identical commands. Which you use is personal preference.
  • rescue can take a dynamic list of exceptions:
1
2
3
4
5
begin
  
rescue *exceptions_of_Interest => e
  warn "Got exception #{e.class} with message: #{e.message}"
end
  • Never, ever return inside of an ensure block. It swallows any exceptions.
1
2
3
4
5
6
7
begin
  raise "Oh no!"
ensure
  return "hi"
end

=> "hi"
  • $! is a thread-local variable that always contains the last (unhanded) exception. This is useful for making code that returns the encountered exception instead of raising it:
1
2
answer = get_answer() rescue $!
# answer is either the answer, or an exception object
  • It’s generally dangerous to use postfix rescues (foo rescue nil) because they swallow any exceptions inside of them. Use with great care!

  • Never rescue Exception unless you want to rescue EVERYTHING that can go wrong, including OutOfMemory, SystemExit, etc. It’s better to rescue StandardError, which is also what a naked rescue statement does.

  • Along the same lines, rescue the most specific Exception you can, even if you have to match the message:

1
2
3
4
5
6
7
8
9
10
11
begin
  
rescue SpecificErrorOne => e
  
rescue => e
  if e.message =~ /some error pattern/
    ..
  else
    raise
  end
end
  • When an exception is raised inside of a rescue block, you generally lose the original exception, which makes debugging hard. A better option may be to raise a second exception that keeps a reference to the original exception.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
class WrappingError < StandardError
  def initialize(message, original = $!)
    super(message)
    @original = original
  end
end

begin
  begin
    raise "something"
  rescue
    raise WrappingError, "another error"
  end
rescue => e
  puts e.message # => "another error"
  puts e.original.message # => "something"
end
  • When an exception occurs in a Thread, the exception is re-raised when the Thread is joined. If it’s never joined, the exception will not be re-raised.
1
2
3
4
thread = Thread.new { raise "foo" }


thread.join # => RuntimeError: foo
  • Threads have a stored final return #value in Ruby which can avoid the use of a temporary variable. Calling #value before the Thread finishes will block until it returns.
1
2
3
a = Thread.new { compute_something }
a.join
puts a.value
  • One pattern for dealing with exceptions in external services is to use a Circuit Breaker. If too many exceptions occur in a certain time period, trip the breaker and disable the external service in the GUI until either a specific time period has elapsed or a human has turned it back on. Once re-enabled, put the breaker in a half-open state that will break quickly again on new exceptions. Transition to a fully open state again if things go well after a time period. There is apparently a Gem for this.

  • When a library raises exceptions, it’s often a good idea to raise custom exceptions that all inherit from a base exception in the library. If this doesn’t make sense, another option is to tag all exceptions that pass through the library:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
module MyLib
  module Error; end
end

def tag_errors
  yield
rescue Exception => error
  error.extend(MyLib::Error)
  raise
end

def some_library_method
  tag_errors do
     
  end
end

Turning on Strict Mode for MySQL Installed With Homebrew

This seemed to be worth some documentation.

Strict mode is useful for finding database inconsistencies and web application bugs. Notably, Travis CI and Rails 4 both use strict mode by default.

On a Mac with MySQL installed with Homebrew, you can turn on strict mode by editing /etc/my.cnf. This file may not exist yet since Homebrew installs mysql with no configuration beyond the default. Use sudo vi /etc/my.conf or similar and add the following to the file:

[mysqld]
sql_mode="STRICT_ALL_TABLES"

Restart mysql:

mysql.server restart

And check that it worked:

mysql -e 'select @@GLOBAL.sql_mode;'

Now run your specs!

Seven Habits of Highly Effective Programmers

Philip Chu at Technicat penned a nice piece on being an effective programmer.​ In it, he talks about spiking on design, iteratively delivering real software, keeping things functioning during development, and managing refactorings. Unlike building a basic physical structure, building a sensible software interface usually presents on-the-fly issues each time something is designed.

The first step in becoming an effective programmer is to ensure that you are spending your time wisely. And there is no greater waste of time than in working on something that is not useful or never shipped.

Many of the thoughts in Philip’s piece reminded me of Zuck’s thoughts in The Hacker Way.

The Hacker Way is an approach to building that involves continuous improvement and iteration. Hackers believe that something can always be better, and that nothing is ever complete. They just have to go fix it — often in the face of people who say it’s impossible or are content with the status quo.

Hackers try to build the best services over the long term by quickly releasing and learning from smaller iterations rather than trying to get everything right all at once.

Mark Zuckerberg Letter to Investors

The WSJ Got It Wrong on Pair Programming

On Sunday evening, Joseph Walker at the Wall Street Journal published an article belittling pair-programming and taking quotes out of context.

As co-founder and CTO of Mavenlink, I’ve been pair-programming with my team for nearly 4 years. I want to set the record straight.

A (somewhat) brief history

At my last job, we did a lot of custom software delivery and quite a few proof-of-concepts under incredible time pressure. Often, we would find ourselves behind on a complex project or getting into a sales cycle a little late. When that happened, many of us would drag together a couple of chairs and crank out some code. Other than group projects in college, this was my first experience with any type of pair-programming, although I didn’t know it at the time.

Ironically, due to social and management pressures, we only ever did it when there was absolutely nothing to lose and we just had. to. make. it. work.

When I left that job to co-found Mavenlink, we decided early on to use Ruby on Rails and started looking for an early-stage development partner. We found Pivotal Labs. During our project’s scoping (hi Davis and Rajan!), we talked about Pivotal’s general agile practices, XP, test-driving, and pairing. For us, at the time, pairing was a great way to get me trained up on their best practices. Little did I know that we’d spend the next 3 years working there and would hire our own team based on pair-programming acuity.

Why the WSJ got it so wrong

1) Pairing is not for everyone

I would never go into a company as a new engineering manager and tell everyone that they have to pair, full-time, starting right then. That makes no sense. If an engineering department or team isn’t structured around pairing from the get-go, it’s very possible that it simply wouldn’t work. Some people, and certainly some developers, just like to put on headphones and solve problems quietly on their own.

For better or worse, in our hiring, we look for people who fall on the social-side of the spectrum when they’re working on problems. Our interviews are specifically geared towards collaborative problem solving and we include a pair-programming component as a further check.

WSJ: Thanks for giving us one example of a company whose culture didn’t like pairing and an example of some guy who didn’t like it. What about all of the other companies and people who do?

2) Pairing, like anything else, is not about the ideal

In the ‘90s, many people considered Yoga pretty weird. If you did Yoga, you were probably “from California” or someplace oddball like that, right? Now, in 2012, Yoga has non-weird proponents everywhere who talk about the increase in strength, flexibility, and mental acuity that comes from regular practice. However, not everyone who does yoga looks like a pretzled Yogi.

Pairing is about collaboration, writing better code, teaching, and keeping each other in check. With some practice, collaboration and bouncing ideas around comes more easily, but not everyone is going to grunt ‘n’ code, as Kent Beck mentioned. You don’t stop doing Yoga because your hamstrings are still tight and you don’t stop pairing because you need to talk a little bit (or a lot).

WSJ: Thanks for taking Kent’s quote, using it as some freakshow article byline, and then sidelining him with “That’s the theory, anyway.”

3) Writing software, like building anything, is a team sport

The article paints a picture of overly-aggressive, smelly, bad-breathed, terribly-socially-matched people not being able to pair program. My question is: pairing or no pairing, are these people really able to work together AT ALL? Everyone has their quirks, but is the alternative to sit in separate offices with closed doors doing your own thing and expecting good results? What if a sports team did this?

If we’re going to take an extreme view on pair programming, that’s the other extreme: non-collaborative, opinionated, siloed work that comes out of one person’s head. In reality, nothing, including software, is ever built like that. There are status meetings, whiteboard sessions, scrum sessions, architecture meetings, and all the other checkpoints. All of these were invented to increase communication and collaboration between team members.

Pair programming and daily pair rotations help reduce the need for many of these meetings. If we all frequently cycle through much of the product, everyone has experience with the business’s parallel development tracks and every pair is able to make their own decisions around the day’s work.

WSJ: Thanks for the dating analogies to drive home your point. I guess it doesn’t matter that over 30% of traditional (non-paired) office workers say they have a “work spouse“…?

4) How rotations work in the real world

One traditional model is to build software separately, doing regular peer code reviews, and coaching each other. This can work well when an organization is extremely disciplined, but what often happens in reality is that every developer has 16 hours of code to write in an 8 hour day, so the last thing that they’re thinking about is someone else’s problem. Also, without context or a large time investment, how am I supposed to get my head around a problem that someone else has been working on for days? Time-strapped reviews often lead to surface-level re-factorings, but not to deep discussions and improvements.

With pair-programming, we’re live reviewing our pair’s ideas and constantly collaborating on a solution. Sure, it can be a little daunting at first, but you quickly realize that we’re all playing for the same team with the same goals in mind. Holding dogmatic views about how something should be done is not unique to engineers, nor is it unique to engineers who pair. By combining agile design techniques with constant communication and rotations, pair programming is a great way to iteratively come to consensus, rather than “throwing the cards in the air” and starting over after something is half built.

*WSJ: We’re so “promiscuous,” we rotate daily. Yeah, it’s hot.**

5) Lack of shared knowledge is a massive business risk (pay attention Mr./Ms. Executive)

Software developers are often hired into a very specific position to do a very specific piece of the product. Let’s say your product has widgets, dongles, and plugs. Well, it’s natural to hire a person who only ever works on widgets. Over time, that person becomes the “widget guy” and has most, or all, of the context on widgets. Whenever Susan in sales has a question about widgets, she calls the “widget guy”; when Joe in support has a customer with a bug, he calls the “widget guy”; and so forth.

Now, “widget guy” leaves the company and everyone is lost. Bugs take forever to figure out, adding anything is difficult, and the new engineer tasked with taking over widgets ends up unhappy and either pushes for a re-write (sound familiar?) or is miserable and also leaves. Pairing and regular rotations between pairs completely avoid this issue by allowing each team member to see the evolution of widgets, dongles, and plugs over time. Nobody is a silo. Nobody inherits terrible code.

WSJ: This is about running a good business. All businesses have people with quirks. Where’s the news in that? So long and thanks for all the fish (that go bad after 3 days)!

Countless other benefits

There are countless other benefits to pairing: fun, camaraderie, avoidance of distractions (Facebook, Twitter, etc), efficient training of junior developers, etc. The list goes on and on.

We know pair-programming isn’t for everyone, but we know it’s for us.

Posted by Roger Neel. Contact me at roger [at] mavenlink [dot] com or follow me @mavenroger