A/B testing using Django

Whenever we roll out an improvement on our platform at HackerEarth, we love to conduct A/B tests on the enhancement to understand which iteration helps our users more in using the platform in a better way. As the available third-party libraries did not quite meet our needs, we wrote our own A/B testing framework in Django. In this post, we will share a few insights into how we accomplished this.

The basics

A lot of products, especially on the web, use a method called A/B testing or split testing to quantify how well a new page or layout performs compared to the old one. The crux of the method is to show layout “A” to a certain set or bucket of users and layout “B” to another set of users. The next step is to track user actions leading to certain milestones, which would provide critical data about the “effectiveness” of both the pages or layouts.

Before we began writing code for the framework, we made a list of all the things we wanted the framework to do:

• Route users to multiple views (with different templates)
• Route users to a single view (with different templates)
• Make the views/templates stick for users
• A/B test visitors who do not have an account on HackerEarth (anonymous users)
• Sticky views/templates for anonymous users as well
• Support for A/A/B or A/B/C/D…./n/ testing (just for the heck of it!)
• Analytics

We went out to grab some pizza and beer, and we came up with this wire-frame when we got back:

A/B for Views

A/B for Templates

Getting the logic right

To begin with, we had to categorize our users into buckets. So all our users were assigned a bucket number ranging from 1 to 120. This numbering is not strict and the range can be arbitrary or per your needs. Next, we defined two constants—the first one specifies which view a user is routed to, and the second one specifies the fallback or primary view. The tuples in the first constant are the bucket numbers assigned to users. The primary view in the second constant will be used when we do not want to A/B test on anonymous users.

AB_TEST = {
        tuple(xrange(1,61)): 'example_app.views.view_a',
        tuple(xrange(61,121)): 'example_app.views.view_b',
    }

AB_TEST_PRIMARY = 'example_app.views.view_a'

Next, we wrote two decorators which we could wrap around views—one for handling views and the other for handling templates. In the first scenario, the decorator would take a dictionary of views, that is, the first constant that we defined, a primary view, that is, the second constant, and a Boolean value which specifies if anonymous users should be A/B tested as well.

Here’s what the decorator essentially does for logged in users:

• Get the user’s bucket number
• Check which view is assigned to that bucket number
• Return the corresponding view

The flow is a bit different in case of anonymous users. If we do not want to perform A/B testing on anonymous users, then we just return the primary or fallback view that we had defined earlier. However, if we want to include anonymous users in the A/B tests, we need a couple of extra things to begin with:

• Set a unique cookie for the user which is independent of the session
• A simple and fast key-value pair storage (e.g., Redis)

Once we have these things in place, here’s what we need to do:

• Get the user’s unique cookie
• Check if a key exists in Redis for that cookie value
• If a key is found, get the value of the key and return it
• If no key is found, choose a view randomly from the view dictionary
• Set a key in Redis corresponding to the user with the chosen view as value
• Return the chosen view

Now, the A/B will work perfectly for anonymous users as well. Once an anonymous user gets routed to one of the views, that view will stick for him or her.

Let’s dive into code

Here’s an example of the view decorator:

"""
Decorator to A/B test different views.
Args:
    primary_view:       Fallback view.
    anon_sticky:        Determines whether A/B testing should be performed on   
                        anonymous users as well.
    view_dict:          A dictionary of views(as string) with buckets as keys.
"""
def ab_views(
        primary_view=None,
        anon_sticky=False,
        view_dict={}):
    def decorator(f):
        @wraps(f)
        def _ab_views(request, *args, **kwargs):
            # if you want to do something with the dict returned
            # by the view, you can do it here.
            # ctx = f(request, *args, **kwargs)
            view = None
            try:
                if user_is_logged_in():
                    view = _get_view(request, f, view_dict, primary_view)
                else:
                    redis = initialize_redis_obj()
                    view = _get_view_anonymous(request, redis, f, view_dict,
                            primary_view, anon_sticky)
            except:
                view = primary_view
            view = str_to_func(view)
            return view(request, *args, **kwargs)

        def _get_view(request, f, view_dict, primary_view):
            bucket = get_user_bucket(request)
            view = get_view_for_bucket(bucket)
            return view

        def _get_view_anonymous(request, redis, f, view_dict,
                primary_view, anon_sticky):
            view = None
            if anon_sticky:
                cookie = get_cookie_from_request(request)
                if cookie:
                    view = get_value_from_redis(cookie)
                else:
                    view = random.choice(view_dict.values())
                    set_cookie_value_in_redis(cookie)
            else:
                view = primary_view
            return view

        return _ab_views
    return decorator

The noteworthy piece of code here is the function str_to_func(). This returns a view object from a view path (string).

def str_to_func (func_string):
    func = None
    func_string_splitted = func_string.split('.')
    module_name = '.'.join(func_string_splitted[:-1])
    function_name = func_string_splitted[-1]
    module = import_module(module_name)
    if module and function_name:
        func = getattr(module, function_name)
    return func

We can write another decorator for A/B testing multiple templates using the same view in a similar way. Instead of passing a view dictionary, pass a template dictionary and return a template.

Putting things together

Now, let’s assume that we have already written the A and B views which are to be A/B tested. Let’s call them “viewa” and “viewb.” To get the entire thing working, we will write a new view. Let’s call this view “view_ab.” We will wrap this view with one of the decorators we wrote above and create a new URL to point to this new view. You may refer to the code snippet below.

@ab_views(
        primary_view=AB_TEST_PRIMARY,
        anon_sticky=True,
        view_dict=AB_TEST,
        )
def view_ab(request):
    ctx = {}
    return ctx

For the sake of convenience, we require this new view to return a dictionary.

Finally, we need to integrate analytics into this framework so that we have quantifiable data about the performance or effectiveness of both the views or layouts. We decided to use mixpanel at the JavaScript end to track user behavior on these pages. You can use any analytics or event tracking tool for this purpose.

This is just one of the ways you can do A/B testing using Django. You can always take this basic framework and improve it or add new features.

P.S. If you want to experiment with an A/A/B or A/B/C testing, all you need to do is change the first constant that we defined, i.e., AB_TEST

Feel free to comment or ping us at support@hackerearth.com if you have any suggestions!

This post was originally written for the HackerEarth Engineering blog by Arindam Mani Das.