Hire top tech talent with our recruitment platform
Access Free DemoA/B testing using Django
Whenever we roll out an improvement on our platform at HackerEarth, we love to conduct A/B tests on the enhancement to understand which iteration helps our users more in using the platform in a better way. As the available third-party libraries did not quite meet our needs, we wrote our own A/B testing framework in Django. In this post, we will share a few insights into how we accomplished this.
The basics
A lot of products, especially on the web, use a method called A/B testing or split testing to quantify how well a new page or layout performs compared to the old one. The crux of the method is to show layout “A” to a certain set or bucket of users and layout “B” to another set of users. The next step is to track user actions leading to certain milestones, which would provide critical data about the “effectiveness” of both the pages or layouts.
Before we began writing code for the framework, we made a list of all the things we wanted the framework to do:
- Route users to multiple views (with different templates)
- Route users to a single view (with different templates)
- Make the views/templates stick for users
- A/B test visitors who do not have an account on HackerEarth (anonymous users)
- Sticky views/templates for anonymous users as well
- Support for A/A/B or A/B/C/D…./n/ testing (just for the heck of it!)
- Analytics
We went out to grab some pizza and beer, and we came up with this wire-frame when we got back:
A/B for Views
A/B for Templates
Getting the logic right
To begin with, we had to categorize our users into buckets. So all our users were assigned a bucket number ranging from 1 to 120. This numbering is not strict and the range can be arbitrary or per your needs. Next, we defined two constants—the first one specifies which view a user is routed to, and the second one specifies the fallback or primary view. The tuples in the first constant are the bucket numbers assigned to users. The primary view in the second constant will be used when we do not want to A/B test on anonymous users.
AB_TEST = {
tuple(xrange(1,61)): 'example_app.views.view_a',
tuple(xrange(61,121)): 'example_app.views.view_b',
}
AB_TEST_PRIMARY = 'example_app.views.view_a'
Next, we wrote two decorators which we could wrap around views—one for handling views and the other for handling templates. In the first scenario, the decorator would take a dictionary of views, that is, the first constant that we defined, a primary view, that is, the second constant, and a Boolean value which specifies if anonymous users should be A/B tested as well.
Here’s what the decorator essentially does for logged in users:
- Get the user’s bucket number
- Check which view is assigned to that bucket number
- Return the corresponding view
The flow is a bit different in case of anonymous users. If we do not want to perform A/B testing on anonymous users, then we just return the primary or fallback view that we had defined earlier. However, if we want to include anonymous users in the A/B tests, we need a couple of extra things to begin with:
- Set a unique cookie for the user which is independent of the session
- A simple and fast key-value pair storage (e.g., Redis)
Once we have these things in place, here’s what we need to do:
- Get the user’s unique cookie
- Check if a key exists in Redis for that cookie value
- If a key is found, get the value of the key and return it
- If no key is found, choose a view randomly from the view dictionary
- Set a key in Redis corresponding to the user with the chosen view as value
- Return the chosen view
Now, the A/B will work perfectly for anonymous users as well. Once an anonymous user gets routed to one of the views, that view will stick for him or her.
Let’s dive into code
Here’s an example of the view decorator:
"""
Decorator to A/B test different views.
Args:
primary_view: Fallback view.
anon_sticky: Determines whether A/B testing should be performed on
anonymous users as well.
view_dict: A dictionary of views(as string) with buckets as keys.
"""
def ab_views(
primary_view=None,
anon_sticky=False,
view_dict={}):
def decorator(f):
@wraps(f)
def _ab_views(request, *args, **kwargs):
# if you want to do something with the dict returned
# by the view, you can do it here.
# ctx = f(request, *args, **kwargs)
view = None
try:
if user_is_logged_in():
view = _get_view(request, f, view_dict, primary_view)
else:
redis = initialize_redis_obj()
view = _get_view_anonymous(request, redis, f, view_dict,
primary_view, anon_sticky)
except:
view = primary_view
view = str_to_func(view)
return view(request, *args, **kwargs)
def _get_view(request, f, view_dict, primary_view):
bucket = get_user_bucket(request)
view = get_view_for_bucket(bucket)
return view
def _get_view_anonymous(request, redis, f, view_dict,
primary_view, anon_sticky):
view = None
if anon_sticky:
cookie = get_cookie_from_request(request)
if cookie:
view = get_value_from_redis(cookie)
else:
view = random.choice(view_dict.values())
set_cookie_value_in_redis(cookie)
else:
view = primary_view
return view
return _ab_views
return decorator
The noteworthy piece of code here is the function str_to_func(). This returns a view object from a view path (string).
def str_to_func (func_string):
func = None
func_string_splitted = func_string.split('.')
module_name = '.'.join(func_string_splitted[:-1])
function_name = func_string_splitted[-1]
module = import_module(module_name)
if module and function_name:
func = getattr(module, function_name)
return func
We can write another decorator for A/B testing multiple templates using the same view in a similar way. Instead of passing a view dictionary, pass a template dictionary and return a template.
Putting things together
Now, let’s assume that we have already written the A and B views which are to be A/B tested. Let’s call them “viewa” and “viewb.” To get the entire thing working, we will write a new view. Let’s call this view “view_ab.” We will wrap this view with one of the decorators we wrote above and create a new URL to point to this new view. You may refer to the code snippet below.
@ab_views(
primary_view=AB_TEST_PRIMARY,
anon_sticky=True,
view_dict=AB_TEST,
)
def view_ab(request):
ctx = {}
return ctx
For the sake of convenience, we require this new view to return a dictionary.
Finally, we need to integrate analytics into this framework so that we have quantifiable data about the performance or effectiveness of both the views or layouts. We decided to use mixpanel at the JavaScript end to track user behavior on these pages. You can use any analytics or event tracking tool for this purpose.
This is just one of the ways you can do A/B testing using Django. You can always take this basic framework and improve it or add new features.
P.S. If you want to experiment with an A/A/B or A/B/C testing, all you need to do is change the first constant that we defined, i.e., AB_TEST
Feel free to comment or ping us at support@hackerearth.com if you have any suggestions!
This post was originally written for the HackerEarth Engineering blog by Arindam Mani Das.
Get advanced recruiting insights delivered every month
Related reads
Top 8 Sourcing Tools for Recruiters: A Comprehensive Guide
In today’s competitive talent landscape, attracting top candidates requires going beyond traditional job board postings. This is where effective sourcing tools comes into…
Benefits of Technical Interview Outsourcing for Growing Companies
With growth, recruiting the best technical talents becomes one of the most important, but also the hardest, processes. Screening technical candidates requires time,…
Enterprise Recruitment – Process & Challenges
In recent years, recruitment practices have changed tremendously. As the times advanced, organisations took numerous steps towards adopting technology-based recruitment, addressing the various…
Leveraging Recruitment Metrics to Improve Hiring Decisions
Today’s job market is very competitive. Organizations must adopt data-driven approaches to amplify their recruitment efforts to stay afloat in the face of…
The Impact of Talent Assessments on Reducing Employee Turnover
Organizations of all industries struggle with employee turnover. The high turnover rates cause increased hiring costs, lost productivity, and broken team dynamics. That’s…
Pre-Employment Assessment Testing – The Complete Guide
Candidate assessment is a major part of the hiring process. The talent acquisition system emphasizes conducting pre-employment assessment testing to derive quality results….