An open letter to our users

We would like to convey our deepest apology to all the participants of InMobi Hiring Challenge. We messed up this one, we realize it and we regret it. Servers shot to a 100% CPU utilization, page loads were excruciatingly slow and the random 500s in the first 20 mins screwed what we had expected to be a smooth challenge. It was our fault, we miscalculated. We thought that 20 large instances would be enough to support the load, but we learned it the hard way when there are more than 600 people simultaneously submitting code, you need much more server bandwidth than that. However, it’s fortunate that we had invested time in scripts to auto launch server instances on AWS and were able to immediately fire up more instances. But as soon as we did that, we realized that database was going to become a bottleneck. Though we were able to resolve all  issues within the first 30 mins, it was still disheartening to see people trying to access the event page and not being able to do it.

We have started HackerEarth with a vision of changing the way recruitment is done in the industry today. We want to eliminate all kind of bias in the hiring process and let the code speak over all the big talk. We are building a platform where every developer has an equal chance to apply to companies with the skill they best know i.e. coding. But we fell short of our promise this Sunday. We want to make a developer’s code his/her identity, we want you to write code on our platform and we want you to have absolute fun while you do that.

Ever since this crisis happened, we have invested all our resources in making sure that something like this never happens again. It’s not that our platform cannot deliver, it’s just that our system was not intelligent enough to anticipate the load and scale accordingly. We are young, we code fast and deploy faster, but amidst this, we did a mistake that we did not invest in testing. We have never stressed our system, never tried to understand under what load it would break down, but we are doing that now.

It hurt us badly to read the responses of our unhappy users, but we are in a way thankful to each and every one of you who tweeted us, reached out to us on Facebook and emailed us. Though we wish, we had not made so many of you angry, but we are glad we got to hear you. It has helped us realize what are the shortcomings and strive harder to make our platform even more robust and failure proof. We also express our sincere apologies for the inconvenience caused to our stakeholders.

Thanks for taking the time to read this.