How Does HackerEarth Combat The Use Of ChatGPT And Other LLMs In Tech Hiring Assessments?

Ever since ChatGPT made a public debut in November 2022, it has been the fodder for headlines. Its popularity proves that there isn’t a single industry or vertical that will not be fundamentally reshaped by generative AI platforms in the near future. Recruiting, in general, and technical assessments, in particular, are no different.

While ChatGPT can be used in technical recruiting to make manual work more manageable, it also has a proven drawback – candidates have been using it to answer take-home coding tests during the hiring process.

Due to the growing concern around the use of generative AI in coding tests, we decided to address the topic head-on and help our users understand the measures we have put in place to detect, prevent, and manage such practices.

But first, a note about LLMs and their use cases

LLM stands for Large Language Model, a machine-learning model designed to process and generate human-like natural language. LLMs are typically built using neural networks and deep learning algorithms and trained on vast amounts of text data to learn patterns and relationships between words and phrases.

LLMs aim to generate coherent and relevant responses to natural language inputs, such as questions, statements, or commands. This makes them useful for a wide range of applications, including language translation, chatbots, content generation, sentiment analysis, and answering questions.

LLMs have become increasingly popular in recent years due to advances in deep learning algorithms and the availability of large datasets. Some of the most well-known LLMs include GPT (Generative Pre-trained Transformer), BERT (Bidirectional Encoder Representations from Transformers), and T5 (Text-to-Text Transfer Transformer).

The growing demand for LLMs has led to some burning questions. Businesses are wondering about a future where LLMs are integral to day-to-day work and can generate more profits. In the tech industry, many have welcomed LLMs like ChatGPT as an extension of the existing coding tools, and are looking at ways of integrating the platform into their coding process.

Here’s how LLMs can transform the way we interact with computers and other digital devices:

1. Language translation: You can use LLMs to automatically translate text from one language to another. This is particularly useful for businesses operating in multiple countries and trying to reach a global audience.
2. Chatbots: LLMs can help chatbots respond to customer inquiries in natural language, saving significant time and money by automating customer service tasks.
3. Content generation: Use LLMs to generate content for websites or social media. For example, an LLM could be trained to write news articles or social media posts based on a given topic.
4. Sentiment analysis: Analyze text data and determine the sentiment behind it with LLMs. This is useful for businesses looking to monitor customer feedback or social media activity.
5. Answering questions: You can leverage LLMs to answer questions in natural language. For example, an LLM could be trained to answer questions about a company’s products or services.
6. Summarization: Automatically summarize long documents or articles with LLMs. This is useful for businesses looking to quickly extract key information from large volumes of text.

So now, what is ChatGPT?

ChatGPT is a Large Language Model (LLM) based on the GPT (Generative Pre-trained Transformer) architecture. It is one of the most advanced LLMs available and is capable of generating human-like responses to natural language inputs.

ChatGPT is trained on vast amounts of text data and uses a deep learning algorithm to generate responses to user inputs. It can engage in conversations on a wide range of topics and is capable of providing contextually relevant and coherent responses.

One of the key advantages of ChatGPT is its ability to generate natural language responses in real time. This makes it a useful tool for a variety of applications, including chatbots, virtual assistants, and customer service platforms.

OpenAI, a leading AI research organization, developed ChatGPT. It is based on the GPT-3 architecture, which was trained on a massive dataset of over 45 terabytes of text data. Overall, ChatGPT represents a significant advancement in the field of Natural Language Processing and has the potential to transform the way we interact with computers and other digital devices.

It is a powerful tool that is being used in a variety of applications and has the potential to drive innovation and growth across a spectrum of industries.

How to use ChatGPT for answering coding tests?

Many developers use this tool to generate code snippets to solve specific problems in coding tests. If they can define their parameters and conditions, ChatGPT can produce a working code that can be used in the functions.

ChatGPT can answer complex technical questions which are both theoretical and practical. However, one of the shortcomings of ChatGPT is that it is not yet fully capable of answering questions based on logical reasoning. It interprets the question literally instead of contextually. This means that ChatGPT can also not answer context-based questions accurately.

ChatGPT works well when answering technical questions that are theoretical. It has been trained rigorously on that database. Even with easy coding questions, ChatGPT provides excellent results but with complex scenario-based questions, it fails to provide the right solution sometimes. It is not yet able to create complete modules for a full-stack question.

Also read: 8 Unconsciously Sexist Interview Questions You’re Asking Your Female Candidates

ChatGPT + Coding tests — Plagiarism or Progress?

The bottom line is: ChatGPT will make it infinitely easier for candidates to generate code and ace their take-home assignments. Currently, this capability is limited to simple, theory-based questions. However, the platform will inevitably learn and get better at generating complex code. Consequently, it could be used to answer all coding tests.

At HackerEarth, we have always maintained that skills are the only criteria for evaluation. However, a developer using an AI tool to answer a question muddles the selection and evaluation process.

The AI-shaped elephant in the room then begs us to pick a side. Either we conclude that the use of any generative AI by a candidate in a coding test amounts to plagiarism and is unacceptable. Or, we chalk it up to changing times and get on board with the progress.

The first approach

This is best suited for mass hiring drives, where recruiters are hard-pressed to curate a pool of candidates through a process of elimination. Plagiarism via ChatGPT in hiring assessments can be one of the criteria for elimination. It allows you to narrow your candidate list down to the developers who answered the coding test without the support of an external tool.

The second approach

This works well when hiring fewer candidates, perhaps for a highly technical role. ChatGPT is here to stay; senior developers use it to generate or evaluate complex code. Allowing candidates for such roles to use ChatGPT in coding tests would mean expanding the understanding of skill-based evaluation in these scenarios.

We could draw a parallel between these candidates and writers who use a spellchecker to proofread their assignments. AI-based writing assistants have become an industry-wide best practice, so the writer in this example would not lose any points for using one.

Instead, they would be evaluated on their research and analytical skills or creativity – which an AI–based writing assistant cannot substitute – and not necessarily on their use of an external tool. In theory, one could use the same rationale to justify and accept the use of ChatGPT in hiring assessments by candidates.

Given both these approaches, we at HackerEarth have decided to support both schools of thought in our Assessments platform. Those who want to ensure their candidates cannot use ChatGPT for answering tests can do so with our advanced proctoring features. And the hiring managers who do not mind the use of ChatGPT can write to support@hackerearth.com to understand how the LLM can be integrated into HackerEarth Assessments.

How does HackerEarth detect the use of ChatGPT in hiring assessments?

With the increasing use of ChatGPT, many of our customers have written to us to ask how we plan to combat the use of ChatGPT in hiring assessments. HackerEarth Assessments is known for its robust proctoring settings. We have added new features to detect the recent spate of plagiarism via ChatGPT in hiring assessments.

Let me walk you through these new additions:

1. Smart Browser

HackerEarth has introduced new advanced proctoring features including a Smart Browser. This is available with the HackerEarth Assessments desktop application. This builds on our existing proctoring features and establishes a highly rigorous proctoring method to prevent the use of ChatGPT and other LLMs.

Smart Browser includes the following settings that detect the use of ChatGPT:

• Candidates are not allowed to keep other applications open during the test
• They are also not allowed to:
  • Resize the test window
  • Use multiple monitors during the test
  • Share the test window
  • Take screenshots of the test window
  • Record the test window
  • Use restricted keystrokes
  • View OS notifications
  • Run the test window within a Virtual Machine
  • Use browser developer tools

To learn more about the Smart Browser, read this article.

At the time of writing this article, Smart Browser is only available upon request. To request access, please get in touch with your Customer Success Manager or contact support@hackerearth.com.

Also read: 3 Things To Know About Remote Proctoring

2. Tab switch proctoring setting

Use HackerEarth’s tab switch proctoring setting during tests. This setting allows you to set the number of times a candidate can move out of the test environment. The default setting is for 5 instances, which means that candidates are allowed to switch tabs 5 times during the test duration. On the 6th try, they will be automatically logged out of the system. The default number can be changed if required.

When this proctoring setting is enabled, the system warns the candidate each time they move out of the test environment. The following actions are considered as ‘moving out of the test environment. However, please note that this is not an exhaustive list:

• Switching tabs
• Switching windows
• Opening new applications on the computer, including system popups like anti-virus notifications, Lync notifications, Skype notifications, etc.
• Any action taken to close notifications is also counted as leaving the test environment.

The assumption is that candidates would need to switch tabs to access ChatGPT. By not allowing candidates to move out of the test environment beyond a set number of times, we can detect and prevent the use of ChatGPT.

3. Full-screen proctoring setting

Enable this feature to enhance the proctoring of a hiring assessment and allow your candidates to take the assessment only in a full-screen mode. As soon as the candidate opens up the assessment, the screen goes into full screen and candidates cannot exit this mode. If they try to exit the mode, they will be logged out of the assessment.

Reduce ChatGPT usage in your assessments by not allowing candidates to open any new tabs while giving the assessment. To learn more about HackerEarth’s proctoring settings, read this article.

4. Diverse question types

HackerEarth has a rich library of logical reasoning questions that cannot be answered easily via ChatGPT. We tested our questions on ChatGPT, and we can say with reliable accuracy that it cannot answer logical reasoning questions correctly because it cannot understand contextual questions.

Here’s one of the many examples of logical reasoning questions that we asked ChatGPT to test its capabilities:

ChatGPT cannot produce code for full-stack questions. HackerEarth has a vast library of full-stack questions that can be used in the assessments and are well protected from the impact of ChatGPT.

While ChatGPT can help write the code for some modules, it cannot fully answer a full-stack question with all the functions. Compiling these separate functions to create a single module requires skill and ingenuity.

Similarly, recruiters can use file upload questions to make their assessments more robots. These questions have complex scenarios and functions that ChatGPT cannot answer completely.

Essential insights about using ChatGPT in hiring

• LLM stands for Large Language Model. It is a type of machine learning model designed to process and generate human-like natural language.
• ChatGPT is a Large Language Model (LLM) based on the GPT (Generative Pre-trained Transformer) architecture. It is one of the most advanced LLMs available and is capable of generating human-like responses to natural language inputs.
• You can use ChatGPT to answer easy coding questions and MCQs. It can help write accurate code snippets for function modules.
• Recruiters, avoid using MCQs with direct answers as candidates can easily answer them through ChatGPT.
• HackerEarth provides various solutions that help us detect and prevent the usage of ChatGPT. These include:
  • Smart browser
  • Tab switch proctoring setting
  • Full-screen mode
  • Diverse and complex question types

Doing away with using ChatGPT in hiring assessments

In many ways, we are all just waking up to the power of AI. With new advancements every day, no one is sure what the future will unfold, but we all should be ready to embrace the moment when AI becomes an integral part of daily functions.

Technical assessments can still be curated without the interference of AI platforms like ChatGPT to ensure skill-first evaluation. HackerEarth Assessments has introduced advanced proctoring settings like Smart Browser, tab-switch detection, full-screen mode, and a vast library of complex engineering questions that are not easily answerable by ChatGPT.

The product mavens at HackerEarth work relentlessly to ensure our product is firewalled against the latest challenges and developments. Tech recruiters and hiring managers can rest assured that the validity and sanctity of our assessments haven’t been affected by the use of ChatGPT.

We will keep a keen eye on upcoming changes in this area and improve the product over time to combat future challenges and ensure a plagiarism-free hiring experience for our clients.

Frequently Asked Questions (FAQs)

#1 Where does HackerEarth see pre-interview tests and interviewing to be moving to in a world where ChatGPT exists?

The world of interviewing and pre-interview tests will see significant changes in the foreseeable future. We also need to understand that as new features and platforms emerge, the solutions to detect and prevent their use will go through multiple iterations.

In the near future, advanced proctoring settings and new question types that are not easily answerable using ChatGPT can help protect pre-interview tests from the impact of ChatGPT. We are also working on foolproof methods for plagiarism detection, which can circumnavigate ChatGPT’s upgrades.

#2 With ChatGPT being able to solve MCQs, programming, etc. in a few minutes, does HackerEarth have a different set of problems that can be used? 

ChatGPT can quickly solve MCQs and simple programming problems, which is a big concern. However, HackerEarth has a wide variety of questions that recruiters can use to combat the usage of ChatGPT. We have a library of full-stack question types. As previously discussed, it will be difficult for a candidate to search for different modules and compile them to complete the question. It is a time taking and complex process, so candidates will prefer to do these questions on their own.

Moreover, ChatGPT cannot understand logical real-life scenarios. The accuracy of such answers is poor. Use a mix of logical reasoning MCQs, DevOps, and Selenium questions to check the versatility of a candidate.

Shivam Gupta Author 3 Blogs