Science series materials are brought to you by TestGorilla’s Assessment Team: A group of IO psychology, data, and psychometric experts with a deep understanding of the science behind skills-based hiring.
TestGorilla takes a deeply scientific and multi-layered approach to creating tests for skills-based hiring. This series of brief introductions aims to conjure transparency when it comes to how we build our tests and how we advise they are used.
In this post, we’ll introduce you to test order and offer some test order best practices that will help you increase the reliability of your assessments.
When we refer to test order, we’re talking about the order in which tests are placed within an assessment – where an assessment is made up of two to five different tests and/or custom questions.
Note that this is specifically how we understand tests and assessments at TestGorilla. (It’s also common to use the terms interchangeably, or to understand a test as a type of assessment where there are correct and incorrect answers.)
When you’re choosing the order of tests in an assessment that you’re creating, it’s important to consider what the candidate is going through. Taking tests is hard – it’s very cognitively demanding and requires a lot of both deep and time-bound thinking, especially when the test is timed.
Because of this, the order in which tests are taken by candidates can have an impact on their assessment-taking experience and overall performance. But it’s difficult for people who don’t have expertise in testing to know how to properly order tests within an assessment.
Without this guidance hiring managers can make suboptimal decisions about test order, resulting in a worse experience for both the candidates and themselves.
Despite test order’s importance, we don’t expect employers, who aren’t experts in testing or assessment psychology, to know how to do it properly. The purpose of this science series article is to support you with test ordering expertise, so that you can improve your test reliability and candidate experience.
Assessment scientists use the term “fatigue effect” to refer to the cognitive demand that makes taking tests hard – the longer an assessment goes on, the more fatigued the candidate will get. Not only is enhanced fatigue a poor experience for the candidate; it can also worsen the psychometric properties of a test. In other words, when the fatigue effect is in play, the data you get about the candidate is noisier and less reliable.
Because of this, there is a broad consensus among assessment scientists in support of two (very simple!) test order best practices:
More cognitively demanding tests should be placed at the start of an assessment, when fatigue effect is low. The impact of this is that candidates can spend their mental energy on the more difficult tests when they are fresh and energized.
Intuitively, less cognitively demanding tests should be placed at the end of an assessment, when fatigue effect is highest. The impact of this is that when candidates begin to get tired as a result of taking tests, they are working on easier tests that require less concentration.
Our Problem-solving test has been used hundreds of thousands of times in assessments created by a multitude of employers. Due to the widely-applicable nature of this test, assessments that include it have been used to hire for a wide range of job roles. Customers can, and do, place this test anywhere from first to fifth in the assessments they create.
The Problem-solving test is a hard test for candidates. It’s challenging and cognitively demanding, and requires that candidates demonstrate substantial problem-solving skills in a short amount of time (10 minutes). In short, it’s exactly the kind of test that, as best practices suggest, should be placed at the start of an assessment.
We gathered some data in order to understand the impact that test order has on the performance of our Problem-solving test – where ‘performance’ is determined by the reliability of the test (where reliability is a statistical measure of how consistent candidate scores are). Below is a summary of our findings.
Note that when considering the reliability of a test for use in hiring, a reliability score of 0.70 is the minimum acceptable standard, with higher values being more desirable.
Where was the Problem-solving test placed within the assessment?
What was the reliability score of the test under this placement?
What percentage of Problem-solving tests appear in this order?
1st test in assessment
2nd test in assessment
3rd test in assessment
4th test in assessment
5th test in assessment
As you can see, when the Problem-solving test is placed first in an assessment, it has a reliability estimate of 0.772. This score sits nicely above the minimum acceptable score of 0.7.
As the test is placed further and further along in the test order of an assessment, its reliability score steadily decreases. By the fifth slot, it barely meets the minimum accepted standard for test reliability. Test order really does make a difference.
Our data also shows that the majority of people using this test place it within the first three tests of an assessment. This shows there is already some intuitive understanding among TestGorilla users that our Problem-solving test should appear early on in an assessment.
Being intentional with test order can help you make the most of your TestGorilla assessments. It’s low-hanging fruit: by switching your test order around slightly, you can improve the candidate experience and the reliability of your test, enabling you to make better hiring decisions.
The best way to develop your understanding of test order is by simply trying out the tests for themselves and seeing what feels hardest.
Place the ones that you find toughest at the start of your assessment, and move the easier ones to the end. You’ll end up with a more reliable assessment, and your candidates will be happy you took that additional step. Why not experiment with some of our tests for free by signing up for a free plan?
You can check out additional blogs about the science behind TestGorilla here:
To address its increased recruitment needs and influx of applicants for roles that include customer support and leadership, Dyninno Group implemented TestGorilla. See how the Dyninno Group of companies improved candidate screening and recruitment productivity by 400%.
Create pre-employment assessments in minutes to screen candidates, save time, and hire the best talent.
No spam. Unsubscribe at any time.
Our screening tests identify the best candidates and make your hiring decisions faster, easier, and bias-free.
This checklist covers key features you should look for when choosing a skills testing platform
This resource will help you develop an onboarding checklist for new hires.
How to assess your candidates' attention to detail.
Learn how to get human resources certified through HRCI or SHRM.
Learn how you can improve the level of talent at your company.
Learn how CapitalT reduced hiring bias with online skills assessments.
Learn how to make the resume process more efficient and more effective.
Improve your hiring strategy with these 7 critical recruitment metrics.
Learn how Sukhi decreased time spent reviewing resumes by 83%!
Hire more efficiently with these hacks that 99% of recruiters aren't using.
Make a business case for diversity and inclusion initiatives with this data.