Test Your Screening Rubric Like an Experiment

How I’d check a rubric change before it ships, using candidates your team has already decided on.

Say you’re hiring an operations coordinator, and your rubric asks for three years of operations experience. A candidate your recruiter liked comes back with a 4 out of 10. She has run a retail store for years, but the rubric didn’t count any of that as operations. So your recruiter adds a line that gives credit for related experience and scores her again. This time she gets an 8, and the change ships that afternoon.

I wouldn’t ship it yet. Maybe the rubric was too strict with her. It’s also possible the rubric was right and your recruiter liked her for reasons the role doesn’t need, and one candidate can’t tell you which. The 8 doesn’t settle it either. It came from a single run on the one person the new line was written for, so it would be strange if her score hadn’t gone up.

Why one good run proves little

That line was written for one person, but it applies to every candidate the rubric scores for the role, and nobody has checked what it does to them. It might lift a few people who deserved more credit, or a lot of people who didn’t. If the shortlist gets longer and weaker, I doubt anyone will connect it to Tuesday’s edit. Someone just tightens the rubric to shorten the list, and the store manager drops back to a 5.

I call this edge-case hell. You fix the rubric for one candidate, the fix moves other candidates, and then one of them needs a fix of their own. A few rounds in, nobody can explain why the rubric scores the way it does anymore. The people who applied in March were judged by a different rubric than the people who applied in May.

I spent four years building search systems, which rank results for a query much the way a rubric ranks candidates for a role. Working on search taught me to treat every change to a ranking as an experiment. If I’d only checked a change on one example, I didn’t count it as tested.

If an AI tool does the scoring, ask how consistent it is before you test a change. Tools differ, and you can find out by scoring the same candidates twice and comparing. If yours does vary, a test needs repeat runs before it can tell a real improvement from ordinary variation. Either way, a person should still review borderline candidates, whatever the score says.

Run each change many times

Start with a fixed set of past candidates for one role, people your team has already made a call on. I’d pick about 50, roughly half of them people your recruiters moved forward, and leave out the candidate who started all this. The change was written for her, so her score will almost always rise. Check her on her own. Keep that set and reuse it, so every change gets tested on the same people as the last one.

Score every candidate three times with the current rubric and three times with the new one. Everyone at or above the score you’d normally advance on is the rubric’s top group. The repeat runs show you whether that group moves on its own, and by how much. If it changes by three people between two runs of the same rubric, a new version that agrees with your team on two more candidates hasn’t shown you anything yet.

Here’s a small example from Skyvern. Our agent uses a language model to work on websites, and it sends that model a description of the elements on each page. We tried writing that description as HTML instead of JSON. In a production A/B test, the success rate went from 59.9% to 63.8%, and net cost fell 11.4%. Those are August 2024 numbers.

I’d expect a good rubric change to look about the same, a few more agreements with your recruiters spread thinly across the set. No single task showed our gain. Each one worked or it didn’t.

Measure against what your recruiters decided

For each version, work out two numbers. Take the candidates your recruiters moved forward and count how many made the rubric’s top group, which tells you who the rubric misses. Then go the other way. Of everyone in the top group, count how many your recruiters actually moved forward. The rest are people your team turned down.

Then read every candidate whose result changed between the two versions, since that’s where you see what the edit really does. Where the rubric and your team disagree, don’t assume the rubric is wrong. Some of those candidates will be people your team passed on too quickly, and they deserve a second look whatever you decide about the rubric.

While you’re at it, compare how each version’s top group breaks down across the groups your ATS already tracks, such as voluntary EEO fields. A rubric that agrees with your recruiters can still repeat patterns from their past decisions, and a change that shifts those numbers is worth knowing about before it ships.

Before you run any of this, write down what counts as better. It’s very easy to look at the numbers first and then pick a bar the change already clears. For the coordinator rubric, I’d want the new version to agree with your recruiters on more candidates than the current one, by more than the run-to-run spread from your repeat runs. I’d also want it to keep everyone they moved forward.

If the new line misses that bar, keep the rubric you have. It was already doing the job. Run this way, a rubric changes rarely and only with a test result behind each change, so a 7 still means what it meant last month. If your rubric is working, this test mostly tells you to leave it alone.

That leaves the store manager. If her 4 still looks off to your recruiter, have someone review her and put a note in a log. I wouldn’t change the rubric over one note. If the same kind of miss shows up in that log a few times, then I’d write a change and run it through this same test.

If you want to start this week, start with the test set. Pull about 50 past candidates for one role your team has already made a call on, and mark who moved forward.

Suchintan Singh is the CEO and co-founder of Skyvern (YC S23). Before Skyvern, he spent 4 years building search systems at Faire and Gopuff.