AI in hiring: the questions to put to a vendor
AI in hiring is a category where the claims are unusually far ahead of the mechanisms, and where the cost of believing a claim is unusually high — you are making decisions about people's livelihoods, and in a growing number of jurisdictions you are accountable for how those decisions were reached.
These are the questions worth putting to any vendor, including us. The pattern that matters is not which answer you get but whether the vendor can answer specifically at all.
1. What exactly does the model score, and against what?
"It evaluates candidates" is not an answer. You want to know the dimensions, the inputs to each, and how they combine. If a vendor cannot break the score into named components with named inputs, either it is a black box or they do not know — and you cannot defend either to a rejected candidate.
A good answer sounds mechanical: these five pillars, from these inputs, blended with these weights, redistributed this way when a round does not apply.
2. Can a candidate be rejected without a human deciding?
The answer you want is no. Fully automated rejection is a legal exposure in a growing number of places and an unforced error everywhere else.
Follow it up: is the human decision recorded, with who and when? A process where a person nominally decides but nothing records it is automated rejection with extra steps.
3. What has been measured, and on what sample?
This is the question that separates real evaluation from marketing. If a vendor claims accuracy, ask what was measured, against what ground truth, on how many cases, and whether you can see the methodology.
An honest vendor may well say their evaluation set is still small. That is a far better answer than a confident round number, and you should be actively suspicious of any accuracy figure quoted without a sample size. We removed exactly such a number from our own homepage in August 2026 because it was not backed by measurement — it is a very easy claim to end up making.
4. What happens when it is wrong?
Every scoring system is wrong sometimes. What matters is whether the product is built as though it might be. Can a recruiter see the reasoning and disagree with it? Is a score presented as evidence or as a verdict? Is there any route by which a candidate scored poorly still gets seen?
A vendor who has thought about being wrong has built review paths. One who has not will tell you it is very accurate, which is the answer to a different question.
5. What integrity signals are collected, and what happens when they cannot be?
For remote assessment, ask exactly which signals are captured — and then ask what happens on a browser or device that cannot emit one.
The right answer is that an unavailable signal is excluded from scoring. The wrong answer, which is common, is that a missing signal is scored as though nothing was detected, which quietly advantages candidates on particular hardware. That asymmetry is invisible in a demo and unfair at scale.
6. Where does candidate data live, and for how long?
Candidate data is personal data of people who mostly will not be hired by you and never agreed to indefinite retention. Ask where it is stored, how tenant isolation works between customers, what is retained after a rejection, and what deletion actually does.
Isolation enforced by the database is meaningfully different from isolation enforced by every query being written correctly. Ask which it is; the answer is usually specific and revealing.
7. What does it not do?
Ask it directly and listen to how readily it is answered. A vendor who names real limitations quickly is describing a product they know. One who cannot think of any is describing a brochure.
Write the answer down. A limitation acknowledged before purchase is a very different conversation from one you discover in month four.
The takeaway
The signal is specificity, not enthusiasm. A vendor who can name the scoring dimensions, the measurement they have actually done, what happens when the model is wrong, and what their product does not do is describing something real. Confidence without those details is the thing to walk away from.
Questions we get asked
Is AI interviewing legal in India?
There is no prohibition, but data-protection obligations under the DPDP framework apply to candidate data, and you remain accountable for how decisions are made. Get advice for your specific situation rather than relying on a vendor's reassurance.
Should we tell candidates AI is involved?
Yes. It is increasingly required in some jurisdictions, it is straightforwardly the right thing to do, and candidates who are told tend to perform more honestly and complain less.
What is the single most revealing question?
"What has been measured, and on what sample?" It is the one that cannot be answered with enthusiasm.
See HireOrbyt on a real role
A 30-minute walkthrough against a position you are actually hiring for. No slide deck.