How do you hire in the age of AI? Test judgment, not tool knowledge

What Canva, Meta, Anthropic, Amazon and Google now do in interviews, what the research says predicts job performance, four AI-allowed work-sample tasks with a planted mistake, how much cheating detection can really catch, and the rules in New York City, Illinois and the EU.

By Andrej Ruckij · · 19 min read
How do you hire in the age of AI? Test judgment, not tool knowledge

How do you hire in the age of AI? Test judgment, not tool knowledge

By Andrej Ruckij · October 30, 2026 · updated November 4, 2026

TL;DR: Stop testing what AI can already do for the candidate. Give them a real task with AI allowed, put one confident mistake in the AI material they start from, then spend 20 minutes having them explain what they kept, changed and checked, scored on a written scale. The task alone isn’t enough: in a 2022 re-analysis of hiring research by Paul Sackett and colleagues, structured interviews predicted job performance better than work samples (.42 against .33), so the scored debrief matters as much as the task. Keep one short check without AI, because people who hand everything to AI understand less of what they produce. Canva and Meta now run AI-allowed interviews; Amazon and Google went the other way. No detector I found has an independently measured error rate, so design the process so hidden AI use doesn’t help. If software screens or scores candidates for you, New York City, Illinois and, from December 2027, the EU regulate it.

What changed

Felix Rieseberg, an engineering manager at Anthropic who has led engineering for Claude.ai, Cowork and Claude Code Desktop, wrote on his personal blog on October 27 that he stopped giving coding interviews: “As a hiring manager at Anthropic, I stopped asking candidates to do them.” His reasoning: being able to quickly define problems and find elegant solutions used to be a weak but useful predictor of success on his teams, and that’s no longer true: “There’s no more point in giving you a little React challenge than there is in quizzing people about assembly; Claude is now better at both.” He now scans resumes and portfolios “for little prototypes that exhibit taste, empathy, and principled thinking”, because “Execution isn’t expensive anymore.”

That’s one manager’s practice, not an Anthropic policy, and he says openly that he’s still working it out. Anthropic’s own candidate guidance (last updated July 2025) still says of live interviews: “This is all you–no AI assistance unless we indicate otherwise.”

But the shift is real, and companies have split into two camps.

CompanyWhat they do nowReported by
CanvaPiloted an “AI-Assisted Coding” round in place of its computer-science fundamentals screen for backend and frontend roles; it says it still assesses fundamentals “through the new process”. Candidates “are expected to use their preferred AI tools”.Canva engineering blog, June 2025
MetaFrom October 2025, for software engineer and engineering manager roles up to E7 and M2, one of two onsite coding rounds allows an AI assistant (several models to pick from). The other round stays classic, with no AI.Hello Interview prep guide (secondary)
AnthropicTake-homes: “Complete these without Claude unless we indicate otherwise.” Live interviews: no AI “unless we indicate otherwise”. Its performance-engineering take-home explicitly allows AI; when Claude matched the best human scores, the team redesigned the task instead of banning AI.Anthropic candidate guidance; Anthropic engineering blog, Jan 2026
Amazon”please do not use GenAI tools during your interview”, unless explicitly permitted; breaking the rule “may result in disqualification”.Internal guidelines reported by Business Insider, via IT Pro, Mar 2025
GoogleReintroducing “at least one round of in-person interviews for people”, per Sundar Pichai in June 2025.Entrepreneur, Aug 2025
McKinseyAsks hiring managers to hold at least one in-person meeting before an offer.Entrepreneur, citing the Wall Street Journal
Two Chairs (mental health provider)Publishes a hiring AI policy: AI allowed to polish a resume, banned in live interviews, allowed in take-homes only if the hiring manager approves in advance; breaking it means disqualification.Two Chairs policy page, Aug 2026

For a smaller company, the Two Chairs page is the most useful template I found: one rule per stage, and a line saying “all hiring decisions” are made by its team.

Why I care: the risk is people who trust AI too much

When AI does the drafting, the coding and the first analysis, the scarce skill moves to the step after it: deciding whether the output is right and what to do with it. Execution gets cheap and judgment keeps its value (glossary/automation-eats-execution).

A lot of people skip that step, and three studies show why it matters for hiring:

  • Trust in AI replaces thinking. Microsoft Research and Carnegie Mellon surveyed 319 knowledge workers about 936 first-hand examples of using AI at work (CHI 2025). In their self-reports, people with more confidence in GenAI reported less critical thinking, and those with more confidence in their own skill reported more. It’s an association from a survey, not an experiment. Their summary: AI “shifts the nature of critical thinking toward information verification, response integration, and task stewardship.”
  • AI makes people faster at the wrong answer. In a Harvard and BCG field experiment with 758 consultants, those using AI on tasks inside its abilities completed 12.2% more tasks, 25.1% faster. On a task outside its abilities, they were 19% less likely to produce a correct solution. As one of the researchers put it, “getting done faster to the wrong answer in many cases is not ideal.”
  • Handing everything to AI leaves people understanding less. In Anthropic’s randomized trial (January 2026) with 52 mostly junior engineers learning a new Python library, the group with an AI assistant scored 50% on a follow-up quiz against 67% for those who coded by hand, and the gap was widest on debugging. How people used the AI mattered: those who asked it to explain concepts or asked follow-up questions scored 65% or more, while those who delegated the work to it scored below 40%. The sample is small and it measured understanding right after one session, so treat it as a direction rather than a final number.

So polished work done with AI tells you very little, because AI makes polish cheap for everyone. What you want to know is whether the person notices when it’s wrong and understands what they’re handing in. More on where AI can and can’t be trusted: When can you trust AI? and Does AI help beginners or experts more?

What the research says predicts job performance

This section matters for how you build the process. The usual numbers came from Schmidt and Hunter (1998). In 2022, Paul Sackett and colleagues re-ran the numbers in the Journal of Applied Psychology and found that the corrections used in the Schmidt and Hunter (1998) line of work over-corrected for range restriction, inflating the estimates. Their revised validities, as summarised by HumRRO researchers in SIOP’s journal TIP:

MethodRevised validity (higher predicts better)
Structured interview.42
Job knowledge test.40
Empirically keyed biodata.38
Work sample test.33
Cognitive ability test.31
Interests (matched to the job).24

Two things follow for this article. First, a work sample on its own is good but not the best tool you have; a structured interview (the same questions for everyone, scored on a written scale) came out on top. Second, these numbers come from studies run before AI assistants existed. I found no peer-reviewed study yet on how well an AI-allowed work sample predicts later job performance. Canva says its new round gives “strong predictive signals” about candidate performance, but it published no figures.

So my version combines both: the AI-allowed task gives you something real to talk about, and a scored, structured debrief on that task is where most of the signal comes from.

What I’d expect people to know and do now

  • Use AI fluently for the routine part of the work. In my view, refusing to use it now costs the team time.
  • Know where it breaks. Common, well-documented tasks are usually fine; novel or high-stakes ones need everything checked.
  • Check before they ship, and own the decision. “The AI said so” isn’t a reason.
  • Show initiative. Something they built on their own because they saw a problem, the kind of small prototype Rieseberg looks for.

Work-sample tasks that show judgment with AI

The task design borrows from three companies that described theirs. Canva made its questions more complex, ambiguous and realistic, so that they “can’t be solved with a single prompt”, and checks “Can they identify and fix issues in AI-generated code?” Meta’s AI-allowed round runs in three phases on an existing multi-file codebase: fix a bug, build the core feature, then optimise. An internal Meta source quoted by Hello Interview summed up the criteria as “Should use AI, but need to show you understand the code.” Anthropic’s Tristan Hume describes redesigning his team’s take-home after Claude models beat it; my read of his account is that you should run your own task through current models and expect to redesign it.

The planted mistake is my addition. Canva tests whether candidates fix AI-generated errors that happen to be there; I’d put one there on purpose, so every candidate faces the same one and you can compare them.

Here are four tasks I’d use. My rough estimate: one to two hours for the candidate, about an hour for you to build. I haven’t timed these.

RoleMaterial you hand overThe planted mistakeWhat they deliver
MarketingA short brief (product, goal, budget, last quarter’s channel numbers) plus an AI-written campaign planThe plan’s cost-per-acquisition figure doesn’t match the numbers in the brief, or it targets an audience the brief rules outA one-page plan and a note listing what they kept, changed and checked
SalesA target company’s website and LinkedIn page plus an AI-written account summary and first emailThe summary states a fact about the company that its own website contradicts, such as a product it no longer sellsA rewritten first email and three questions for the first call
Operations / analystA spreadsheet of a few hundred rows (orders, tickets, invoices) plus an AI-written summary of trendsThe summary misreads one column or counts refunds as sales, so a headline number is wrongThree findings with the formula or filter behind each, and one recommendation
EngineeringA small existing repo with tests, plus an AI-written pull request for a new featureThe pull request passes the existing tests but breaks an edge case the tests don’t coverThe fix, a test that catches it, and a short note on what they’d change in the AI’s approach

Rules for building these:

  • Use real work, lightly anonymised. A task from your own backlog tells you more than a puzzle, and Rieseberg’s point about prototypes applies here too: you want to see how the person thinks about your users.
  • Make the mistake plausible. It should be confidently stated and the kind of error AI actually makes: a wrong number, an outdated fact, or a recommendation that ignores the stated goal. Check: a colleague who doesn’t know about it should find the draft convincing.
  • Run the task through a current model yourself first (15 minutes). If one prompt solves it completely, including catching the mistake, the task is too easy. That’s my takeaway from Hume’s account of Claude Opus 4.5 matching Anthropic’s strongest candidates on a two-hour take-home.
  • Cap it at two hours. It’s a sample of the work, not unpaid work.

How to do it

Time estimates in the steps are my rough guesses, not measured.

Who this is for: founders and hiring managers at small and mid-size companies hiring for knowledge work (marketing, sales, operations, engineering) in a team that already uses AI daily. Who it’s not for: roles where AI isn’t part of the job yet, and hiring where the test format is fixed by policy or a regulator; check with HR or counsel before changing it.

  1. Rewrite the role around decisions (20 minutes). In the job description, replace tool lists (“5 years of Semrush”) with what the person decides (“chooses which keywords to go after and explains why”). Check: every line answers “what will they decide?”
  2. Publish a short AI policy for candidates (30 minutes). One paragraph per stage, as Two Chairs does: application, take-home, live interview. Say where AI is expected, where it isn’t, and that people make the hiring decisions. Put it on the careers page and in the invite email. Check: a candidate can tell from the invite whether AI is allowed in the next step.
  3. Build one AI-allowed task with a planted mistake (about an hour). Pick the row from the table above that matches the role, or adapt one. Run it through a current model first.
  4. Write the scoring sheet before anyone takes the task (20 minutes). Four lines, each scored 1 to 5 with a written description of what a 1, 3 and 5 look like:
    • Caught the mistake: 1 = repeated it; 3 = noticed something was off but didn’t fix it; 5 = found it, fixed it and said how they found it.
    • Verification: what they checked, against what source.
    • Decision against the goal: does the result serve the brief, or only look finished?
    • Explanation: can they say why they kept or changed each main part?
  5. Run a 20-minute structured debrief live, on video or in person. Same questions for every candidate, in the same order (list below). Ask them to change one thing in their work while you watch; that tells you whether it’s theirs. Each interviewer scores on the sheet before discussing the candidate with anyone.
  6. Keep one short check without AI (15 minutes), especially for junior roles. Meta keeps a no-AI round next to its AI round. A small live problem, talked through without tools, is enough.
  7. Ask for something they built on their own initiative (in the first call): a prototype, a side project, an automation they set up for themselves.
  8. Check it worked (after three or four hires). Compare the scores with how those people did in their first three months. If the scores don’t line up with performance, change the task before the next round.

Debrief questions about AI use

These work for any role. Listen for specifics, not enthusiasm.

  • “Walk me through what you kept from the AI, what you changed, and why. How did you check the numbers?”
  • “Tell me about the last time AI gave you something wrong at work. How did you notice?”
  • “Which parts of your work do you never hand to AI, and why?”
  • “When did you throw away an AI answer entirely?”
  • “What’s something you automated for yourself that nobody asked you to?”

If someone can’t name a single time the AI was wrong, I’d take it as a sign they aren’t checking.

How do you tell if someone trusts AI blindly?

The planted mistake is the fastest signal. Other tells: they quote numbers without saying where they came from, their explanation repeats the AI’s wording, or they can’t say what they changed. None of these mean they can’t use AI well. They mean they haven’t learned to doubt it yet, and that’s the part I’d be hiring for.

Cheating and AI detection: what’s real

Once AI is allowed in the task, using it stops being cheating. What’s left is someone else doing the work, or a hidden assistant in the parts where you said no AI. The numbers on how common that is are softer than the headlines:

  • Gartner (reported by HR Dive, August 2025): in a survey of 3,000 job candidates, 6% admitted to interview fraud, either posing as someone else or having someone pose as them. About 4 in 10 candidates said they use AI in applications. Gartner’s forecast that by 2028 “1 in 4 candidate profiles worldwide could be fake” is a prediction, not a measurement.
  • Fabric, a company that sells AI interviews with cheating detection, says 38.5% of candidates in 19,368 interviews on its platform were flagged for cheating, 48% in technical roles against 12% in sales, and junior candidates at nearly double the senior rate. A “flag” there means its model put the probability of cheating above 40%. It says it “detects cheating in 85% of cases” without saying how that was measured, and it published no false-positive rate, so this is a vendor’s flag rate, not a measured cheating rate.
  • Amazon’s internal tips for spotting AI use, as reported, are watching for candidates typing while being asked questions, appearing to read rather than answer, or having wandering eyes. My concern with these: nervous candidates and people who take notes do the same things.

I found no independent study that measures how accurately any interview-cheating detector works, so I wouldn’t reject anyone on a detector score. What I’d do instead:

  • Allow AI where you can’t supervise. Canva saw candidates using AI “sometimes covertly through tools specifically designed to avoid detection” and chose transparency over “trying to police AI usage”. Allowing it removes the incentive to hide it.
  • Make the debrief live. Explaining and changing your own work while someone watches is hard to fake with a hidden assistant. Google’s in-person round and McKinsey’s in-person meeting are the big-company version.
  • Check identity at the live step. With 6% admitting impersonation in Gartner’s survey, it’s worth confirming that the person in the final interview is the person who did the task.

The rules if software screens or scores candidates

Everything above is people scoring people. The rules below are about the other case: a tool that screens, ranks, scores or analyses candidates for you, which can include AI features inside your applicant-tracking system. This is a summary, not legal advice.

WhereLawWhat it requires
New York CityLocal Law 144 (enforced since July 5, 2023)An automated employment decision tool must have had a bias audit within one year before use; a summary of the results must be public; candidates must get notice 10 business days before the tool is used (per the city’s DCWP page). Penalties up to $1,500 per violation per day, per DLA Piper.
IllinoisHB 3773, amending the Human Rights Act (since January 1, 2026)Bans AI that has a discriminatory effect on protected classes and bans using ZIP codes as a proxy for them. Employers with one or more Illinois employees must tell applicants and employees when AI is used in hiring and other employment decisions (per Burke, Warren, MacKay & Serritella). The state’s detailed notice rules were withdrawn on June 2, 2026; the law’s own duties still apply.
IllinoisArtificial Intelligence Video Interview Act (since January 2020)If AI analyses recorded video interviews: notify the applicant beforehand, explain how the AI works, get consent, share videos only with people needed to evaluate them, and delete them within 30 days of a request.
EUAI Act, Annex III point 4AI used to recruit or select people (targeted job ads, screening applications, evaluating candidates) is high-risk. Those obligations now apply from December 2, 2027, moved from August 2, 2026 by the AI Omnibus (Regulation (EU) 2026/1744). Employers using such systems must assign human oversight to people with “the necessary competence, training and authority” and inform workers’ representatives and affected workers.
EUAI Act, Article 5(1)(f) (since February 2, 2025)Bans AI that infers emotions “in the areas of workplace and education institutions”, except for medical or safety reasons.

The New York rule has had weak enforcement so far. A New York State Comptroller audit (December 2025) found that 75% of its test calls to the city’s 311 hotline about these tools were routed wrongly, and that in 32 published bias audits the city found one compliance issue where the Comptroller’s own review found at least 17 potential issues. The city agreed to most of the recommendations, and DLA Piper reads the audit as raising the risk for employers, not lowering it.

Whether a job interview counts as “workplace” under the EU emotion ban is a question for a lawyer. I wouldn’t buy a tool that claims to read candidates’ emotions in any market.

What not to do

  • Don’t ban AI in a take-home you can’t supervise. You’ll mostly test who’s willing to break the rule.
  • Don’t reject anyone on a detector score alone.
  • Don’t reuse a well-known task or a puzzle from the internet. Models solve those easily.
  • Don’t score the polish. AI makes every submission look finished. Score the four lines on the sheet.
  • Don’t let software reject candidates before you know which of the rules above applies to you.
  • Don’t drop the check without AI for junior roles. People who delegate everything can hand in working output they don’t understand.

What I’d do this week

Take the next role you’re hiring for and pick the matching row from the task table. Build the task with one planted mistake, run it through a current model to make sure it isn’t solved in one prompt, and write the four-line scoring sheet before the first candidate sees it. Add one paragraph to your careers page saying where AI is allowed. By my rough estimate that’s about two hours of work, and I’d expect the first two debriefs to tell you more about a candidate than a coding or writing test would.

More on how AI changes which jobs change: AI eats execution, not strategy, and glossary/ai-skill-leveling.

Sources

More of this every day on the channel: https://t.me/aidaily_business