How to screen candidates for AI developer

An AI developer test should prove one thing: can this person build, evaluate and own an AI system in production, not just name the tools. The screen that works pairs a scored work sample with a structured interview, and it ignores years of experience almost entirely.

That last part is the uncomfortable bit. Van Iddekinge and colleagues, writing in Personnel Psychology in 2019, meta-analyzed prehire work experience and found that experience measured the usual way, in years, has little relationship with later job performance. So the line on a resume that most hiring teams weight hardest is close to noise. What holds up is evidence: a sample of the work, scored the same way for everyone.

TL;DR

• Split the role into two things and test both: shipped AI skill (can they build and deploy it) and AI aptitude (can they learn a stack that changes every quarter).
• A scored work sample plus a structured interview beats a resume screen and a trivia quiz. Both have decades of validity evidence behind them.
• The new bar is AI production literacy: reviewing, explaining and taking responsibility for code a model wrote. Research on AI-authored commits shows why this matters.
• Publish your rubric before the first call. If reviewers score from memory, you have an opinion contest, not a screen.
• Budget roughly 6 to 10 weeks. AI and data roles are growing far faster than software roles overall, so the market is tight and slow screens lose candidates.
• Any AI you use to screen people is regulated. Keep a human making the decision and keep the paperwork.

What is an AI developer?

An AI developer builds and integrates AI features into working software. They handle data preparation, model selection or fine-tuning, evaluation, deployment and the monitoring that follows. The title overlaps three others, and the overlap is where most bad hires start.

Write down which of those four you are actually hiring before you write a single interview question. A team that screens an AI developer with a research-scientist bar rejects the people who would have shipped the feature, and hires someone who wanted a lab.

What skills should you screen for in an AI developer?

Screen for six things, in this order: software engineering fundamentals, data handling, model selection and fine-tuning, evaluation, deployment and monitoring, and responsible use. The first and the last are the two most teams skip, and they are the two that decide whether the feature survives contact with real users.

Split those six across two buckets, because they are screened differently.

Shipped AI skill is what the candidate has already built. Test it with a work sample. Look for: writing and maintaining production code, preparing messy data, choosing an evaluation metric that matches the business problem, versioning models and prompts, and monitoring drift after release.

AI aptitude is whether they can keep up. The stack this role uses will not look the same in eighteen months. Test it with reasoning: give them an unfamiliar problem and watch how they decompose it, what they ask about, and where they flag the one thing they would check before committing.

The Testlify AI-Era Capability Framework is built on exactly this split. It evaluates whether a candidate can perform in a workplace where people and AI work together, across core role skill, problem-solving, AI fluency, human judgment, communication, adaptability and workflow execution. For a developer role that translates to coding fundamentals, debugging, code review, system reasoning, AI-assisted development, and judging AI output rather than accepting it. AI fluency is not a substitute for core engineering skill, and it is not only a technical trait.

Pro tip: stop screening on framework names. A candidate who has shipped with one model provider and can explain the tradeoff will pick up another in a fortnight. A candidate who lists nine frameworks and cannot explain one evaluation metric will not.

How to test AI developer skills with a work sample

Give the candidate a small, job-relevant task and score it against a rubric you wrote first. The US Office of Personnel Management puts work samples among the methods whose performance relates highly to performance on the job, and notes they tend to show little difference between demographic groups. Roth, Bobko and McFarland reached the same conclusion meta-analysing the work-sample literature in Personnel Psychology in 2005, while trimming some of the more optimistic earlier estimates. Validity is not automatic, though. It depends on job relevance, standardization, reliable scoring and a candidate workload that stays proportionate, and OPM is blunt about the cost: good work samples take time to build and trained people to score.

So keep it short. Ninety minutes, not a weekend. A good AI developer sample looks like one of these:

1. Here is a messy CSV of support tickets and a rough labeling scheme. Build a classifier, report the metric you chose, and say why that metric and not accuracy.
2. Here is a working retrieval setup that returns bad answers for one class of question. Diagnose it and propose two fixes with costs.
3. Here is a model that performs well offline and badly in production. List what you would instrument, in priority order.

Then score the reasoning, not only the output. The candidate who gets a mediocre score and explains precisely why their approach was wrong is usually the better hire, and a rubric that only counts test cases passed will miss them.

On format. Testlify supports coding questions as single-file or multiple-file projects in an embedded VS Code editor, with up to 20 visible or hidden test cases, per-test-case scoring, and SQLite database test cases. It also has a vibe coding mode, where candidates direct AI tools to reach a working solution instead of writing syntax by hand. For this role that is the more honest test, because it is the job.

The developer screening process, step by step

A developer screening process for AI roles runs in five stages. The point of the order is to spend reviewer time last, not first.

1. Define the role and the rubric. Four or five competencies, each with what a 1, 3 and 5 looks like. Written down before you open the requisition.
2. Sift on qualifiers, not on resumes. Two or three yes/no gates: work authorization, production AI experience, the specific domain if it genuinely matters. Nothing subjective.
3. Run the scored assessment. The work sample above, plus a short reasoning section. Everyone who clears the gates gets the same one.
4. Structured interview. Same questions, same order, scored independently by two reviewers before they compare notes.
5. Reference and decision. Verify the specific project the candidate claimed, not their general character.

Stage 4 is where the evidence is strongest and the discipline is weakest. A 2022 reanalysis of selection-method validity places structured interviews among the strongest predictors of performance, and places years of education and general years of experience among the weakest. The exact coefficients in that literature are still argued over in print. The ranking is not: structured beats unstructured, every time anyone re-runs the numbers.

Unstructured means a different conversation with every candidate and a decision made on feel. If your loop has four interviewers who each go where the conversation takes them, you do not have four signals. You have four opinions with no way to compare them.

How to screen candidates for the right developer skills

Match the test to the work the person will actually do in their first six months. That sounds obvious and it is routinely ignored, because it is easier to reuse a generic coding test than to ask what the role needs.

Two practical moves. First, take the last three tickets someone in that role shipped and turn the smallest one into the assessment. Second, cut every competency you cannot describe an observable behavior for. If nobody can say what strong communication looks like at a 4 out of 5, it is not a criterion, it is a preference.

Here is the part most guides miss. Watch what the frontier labs are hiring for. An AI lab hiring hints at the next model long before anything ships, and the job descriptions those teams post are a leading indicator of the skills your own roadmap will need in a year. Evaluation engineering and inference cost optimization both showed up in lab postings well before they showed up in anyone's competency matrix.

How do you score an AI developer fairly?

Score against a published rubric, with at least two independent reviewers, and keep the final call with a named human. Fair scoring is mostly a documentation problem: the same evidence, the same scale, written down before anyone sees a candidate.

Hiring rubric for a technical leader on AI deployment

Hiring for the person who owns AI in production is a different job from hiring the person who writes the model code. Use five dimensions, weighted.

Testlify exposes both weighting models directly: score-based, where each test's impact follows its total score, and weights-based, where you set a weight from x0 to x5 per test. The rubric above maps onto the second one without any spreadsheet work.

One boundary worth stating plainly. If you use AI to help score, keep a person deciding. Testlify ships that as a product default, and the in-app wording is unambiguous: AI scores and insights are for guidance only. Use human judgment for final decisions. Showing AI scores to reviewers and including them in the final average are separate toggles, so a team can run AI scoring as advisory only.

Why is AI production literacy the new bar?

Because a growing share of the code your hire will touch came out of a model, and somebody has to be accountable for it. This is the screening criterion that did not exist three years ago and now belongs in every technical loop.

The evidence is getting hard to wave away. A 2026 study of 302,600 verified AI-authored commits across 6,299 GitHub repositories found that more than 15% of commits from every AI coding assistant introduced at least one issue, and that 22.7% of those issues were still present at the repository's latest version. Not caught in review. Still there. And in an earlier security audit of 1,689 generated programs across 89 scenarios, researchers found roughly 40% to be vulnerable.

So ask for it directly in the screen. Hand the candidate a plausible AI-generated pull request with a subtle bug and ask them to review it. What you are looking for is whether they read it as code or skim it as output. The candidates who catch the bug tend to describe a habit, not a talent: they run it, they check the edge case, they look at what the tests do not cover.

What soft skills should you assess in an AI developer?

Four, and they are all observable. Explaining a model to someone non-technical, changing your mind when the data says so, documenting decisions a future maintainer can follow, and saying clearly when something is not ready to ship.

Assess these inside the technical work, not in a separate personality chat. Ask the candidate to explain their work-sample approach to a hypothetical product manager in four sentences. You will learn more about their communication in those four sentences than in a half-hour conversation about teamwork.

How long does it take to hire an AI developer?

Plan for 6 to 10 weeks from requisition to signed offer, and expect the market to be tight rather than the process to be slow. The demand picture is lopsided in a way worth knowing before you set the timeline.

US Bureau of Labor Statistics projections put employment of data scientists up 35% from 2025 to 2035, about 95,400 additional jobs, against 2025 median pay of $120,230. Software developers over the same decade are projected to grow 10%, adding 174,700 jobs, at a median wage of $135,980. Both are above average, but the AI and data side is growing more than three times faster than software development as a whole.

The practical consequence: a screening loop with three separate scheduling rounds will lose candidates to companies running one scored assessment and one interview. Compressing the loop is usually worth more than widening the funnel. If time-to-hire is the constraint you are fighting, the sequencing matters more than the sourcing, and reducing time-to-hire is mostly about removing handoffs.

How do you choose AI screening tools for developers?

Judge a screening tool on four things: whether the task resembles the job, whether scoring is standardized and auditable, whether a human stays in the decision, and whether it gives you the records a regulator would ask for.

That last one is not optional any more. The EU's AI Act classifies AI systems used to recruit, screen, filter or evaluate candidates as high-risk under Annex III, with obligations phasing in over time. None of this is legal advice, and no design pattern is a safe harbor, so confirm the current rules for your jurisdiction and use case. But the direction is settled: if software helps decide who gets a job, you will be asked to show how.

Bias is the reason. A 2024 study simulating resume retrieval with language-model embeddings across occupations found significant demographic disparities in that experimental setting. That is evidence such systems can carry bias, not proof that every product does, and it is an argument for auditing the one you buy rather than assuming either way.

For a governance structure auditors already recognize, the US National Institute of Standards and Technology published the AI Risk Management Framework (AI RMF 1.0) in January 2023, organized around four functions: Govern, Map, Measure and Manage. Mapping your screening stack onto those four is a short exercise and it answers most of the questions a procurement review will raise.

Testlify covers the audit side with per-question and per-test scoring, percentile benchmarking against other candidates and other reviewers, reviewer routing where a question requires manual review from reviewer, and an AI checker that classifies an answer as human, AI generated or mixed. It integrates with the applicant tracking system you already run rather than asking you to move your system of record, and it also ships a basic, mini ATS for teams that do not have one yet.

8 AI developer interview questions to ask

Ask all eight, in the same order, to every candidate. Score each one independently before comparing notes.

1. Walk through an AI project you took from data preparation to deployment. What broke?
2. Tell us about a model that performed poorly. How did you diagnose it and what changed?
3. How would you choose an evaluation metric for this specific problem, and what would it hide?
4. Describe a time you found a problem in your training data. What did you do about it?
5. Explain a complex model or result to someone with no technical background.
6. Tell us about a technical decision you reversed during an AI project.
7. How would you identify and reduce bias in a system you shipped?
8. You inherit a pull request written mostly by an AI assistant. How do you review it?

Question 8 is the one to add if you only add one. It separates the candidates who treat generated code as a draft from the ones who treat it as done.

Hire AI developers on evidence, not resumes

Build the rubric, run one scored work sample, interview the same way twice, and keep a human on the decision. That is the whole method, and it works whether you are hiring your first AI developer or your fortieth.

If you want the assessment side handled, Testlify's AI engineer assessment and its machine learning test are built for job-relevant screening, and the coding assessments cover the multi-file and vibe-coding formats described above. You can pair either with a problem-solving assessment for the aptitude half of the split. Book a demo at hs.testlify.com to see the rubric and reviewer workflow on your own role.

Frequently asked questions (FAQs)