Algorithmic Transparency

Why transparency matters: the black box problem

Modern AI hiring systems – particularly deep learning models – often operate as “black boxes,” producing predictions or scores without revealing the reasoning underneath. The black box problem creates four distinct concerns:

Summarise this post with:

• Candidate fairness. Without insight into why an application was rejected, candidates cannot identify whether the decision was based on legitimate job-related factors or on biased proxies.
• Bias detection. Without explainability, employers cannot identify when an AI tool is using discriminatory features (even by proxy) as the basis for selection.
• Legal compliance. Anti-discrimination law requires that selection decisions be job-related. Without insight into how the AI reached its decision, demonstrating job-relatedness becomes difficult or impossible.
• Trust and adoption. Hiring managers, HR teams, and candidates are appropriately skeptical of decisions they cannot understand.

The classic case study: Amazon’s recruiting AI, which the company discontinued in 2018 after discovering that the system had learned to penalize resumes containing the word “women’s” (as in “women’s chess club captain”) and to downgrade graduates of women’s colleges. The bias was not obvious from the model output; it surfaced only through detailed analysis of feature importance.

GDPR Article 22 and the right to explanation

The EU’s General Data Protection Regulation, in force since 2018, establishes the most consequential algorithmic transparency framework globally:

Article 22 gives data subjects the right not to be subject to a decision based solely on automated processing – including profiling – that produces legal effects or similarly significantly affects them. Hiring decisions clearly fall within this scope. Article 22 carves out exceptions for decisions necessary for entering into a contract, authorized by law, or based on explicit consent – but in each case, the data subject has the right to obtain human intervention, express their viewpoint, and contest the decision.

Recital 71 of the GDPR emphasizes that data subjects have the right to “meaningful information about the logic involved” in automated decisions. This is the textual basis of the “right to explanation” referenced widely in AI ethics literature.

Articles 13, 14, and 15 require that data subjects be informed about the existence of automated decision-making, the logic involved, and the significance and envisaged consequences.

The practical implementation has been contested. EDPB guidance clarified that GDPR does not require employers to disclose the algorithm itself – only meaningful information about how it operates and the criteria used.

US transparency requirements: a state-by-state patchwork

• NYC Local Law 144 (effective July 2023). Candidates must receive notice at least 10 business days before an AEDT is used, including the job qualifications and characteristics the AEDT will use. The bias audit summary must be publicly posted on the employer’s website.
• Illinois AI Video Interview Act (effective January 2020). Applicants must be notified that AI may be used to analyze the video interview, must consent to the use, and must receive an explanation of how the AI works and what characteristics it uses to evaluate.
• Maryland HB 1202 (effective October 2020). Written consent required before facial recognition is used in employment decisions.
• California FEHC Regulations (October 2024-2025). Confirms that California anti-discrimination law applies to AI hiring tools and establishes transparency obligations.

Explainable AI: the technical toolkit

Explainable AI (XAI) is the technical discipline of producing AI systems whose decisions can be interpreted by humans. The mainstream methods:

• SHAP (SHapley Additive exPlanations). Game-theoretic method that assigns each feature a contribution value for a specific prediction. For each candidate, SHAP can produce a list of features and their positive or negative contribution to the score.
• LIME (Local Interpretable Model-agnostic Explanations). Approximates the AI decision with a simpler interpretable model in the neighborhood of a specific prediction.
• Feature importance scores. Global feature importance – which features the model relies on most across all predictions – supports general understanding of the model’s behavior.
• Counterfactual explanations. “If feature X had been Y instead, the decision would have changed.” Particularly useful for candidates.
• Attention visualization. For deep learning systems with attention mechanisms (used in resume parsing), attention maps show which input tokens the model focused on.

The recurring critique: technical explainability tools (SHAP, LIME) produce mathematically valid explanations but often fail to meet the human comprehension standard of “meaningful information” required by GDPR Article 22. The gap between technical explainability and end-user comprehension is a persistent governance challenge.

Transparency vs accountability: the related but distinct concepts

Dimension	Algorithmic transparency	Algorithmic accountability
Focus	What candidates and employees can know	Who is responsible when AI decisions cause harm
Direction	From AI system outward to affected parties	Allocates legal and ethical responsibility
Primary legal basis	GDPR Art. 22, NYC LL 144 notice rules	Title VII / ADEA / ADA, Mobley v. Workday
Technical correlate	Explainable AI (XAI), SHAP, LIME	Bias audit, impact assessment, vendor diligence
Failure mode	Black box decisions without interpretable output	Outsourcing liability to AI vendors

See algorithmic accountability for the companion treatment.

Building a transparent AI hiring program

• Inventory the AI tools. List every AI and automated decision tool in use in hiring, performance management, and scheduling.
• Pre-application notice. Where required by NYC LL 144, Illinois Video Interview Act, or similar laws, provide clear notice that AI is used and explain what characteristics it will use.
• Consent mechanics. Where consent is required (Maryland facial recognition, Illinois video AI), implement granular consent.
• Vendor diligence. Procurement criteria for AI tools should include explainability features. The vendor should be able to produce SHAP-style explanations for individual candidate decisions.
• Bias audit transparency. Where bias audits are conducted, publish the audit summary publicly and use plain-language explanation.
• Candidate request workflow. Where GDPR or similar frameworks apply, provide a workflow for candidates to request information about the automated decision affecting them.
• Human override. Article 22 requires the right to human review. Build the workflow for candidates to request human review of adverse AI decisions.

Pair algorithmic transparency with validated, job-related selection criteria that meet EEOC Uniform Guidelines and produce interpretable scores.

Frequently asked questions

Frequently asked questions