A hiring manager who rejects every woman who applies will, sooner or later, be noticed. A colleague raises it. A pattern emerges in the numbers. A tribunal is convened. The discrimination is human, and so, eventually, is the correction.
An algorithm that rejects every woman who applies may never be noticed at all. It does not tire, it does not leave a paper trail of intent, and it does not announce what it is doing. It simply ranks, and the people it ranks lowest are never interviewed, never hired, and never counted. The exclusion is complete precisely because it is invisible.
For twenty years we treated diversity and inclusion as a problem of human judgement. We trained managers, rewrote job descriptions, set targets, and published reports. The ambition was sincere and the progress was real, if slower than anyone wanted. Then, quietly, we began to remove the human from the first and most consequential decision of all: who gets seen.
What the machine actually learned
Start with the case everyone cites. In 2018, Reuters reported that Amazon had quietly abandoned an experimental recruiting engine. The system had been trained on a decade of CVs submitted to the company, most of them from men. It learned the pattern it was shown. It penalised CVs containing the word “women's,” as in “women's chess club captain,” and downgraded graduates of two all-women colleges. According to Reuters, Amazon's recruiters looked at the tool's recommendations when searching for new hires but never relied solely on them. Amazon said the tool was never used by its recruiters to evaluate candidates.1
The engineers did not tell the system to prefer men. They told it to find candidates who looked like the people the company had hired before. The bias was not an instruction. It was an inheritance.
That mechanism became more capable. In October 2024, researchers at the University of Washington presented a study in which they took more than 500 real résumés, attached names statistically associated with particular races and genders, and used three state-of-the-art language models to rank candidates against real job descriptions.2 The models favoured names associated with white candidates in 85 per cent of the bias tests, and names associated with Black candidates in fewer than 9 per cent. In the direct comparison that matters most, names associated with white men were preferred over names associated with Black men in every one of the 27 tests.23 Not often. Always.
One correction matters for anyone quoting this study. In August 2026 the authors disclosed a coding error that inverted their gender-only results, so those figures should no longer be cited. The race and intersectional findings, including the one above, were replicated by an independent team at the University of Salzburg.4
Same résumé, different name: who the models preferred
Share of 27 bias tests (3 language models × 9 occupations) in which each group was selected at a significantly higher rate (p<0.05). Grey means no significant difference.
That last finding exposes something the diversity conversation has been slow to absorb. The disadvantage facing a Black woman is not the disadvantage of being Black plus the disadvantage of being a woman, tallied up. It is its own distinct penalty, and the systems reproduce it as its own distinct penalty. An organisation can satisfy itself that its AI treats women fairly on average, and ethnic minorities fairly on average, while systematically excluding the people who belong to both groups at once.
Experiment is not the same as evidence
A note of discipline is required, because this is a field where the line between what has been demonstrated and what has been observed in the wild is routinely blurred, usually in the direction of alarm.
The University of Washington study is powerful, but it is an experiment. It shows what models do in controlled conditions. It does not prove that a given bank's deployed system is rejecting a given demographic at a given rate. A further 2025 study, published in Behavioral Science & Policy, found that when applicants were told a hiring algorithm could be biased against women, women became significantly less likely to apply, including the best-qualified among them.5 But that too is experimental evidence about behaviour under disclosed conditions, not an audit of live outcomes.
Why insist on the distinction? Because the human baseline is not experimental. The field experiments of Magnus Carlsson and Dan-Olof Rooth in the Swedish labour market, and a long international literature behind them, sent out thousands of matched CVs differing only in the name at the top. Those with Arabic-sounding names received markedly fewer callbacks than identical CVs with native-sounding names.6 That is the real labour market, discriminating against real people, long before any algorithm was involved.
Put the two bodies of evidence together. Human hiring was already biased; that is established in the field. AI hiring reproduces and can concentrate that bias; that is established in the experiment. Whether it is happening inside your own systems is a question only a live audit can answer. The machine did not invent the problem. It industrialised it.
The evidence ladder: what is proven, and where
Each rung answers a different question. Only the third answers the one your board will be asked.
Human field evidence
Matched CVs sent to real employers for real vacancies. Carlsson and Rooth in Sweden, and a long international literature.6
What it provesHuman hiring discriminates against real people in the real labour market.
Controlled experiments and internal tests
University of Washington résumé study2; applicant behaviour experiment5; Amazon's abandoned engine.1
What it provesModels can inherit and concentrate that bias.
What it does notThe rate inside any given deployed system.
Live audit of your own systems
Outcomes measured in production, on your own applicant population, across groups and intersections of groups.
What it provesWho your AI is actually excluding. No vendor, regulator or study can produce this rung for you.
The machine did not invent the problem. It industrialised it.
The personal data paradox
To know whether your AI is excluding people, you must measure outcomes by the very characteristics you are not allowed to use in the decision. You cannot detect racial bias in your hiring funnel without knowing the race of your applicants. The data you need to catch discrimination is the most sensitive data you hold.
This is the personal data paradox. The instinct of a cautious organisation is to collect as little demographic data as possible, on the sound principle of data minimisation. But an organisation that collects no demographic data has not eliminated bias. It has blinded itself to it. It has made discrimination unprovable, which is not the same as making it impossible.
The resolution is not to collect everything, nor nothing. It is to govern the collection rigorously, and here the distinction between good practice and legal obligation matters. What the law requires, under the GDPR, is a lawful basis. Demographic data is special-category data under Article 9, prohibited unless a condition applies; Article 9(2)(g) permits processing necessary for substantial public interest, which member states have used to allow equality monitoring. Where a decision is solely automated and significant, Article 22 gives the person the right not to be subject to it without safeguards, including meaningful human involvement.7
What good governance adds, beyond the law, is operational discipline. The data used for bias monitoring should be separated from the data used to decide. Pseudonymisation (identifiers replaced, key retained) and anonymisation (re-identification made impractical) serve different purposes and must not be confused: pseudonymised data remains personal data under the GDPR; genuinely anonymised data does not. Masking and tokenisation protect fields in use. A data protection impact assessment, which the law does require for high-risk processing, forces the questions before deployment. Retention limited, purpose fixed, accountability with a named owner. Collect sensitive data to protect people, never to judge them.
The personal data paradox, resolved by design
Collect sensitive data to protect people, never to judge them. The wall between the two uses is the control.
Decision data
Job-related criteria only. Solely automated, significant decisions need safeguards and meaningful human involvement (GDPR Article 22).7
Monitoring data
Special-category data (GDPR Article 9). Needs a lawful condition, such as Article 9(2)(g), and a DPIA.7
Collect sensitive data to protect people, never to judge them.
What the law now says, and when
The EU AI Act is in force. It classifies AI used in recruitment and employment, including systems that filter applications or evaluate candidates, as high-risk, bringing obligations for risk management, data governance, transparency, human oversight and conformity assessment.8 For a bank, the deployer duties are the ones that bite.
But the timing moved. In November 2025 the Commission proposed the Digital Omnibus. It was adopted as Regulation (EU) 2026/1744 of 8 July 2026, entered into force on 27 July 2026, and postponed the applicability of the Annex III high-risk obligations, the category that includes recruitment, from 2 August 2026 to 2 December 2027.910
The meaning for a board is specific. The direction of travel is settled law: recruitment AI is high-risk, and the obligations are coming. But the hard deadline moved to end-2027. This is not a reprieve to be spent doing nothing. Anyone who tells you the law is simpler than this, in either direction, is not reading it carefully. The Act is enacted; the obligations are real; the applicability date is December 2027; and the GDPR applies in full today, deadline or no deadline.
Singapore: the same destination
Singapore is on a parallel track. Parliament passed the Workplace Fairness Act on 8 January 2025, and its companion law on dispute resolution on 4 November 2025. The Ministry of Manpower aims to bring the Act into effect in end-2027.1112 It prohibits adverse employment decisions based on protected characteristics, namely age; nationality; sex, marital status, pregnancy and caregiving responsibilities; race, religion and language ability; and disability and mental health conditions. It covers every stage of employment, from inviting candidates to interview and making offers, to appraisal, promotion and dismissal, and the Tripartite Guidelines on Fair Employment Practices continue to apply alongside it.13 The Act is written around the employment decision, not the tool that informed it. In my reading, that leaves the employer accountable whether a manager or a model shaped the outcome.
Two jurisdictions, one destination: end-2027
The EU deadline moved. The direction did not. Singapore lands in the same window.
Seven questions a board should be able to answer
Not a compliance checklist. A governance framework. Compliance asks whether you met the minimum; governance asks whether you actually know what your systems are doing.
Seven questions a board should be able to answer
Not a compliance checklist. A governance framework.
One. Inventory
Do we know every AI and automated system touching a hiring, promotion or employment decision, including those embedded in tools we bought rather than built?This is the hidden agent problem: AI rarely arrives as a project. It arrives as a feature inside software we already pay for.
Two. Data governance
Do we hold the demographic data needed to detect bias, under a lawful basis, separated from the decision data, governed by access controls and a DPIA?
Three. Fairness testing
Have we tested each system for disparate impact across groups and intersections of groups, before deployment and on a schedule after, using our own applicant population rather than the vendor's assurances?
Four. Privacy by design
Are minimisation, purpose limitation, retention limits and the proper use of pseudonymisation and anonymisation built in, not bolted on?
Five. Accountability
Is there a named owner for each system, with the authority and budget to change or retire it?
Six. Continuous monitoring
Do we watch outcomes in production, not just performance at launch, so drift and emerging bias are caught while small?
Seven. Meaningful human oversight
When the system shapes a decision about a person, is there a human with the information, authority and time to question it? A busy person rubber-stamping a ranking they cannot see inside is not oversight. It is theatre.
A busy person rubber-stamping a ranking they cannot see inside is not oversight. It is theatre.
The question underneath all the others
Diversity and inclusion was never only a matter of good intentions. It was a claim about what an organisation could see, and therefore what it could fix. The danger of the automated age is not that machines will abandon that ambition. It is that they will let us believe we have achieved it, while quietly rebuilding the exclusions we spent twenty years trying to dismantle, in a form we have agreed not to look at.
We did not automate our way out of diversity and inclusion. We automated our way past the point where we could tell.
So the question for your next risk committee is not whether your AI is compliant. It is sharper than that. Do you know who your AI is excluding?
If the honest answer is that you cannot say, then you do not have a diversity problem you have solved with technology. You have a diversity problem you have hidden with it.
Trust sets the rhythm for data and AI.Mohammed Rahim, CEO, Data Heartbeat
Trust sets the rhythm for Data & AI
Know who your AI is excluding, before a regulator asks.
Sources
Reference numbers correspond to the article. All links checked on 17 November 2026.
- 1Reuters (Jeffrey Dastin), “Amazon scraps secret AI recruiting tool that showed bias against women”, 10 October 2018. Reports that recruiters looked at the tool's recommendations but never relied solely on them, and Amazon's statement that the tool “was never used by Amazon recruiters to evaluate candidates”.Reuters report
- 2Kyra Wilson and Aylin Caliskan, “Gender, Race, and Intersectional Bias in Resume Screening via Language Model Retrieval”, Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, vol. 7 (2024), pp. 1578 to 1590. Presented 22 October 2024. Race tests: white-associated names preferred in 85.1% of 27 tests, Black-associated in 8.6%. White male names preferred over Black male names in all 27 tests.AIES proceedings | arXiv preprint
- 3University of Washington News, “AI tools show biases in ranking job applicants' names according to perceived race and gender”, 31 October 2024.UW News release
- 4Authors' correction notice, 29 August 2026, in the study's public code repository. A coding error inverted the gender-only results; race-only and all intersectional results were replicated by researchers at the University of Salzburg.Correction notice
- 5Edwin Ip, “Fair AI in hiring: Experimental evidence on how biased hiring algorithms and different debiasing methods affect the quality and diversity of applicants”, Behavioral Science & Policy, published online 6 July 2025. Mock job application experiment with 736 participants.Journal article
- 6Magnus Carlsson and Dan-Olof Rooth, “Evidence of ethnic discrimination in the Swedish labor market using experimental data”, Labour Economics, 14(4), 2007, pp. 716 to 729. Field experiment on applicants with Arabic-sounding names.Labour Economics | IZA discussion paper
- 7Regulation (EU) 2016/679 (General Data Protection Regulation): Article 9 (special categories), Article 22 (automated decision-making), Article 35 (data protection impact assessment), Recital 26 (pseudonymised and anonymous data).EUR-Lex
- 8Regulation (EU) 2024/1689 (Artificial Intelligence Act), in force 1 August 2024. Annex III, point 4: employment, workers' management and access to self-employment, including recruitment, filtering applications and evaluating candidates.EUR-Lex
- 9Regulation (EU) 2026/1744 of 8 July 2026 (Digital Omnibus on AI), amending Regulation (EU) 2024/1689. Sets 2 December 2027 for Annex III high-risk systems and 2 August 2028 for Annex I.EUR-Lex
- 10European Commission, “AI Omnibus enters into force”, 27 July 2026. Proposed on 19 November 2025.Commission news
- 11Workplace Fairness Act 2025 (No. 8 of 2025), passed by Parliament on 8 January 2025, assented to on 3 February 2025; commencement by ministerial notification.Singapore Statutes Online | MOM press release
- 12Ministry of Manpower, Workplace Fairness (Dispute Resolution) Bill press release, 4 November 2025: “We aim for the Act to take effect in end-2027.”MOM press release
- 13TAFEP, Workplace Fairness, and “Fair recruitment practices for employers in Singapore”, 18 May 2026: protected characteristics, coverage of pre-employment, in-employment and end-employment decisions, and the Tripartite Guidelines on Fair Employment Practices.TAFEP Workplace Fairness | TAFEP recruitment guidance