Mohammed Rahim, CEO, Data Heartbeat
Mohammed RahimCEO, Data Heartbeat

Edition #13Flagship

AI Governance & Inclusion

Have We Just Automated Our Way Out of Diversity and Inclusion?

The machine did not invent our bias. It industrialised it: at scale, in silence, and with the authority of mathematics.

A hiring manager who rejects every woman who applies will, sooner or later, be noticed. A colleague raises it. A pattern emerges in the numbers. A tribunal is convened. The discrimination is human, and so, eventually, is the correction.

An algorithm that rejects every woman who applies may never be noticed at all. It does not tire, it does not leave a paper trail of intent, and it does not announce what it is doing. It simply ranks, and the people it ranks lowest are never interviewed, never hired, and never counted. The exclusion is complete precisely because it is invisible.

For twenty years we treated diversity and inclusion as a problem of human judgement. We trained managers, rewrote job descriptions, set targets, and published reports. The ambition was sincere and the progress was real, if slower than anyone wanted. Then, quietly, we began to remove the human from the first and most consequential decision of all: who gets seen.

01What the machine learned

What the machine actually learned

Start with the case everyone cites. In 2018, Reuters reported that Amazon had quietly abandoned an experimental recruiting engine. The system had been trained on a decade of CVs submitted to the company, most of them from men. It learned the pattern it was shown. It penalised CVs containing the word “women's,” as in “women's chess club captain,” and downgraded graduates of two all-women colleges. According to Reuters, Amazon's recruiters looked at the tool's recommendations when searching for new hires but never relied solely on them. Amazon said the tool was never used by its recruiters to evaluate candidates.1

The engineers did not tell the system to prefer men. They told it to find candidates who looked like the people the company had hired before. The bias was not an instruction. It was an inheritance.

That mechanism became more capable. In October 2024, researchers at the University of Washington presented a study in which they took more than 500 real résumés, attached names statistically associated with particular races and genders, and used three state-of-the-art language models to rank candidates against real job descriptions.2 The models favoured names associated with white candidates in 85 per cent of the bias tests, and names associated with Black candidates in fewer than 9 per cent. In the direct comparison that matters most, names associated with white men were preferred over names associated with Black men in every one of the 27 tests.23 Not often. Always.

One correction matters for anyone quoting this study. In August 2026 the authors disclosed a coding error that inverted their gender-only results, so those figures should no longer be cited. The race and intersectional findings, including the one above, were replicated by an independent team at the University of Salzburg.4

Exhibit 1Controlled experiment

Same résumé, different name: who the models preferred

Share of 27 bias tests (3 language models × 9 occupations) in which each group was selected at a significantly higher rate (p<0.05). Grey means no significant difference.

First group preferredNo significant differenceSecond group preferred
White vs Black namesRace only
White preferred 85.1%Black preferred 8.6%
White men vs Black menIntersectional
White men preferred 100%Black men preferred 0%
Black women vs Black menIntersectional
Black women preferred 66.7%Black men preferred 14.8%
White women vs Black womenIntersectional
White women preferred 48.1%Black women preferred 25.9%
White women vs White menIntersectional
White women preferred 25.9%White men preferred 18.5%
Gender-only results deliberately omitted. In August 2026 the authors reported a coding error that inverted them. The race-only and intersectional results shown here were independently replicated.4
Source: Wilson and Caliskan, AAAI/ACM Conference on AI, Ethics, and Society, presented 22 October 20242; University of Washington News, 31 October 20243. Simulated screening of more than 500 real résumés against more than 500 real job descriptions. Not a measurement of any deployed system.

That last finding exposes something the diversity conversation has been slow to absorb. The disadvantage facing a Black woman is not the disadvantage of being Black plus the disadvantage of being a woman, tallied up. It is its own distinct penalty, and the systems reproduce it as its own distinct penalty. An organisation can satisfy itself that its AI treats women fairly on average, and ethnic minorities fairly on average, while systematically excluding the people who belong to both groups at once.

02Experiment vs evidence

Experiment is not the same as evidence

A note of discipline is required, because this is a field where the line between what has been demonstrated and what has been observed in the wild is routinely blurred, usually in the direction of alarm.

The University of Washington study is powerful, but it is an experiment. It shows what models do in controlled conditions. It does not prove that a given bank's deployed system is rejecting a given demographic at a given rate. A further 2025 study, published in Behavioral Science & Policy, found that when applicants were told a hiring algorithm could be biased against women, women became significantly less likely to apply, including the best-qualified among them.5 But that too is experimental evidence about behaviour under disclosed conditions, not an audit of live outcomes.

Why insist on the distinction? Because the human baseline is not experimental. The field experiments of Magnus Carlsson and Dan-Olof Rooth in the Swedish labour market, and a long international literature behind them, sent out thousands of matched CVs differing only in the name at the top. Those with Arabic-sounding names received markedly fewer callbacks than identical CVs with native-sounding names.6 That is the real labour market, discriminating against real people, long before any algorithm was involved.

Put the two bodies of evidence together. Human hiring was already biased; that is established in the field. AI hiring reproduces and can concentrate that bias; that is established in the experiment. Whether it is happening inside your own systems is a question only a live audit can answer. The machine did not invent the problem. It industrialised it.

Exhibit 2

The evidence ladder: what is proven, and where

Each rung answers a different question. Only the third answers the one your board will be asked.

Rung 1Established in the field
Human field evidence

Matched CVs sent to real employers for real vacancies. Carlsson and Rooth in Sweden, and a long international literature.6

What it provesHuman hiring discriminates against real people in the real labour market.

Rung 2Established in the lab
Controlled experiments and internal tests

University of Washington résumé study2; applicant behaviour experiment5; Amazon's abandoned engine.1

What it provesModels can inherit and concentrate that bias.
What it does notThe rate inside any given deployed system.

Rung 3Your evidence gap
Live audit of your own systems

Outcomes measured in production, on your own applicant population, across groups and intersections of groups.

What it provesWho your AI is actually excluding. No vendor, regulator or study can produce this rung for you.

Real-world, humanControlled, machineReal-world, machine: yours to build
Data Heartbeat analysis of the sources cited. Rung 3 is a governance requirement, not a published finding.

The machine did not invent the problem. It industrialised it.

Mohammed Rahim
03The personal data paradox

The personal data paradox

To know whether your AI is excluding people, you must measure outcomes by the very characteristics you are not allowed to use in the decision. You cannot detect racial bias in your hiring funnel without knowing the race of your applicants. The data you need to catch discrimination is the most sensitive data you hold.

This is the personal data paradox. The instinct of a cautious organisation is to collect as little demographic data as possible, on the sound principle of data minimisation. But an organisation that collects no demographic data has not eliminated bias. It has blinded itself to it. It has made discrimination unprovable, which is not the same as making it impossible.

The resolution is not to collect everything, nor nothing. It is to govern the collection rigorously, and here the distinction between good practice and legal obligation matters. What the law requires, under the GDPR, is a lawful basis. Demographic data is special-category data under Article 9, prohibited unless a condition applies; Article 9(2)(g) permits processing necessary for substantial public interest, which member states have used to allow equality monitoring. Where a decision is solely automated and significant, Article 22 gives the person the right not to be subject to it without safeguards, including meaningful human involvement.7

What good governance adds, beyond the law, is operational discipline. The data used for bias monitoring should be separated from the data used to decide. Pseudonymisation (identifiers replaced, key retained) and anonymisation (re-identification made impractical) serve different purposes and must not be confused: pseudonymised data remains personal data under the GDPR; genuinely anonymised data does not. Masking and tokenisation protect fields in use. A data protection impact assessment, which the law does require for high-risk processing, forces the questions before deployment. Retention limited, purpose fixed, accountability with a named owner. Collect sensitive data to protect people, never to judge them.

Exhibit 3

The personal data paradox, resolved by design

Collect sensitive data to protect people, never to judge them. The wall between the two uses is the control.

Side A: to decide
Decision data
SkillsExperienceQualificationsAssessment resultsJob requirements

Job-related criteria only. Solely automated, significant decisions need safeguards and meaningful human involvement (GDPR Article 22).7

Hiring decision
Side B: to protect
Monitoring data
SexRace or ethnicityAgeDisabilityIntersections

Special-category data (GDPR Article 9). Needs a lawful condition, such as Article 9(2)(g), and a DPIA.7

Bias audit and outcome monitoring
Decision outcomesflow one way, for testing only
PseudonymisationIdentifiers replaced, key retained separately.Still personal data
AnonymisationRe-identification made impractical.Outside the GDPR if genuine
MaskingField hidden or obscured for those who do not need it.Protects data in use
TokenisationValue swapped for a token; original held in a secured vault.Protects data in use
Legal duty: lawful basisLegal duty: DPIA for high-risk processingGood practice: separation and access controlGood practice: retention limitedGood practice: purpose fixedGood practice: named owner
Legal requirements: GDPR Articles 9, 22 and 35, and Recital 26.7 Separation, masking, tokenisation and retention controls are recommended governance, not specific legal mandates.

Collect sensitive data to protect people, never to judge them.

Mohammed Rahim
04What the law says

What the law now says, and when

The EU AI Act is in force. It classifies AI used in recruitment and employment, including systems that filter applications or evaluate candidates, as high-risk, bringing obligations for risk management, data governance, transparency, human oversight and conformity assessment.8 For a bank, the deployer duties are the ones that bite.

But the timing moved. In November 2025 the Commission proposed the Digital Omnibus. It was adopted as Regulation (EU) 2026/1744 of 8 July 2026, entered into force on 27 July 2026, and postponed the applicability of the Annex III high-risk obligations, the category that includes recruitment, from 2 August 2026 to 2 December 2027.910

The meaning for a board is specific. The direction of travel is settled law: recruitment AI is high-risk, and the obligations are coming. But the hard deadline moved to end-2027. This is not a reprieve to be spent doing nothing. Anyone who tells you the law is simpler than this, in either direction, is not reading it carefully. The Act is enacted; the obligations are real; the applicability date is December 2027; and the GDPR applies in full today, deadline or no deadline.

Singapore: the same destination

Singapore is on a parallel track. Parliament passed the Workplace Fairness Act on 8 January 2025, and its companion law on dispute resolution on 4 November 2025. The Ministry of Manpower aims to bring the Act into effect in end-2027.1112 It prohibits adverse employment decisions based on protected characteristics, namely age; nationality; sex, marital status, pregnancy and caregiving responsibilities; race, religion and language ability; and disability and mental health conditions. It covers every stage of employment, from inviting candidates to interview and making offers, to appraisal, promotion and dismissal, and the Tripartite Guidelines on Fair Employment Practices continue to apply alongside it.13 The Act is written around the employment decision, not the tool that informed it. In my reading, that leaves the employer accountable whether a manager or a model shaped the outcome.

Exhibit 4

Two jurisdictions, one destination: end-2027

The EU deadline moved. The direction did not. Singapore lands in the same window.

1 Aug 2024EU AI Act in forceRecruitment AI classed high-risk (Annex III).
8 Jan 2025Workplace Fairness Act passedSingapore Parliament.
4 Nov 2025Dispute resolution law passedClaims framework for the Act.
19 Nov 2025Digital Omnibus proposedEuropean Commission.
27 Jul 2026Omnibus in forceRegulation (EU) 2026/1744.
2 Aug 2026Original Annex III dateMoved by the Omnibus.
15 Nov 2026You are hereGDPR already applies in full.
2 Dec 2027Annex III high-risk duties applyIncluding recruitment and candidate evaluation.
End-2027 (target)Workplace Fairness Act takes effectHiring, appraisal, promotion, dismissal.
European UnionSingaporeSuperseded date
Today The GDPR applies in full, deadline or no deadline: lawful basis, Article 22 safeguards, DPIAs.
Sources: Regulation (EU) 2024/16898; Regulation (EU) 2026/17449; European Commission10; Workplace Fairness Act 202511; Ministry of Manpower12. Singapore date is the Government's stated target; commencement is by Gazette notification.
05Seven board questions

Seven questions a board should be able to answer

Not a compliance checklist. A governance framework. Compliance asks whether you met the minimum; governance asks whether you actually know what your systems are doing.

Exhibit 5 · The board framework

Seven questions a board should be able to answer

Not a compliance checklist. A governance framework.

Data Heartbeat
One. Inventory

Do we know every AI and automated system touching a hiring, promotion or employment decision, including those embedded in tools we bought rather than built?This is the hidden agent problem: AI rarely arrives as a project. It arrives as a feature inside software we already pay for.

Two. Data governance

Do we hold the demographic data needed to detect bias, under a lawful basis, separated from the decision data, governed by access controls and a DPIA?

Three. Fairness testing

Have we tested each system for disparate impact across groups and intersections of groups, before deployment and on a schedule after, using our own applicant population rather than the vendor's assurances?

Four. Privacy by design

Are minimisation, purpose limitation, retention limits and the proper use of pseudonymisation and anonymisation built in, not bolted on?

Five. Accountability

Is there a named owner for each system, with the authority and budget to change or retire it?

Six. Continuous monitoring

Do we watch outcomes in production, not just performance at launch, so drift and emerging bias are caught while small?

Seven. Meaningful human oversight

When the system shapes a decision about a person, is there a human with the information, authority and time to question it? A busy person rubber-stamping a ranking they cannot see inside is not oversight. It is theatre.

Trust sets the rhythm for data & AIDownload the graphic

A busy person rubber-stamping a ranking they cannot see inside is not oversight. It is theatre.

Mohammed Rahim
06The question underneath

The question underneath all the others

Diversity and inclusion was never only a matter of good intentions. It was a claim about what an organisation could see, and therefore what it could fix. The danger of the automated age is not that machines will abandon that ambition. It is that they will let us believe we have achieved it, while quietly rebuilding the exclusions we spent twenty years trying to dismantle, in a form we have agreed not to look at.

We did not automate our way out of diversity and inclusion. We automated our way past the point where we could tell.

So the question for your next risk committee is not whether your AI is compliant. It is sharper than that. Do you know who your AI is excluding?

If the honest answer is that you cannot say, then you do not have a diversity problem you have solved with technology. You have a diversity problem you have hidden with it.

Trust sets the rhythm for data and AI.Mohammed Rahim, CEO, Data Heartbeat

Trust sets the rhythm for Data & AI

Know who your AI is excluding, before a regulator asks.

Sources

Reference numbers correspond to the article. All links checked on 17 November 2026.

  1. 1
    Reuters (Jeffrey Dastin), “Amazon scraps secret AI recruiting tool that showed bias against women”, 10 October 2018. Reports that recruiters looked at the tool's recommendations but never relied solely on them, and Amazon's statement that the tool “was never used by Amazon recruiters to evaluate candidates”.Reuters report
  2. 2
    Kyra Wilson and Aylin Caliskan, “Gender, Race, and Intersectional Bias in Resume Screening via Language Model Retrieval”, Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, vol. 7 (2024), pp. 1578 to 1590. Presented 22 October 2024. Race tests: white-associated names preferred in 85.1% of 27 tests, Black-associated in 8.6%. White male names preferred over Black male names in all 27 tests.AIES proceedings | arXiv preprint
  3. 3
    University of Washington News, “AI tools show biases in ranking job applicants' names according to perceived race and gender”, 31 October 2024.UW News release
  4. 4
    Authors' correction notice, 29 August 2026, in the study's public code repository. A coding error inverted the gender-only results; race-only and all intersectional results were replicated by researchers at the University of Salzburg.Correction notice
  5. 5
    Edwin Ip, “Fair AI in hiring: Experimental evidence on how biased hiring algorithms and different debiasing methods affect the quality and diversity of applicants”, Behavioral Science & Policy, published online 6 July 2025. Mock job application experiment with 736 participants.Journal article
  6. 6
    Magnus Carlsson and Dan-Olof Rooth, “Evidence of ethnic discrimination in the Swedish labor market using experimental data”, Labour Economics, 14(4), 2007, pp. 716 to 729. Field experiment on applicants with Arabic-sounding names.Labour Economics | IZA discussion paper
  7. 7
    Regulation (EU) 2016/679 (General Data Protection Regulation): Article 9 (special categories), Article 22 (automated decision-making), Article 35 (data protection impact assessment), Recital 26 (pseudonymised and anonymous data).EUR-Lex
  8. 8
    Regulation (EU) 2024/1689 (Artificial Intelligence Act), in force 1 August 2024. Annex III, point 4: employment, workers' management and access to self-employment, including recruitment, filtering applications and evaluating candidates.EUR-Lex
  9. 9
    Regulation (EU) 2026/1744 of 8 July 2026 (Digital Omnibus on AI), amending Regulation (EU) 2024/1689. Sets 2 December 2027 for Annex III high-risk systems and 2 August 2028 for Annex I.EUR-Lex
  10. 10
    European Commission, “AI Omnibus enters into force”, 27 July 2026. Proposed on 19 November 2025.Commission news
  11. 11
    Workplace Fairness Act 2025 (No. 8 of 2025), passed by Parliament on 8 January 2025, assented to on 3 February 2025; commencement by ministerial notification.Singapore Statutes Online | MOM press release
  12. 12
    Ministry of Manpower, Workplace Fairness (Dispute Resolution) Bill press release, 4 November 2025: “We aim for the Act to take effect in end-2027.”MOM press release
  13. 13
    TAFEP, Workplace Fairness, and “Fair recruitment practices for employers in Singapore”, 18 May 2026: protected characteristics, coverage of pre-employment, in-employment and end-employment decisions, and the Tripartite Guidelines on Fair Employment Practices.TAFEP Workplace Fairness | TAFEP recruitment guidance