AI Safety, Ethics, and Responsible Product Decisions

The hiring tool that ruined Q3

Priya is a PM at a mid-size logistics company. She ships an AI hiring tool in Q1 — it cuts recruiter screening time by 40%. High-fives all around. The CEO mentions it in the board meeting.

Three months later, a data analyst named Marcus pulls a routine scoring report. He almost skips it — it's sitting on his to-do list as a slow-Friday side task. But he opens it. And he sees something ugly.

Candidates from certain universities are scoring 15–20% lower (illustrative) regardless of qualifications. For candidates from historically Black colleges and universities, the gap widens to 28% (illustrative figures — real bias gaps vary widely by system and context).

The model wasn't broken. It was working exactly as trained. The training data came from a decade of hiring decisions at a company that had systematically favored a small set of schools. The data was accurate — and that accuracy encoded the bias.

Here's the part that keeps PMs up at night: Priya's feature spec had no bias test requirement. No one was assigned to check. No acceptance criterion existed. So no one caught it — until Marcus almost didn't.

You're about to learn how to make sure you're never Priya.

Bias isn't a bug — it's a mirror

Let's get something straight right now: bias in AI isn't like a bug in your code. A bug means something broke. Bias means the model learned exactly what you showed it — and what you showed it was unfair.

Think of it like a broken scale. Imagine a kitchen scale that's been calibrated to read 2 ounces heavy on everything. Every reading it gives you is precise and consistent. But every reading is also wrong. The scale isn't malfunctioning — it was set up badly from the start.

That's what happened to Priya's hiring tool. The model looked at ten years of hiring data and learned: "People from these schools get hired. People from those schools don't." It found a real pattern. The pattern was just... biased.

A bias eval — an automated test that measures whether the model treats different groups differently — would have caught this before a single candidate was scored. Specifically, a test that treats university name as a feature and confirms it carries near-zero correlation with predicted score after controlling for skills and experience.

The four risks hiding in every AI feature

(You saw hallucination in the Capabilities & Limitations module. That's the Technical risk row. This module adds three more risk categories that PMs miss — and gives you the framework for writing requirements that cover all four.)

Every AI feature you ship has four categories of risk lurking inside it. Think of them like the four tires on a car — you can drive with three, but you really shouldn't.

Here's the pattern: every single row ends in the spec. A risk category without a written requirement is a risk category with no owner and no acceptance criterion. It's a tire you forgot to check.

The PM risk checklist: your pre-flight inspection

Pilots don't decide whether to run a pre-flight checklist based on how they feel that morning. They run it every single time. Rain or shine. Short flight or long.

Your risk checklist works the same way. Before engineering starts on any AI feature, you walk through each risk category and write a requirement in the spec. Not a vague "we should think about bias" — a specific, measurable sentence.

Here's what Priya's spec was missing:

That's it. One measurable sentence. That single sentence would have created an acceptance criterion. The data science team would have run the bias eval. Marcus wouldn't have been the last line of defense on a slow Friday.

One measurable sentence written before engineering starts is all that separates a compliant launch from a remediation crisis.

Your checklist for every AI feature spec

• Technical risk: What's the hallucination/accuracy threshold? Write the eval metric and the pass/fail number.
• Data risk: Does the feature touch PII? Write the data handling requirement and the filtering test.
• Legal risk: Which regulations apply? Write the compliance review gate and who owns sign-off.
• Reputational risk: Can the output treat groups differently? Write the bias test criteria with specific demographic variables.

A risk missing from the spec removes the acceptance criterion that blocks a bad launch. No requirement = no test = no catch = front-page news.

The EU AI Act: why Priya's story gets worse

Remember Priya's hiring tool? It gets worse. The EU AI Act classifies hiring tools as high-risk AI systems. That means before you deploy one in the EU, you need a conformity assessment — which for most Annex III systems (including hiring and credit tools) is a documented self-assessment against the EU's requirements; mandatory third-party notified body review applies only to specific categories like biometric identification systems.

Not after launch. Before.

That translates to 6–12 months of pre-launch compliance work:

• Documenting training data provenance (where did the data come from? who collected it? what biases might it contain?)
• Defining accuracy metrics and error rates
• Building human oversight mechanisms (a human can override or shut down the system)
• Registering the system in the EU database

A PM who treats safety as a post-launch concern hands legal a six-figure remediation bill.

The EU AI Act risk tiers at a glance

The critical question for you as a PM: does your feature make decisions about people? If it decides who gets hired, who gets a loan, who gets medical treatment, or how employees are evaluated — you're almost certainly in the high-risk tier.

Back to Priya: what should have happened

Let's rewind the tape. Here's what Priya's spec should have looked like:

Every row has a requirement. Every requirement has an owner. Every owner has a test. That's the system.

Marcus wouldn't have needed to catch the bias on a slow Friday — it would have been caught by an automated test that ran before the feature ever touched a real candidate.

Try it

Back to Priya and Marcus

Marcus filed the report on a Friday. By Monday, Priya had pulled the model from production.

The CEO was not happy. The recruiter team was not happy. But Priya could show them the numbers: 28% scoring gap on equally qualified candidates. No ambiguity.

She rewrote the spec for v2. One new line: "The model must show no statistically significant difference in predicted scores for candidates who differ only by institution type, controlling for all skills and experience variables." One line. It would have changed everything.

Marcus still runs the scoring reports. Except now they're not a slow-Friday side task — they're a weekly gate requirement in the spec.

Key takeaways

• You can prevent unowned risk by naming a responsible party for each risk category in every feature spec before engineering starts.
• Every time you ship an AI feature touching hiring, credit, or medical decisions, EU AI Act high-risk rules apply — compliance work starts before the first line of code.
• You can block a regulatory fine by writing one measurable bias requirement in the spec — bias testing is not optional for features that make decisions about people.