Working with AI Engineering Teams

It's 2 AM, and your feature just broke production

Picture this. You're a PM named Priya. Your team spent six weeks building an "AI meeting notes" feature. Two weeks before launch, your lead engineer Dani pulls you aside and says:

Your stomach drops. You open your spec. It reads:

That looked complete when you wrote it. But it left four massive questions unanswered — and those unanswered questions just ate six weeks of your team's life.

So what went wrong? And how do you make sure it never happens to you?

That's what this entire lesson is about. By the end, you'll know how to write a spec so clear that engineering can build against it without a single follow-up Slack thread.

The handoff is where AI features go to die

Here's the thing most PMs don't realize: AI features don't fail because of bad code. They fail because the spec never defined what "good" looks like. That gap — between what you meant and what engineering built — costs months of re-work, triggers post-launch rollbacks, and breaks trust with your engineering team.

Think about it like baking a cake. If you tell someone "make a good cake," you'll get... something. Maybe it's chocolate. Maybe it's vanilla. Maybe it's three tiers with fondant. You didn't say. But if you say "make a single-layer chocolate cake, 9 inches, that scores at least 4 out of 5 in a taste test with 10 people" — now your baker knows exactly what to aim for.

AI specs work the same way. "Accurate" is not a spec. A number is a spec.

The golden rule: Build the test BEFORE the feature

This is the single most important pattern you'll learn in this lesson. Here's the sequence that actually works:

Notice something? Engineering builds the eval harness FIRST — before writing a single line of feature code. An eval harness is just an automated test suite that checks whether the AI output meets your spec.

Why? Because without it, "done" stays subjective. With it, "done" is a green checkmark on a dashboard.

Think of it like building a house. You wouldn't start pouring concrete without a blueprint, right? The eval harness IS your blueprint. It's how everyone agrees on what the finished house looks like before anyone picks up a hammer.

And here's the part that makes your life easier as a PM: once you approve the eval baseline, you have a concrete artifact for stakeholders. Instead of saying "the team is building it," you can say "we agreed on passing criteria and engineering is building to them." That's a much better answer in a status meeting.

Vague specs vs. real specs: spot the difference

Let's go back to Priya's story. Here's what her original spec said versus what she rewrote it to say:

What's ROUGE-L? It's a standard way to measure how closely an AI summary matches a human-written reference summary. It works by finding the longest sequence of words that appears (in order) in both the AI summary and the human reference, then computing an F1 score from that. A score of 0.65 is a strong result; a score near 0 means the summaries share almost nothing. You don't need to memorize the formula — you just need to know it gives you a number instead of a feeling.

And speech-to-noise ratio? That's just the ratio of clear speech to background noise. If someone's recording a meeting at a construction site, the audio quality is low, and the AI will produce bad summaries. Priya's spec said: "Don't silently produce bad output. Show a warning instead." That's a product decision, not an engineering decision — and it belongs in YOUR spec.

The results tell the story

Before the rewrite: Dani's team logged 47 Slack threads asking Priya to clarify what "good" meant.

After the rewrite: Zero follow-up questions. A two-week build cycle. A launch where Priya could point to a dashboard and say "ROUGE-L is 0.71 — we shipped."

The rewrite took four hours. The vague spec had already cost six weeks of uncertainty.

The five elements of an AI feature spec

Every AI feature spec you write needs exactly five things. Think of them like the five fingers on your hand — if one is missing, your grip on the project is weak.

1. User story — Who wants what, and why?

2. Success metric with a specific number — Not "good." Not "fast." A number.

3. Eval criteria — what does PASS look like? — Write out the exact conditions.

4. At least one edge case and how to handle it — What happens when things go weird?

5. A rollback trigger — What number tells you to pull the plug?

That rollback trigger isn't bureaucracy — it's insurance. It pre-commits everyone to the definition of failure before emotions run high. When the feature is live and numbers are dropping, you don't want to be arguing about whether it's "bad enough" to roll back. You want a number that everyone agreed to in advance.

Think of it like a circuit breaker in your house. You don't debate whether the wiring is overloaded while your kitchen is on fire. The breaker trips automatically at a preset threshold. Your rollback trigger works the same way.

"Accurate" is not a metric — say it with me

This deserves its own section because it's the single most common mistake PMs make in AI specs.

Every time you catch yourself writing one of these words, stop and replace it with a number:

See the pattern? Every vague word becomes a metric name + a number + a dataset. That's the formula. Burn it into your brain.

Here's a trick that works every time: when you finish writing your spec, do a Ctrl+F for the words "accurate," "fast," "relevant," "good," and "quality." If any of them appear without a number next to them, your spec isn't done yet.

The big challenge: Write a real spec

You've learned the pattern. You've seen what vague looks like and what specific looks like. Now put it all together.

Back to Priya and Dani

Two sprints later, Priya walked into the same meeting room, but with a different spec in her hand. Six pages. Role definition, success criteria, a golden evaluation set of 200 meeting recordings, a ROUGE-L threshold of 0.65, and a rollback trigger at 0.55.

Dani read it in ten minutes. "This is what we needed six weeks ago," she said.

"I know," said Priya. "We build the test first now."

The feature shipped. No follow-up Slack threads.

Key takeaways

• Build the test before the feature. Put eval criteria in your spec so engineering has a test harness to build BEFORE writing feature code. It's like agreeing on what "done" tastes like before you start cooking.
• "Accurate" is never a metric. Every time you write a vague word, replace it with a number: "ROUGE-L score of 0.65+ on our 200-example golden set" is a success metric; "accurate" is not.
• Include a rollback trigger in every spec. It shows you planned for when things go wrong, not just when they go right. It's your circuit breaker — it trips automatically so nobody has to argue during a crisis.