Why 89% of Healthcare AI Agents Never See a Real Patient

Back in my DTC days I watched a Stripe dashboard hit a million dollars in a single day. We sold beautiful packaging, fast shipping, a slick brand story. The product itself? Mediocre. The customer service? Worse.

Scale hid the problems for almost two years.

I think about that a lot now, because Corti just published research showing that only 11% of healthcare AI agents reach production. The other 89% die in pilot. Why? "Safety spirals." Error cascades where one mistake, in coding or documentation or scheduling, propagates through five connected systems before any human notices.

It's a great term. Safety spiral. It captures exactly what I worry about every night running HANA. Because when your AI talks to a million patients, you don't get a Stripe dashboard ticking up. You get one error that turns into a hundred errors that turns into a regulator on the phone.

So let me tell you how we think about it.

What's a safety spiral in healthcare AI?

A safety spiral happens when an AI agent makes a small error inside a workflow, and the downstream systems treat that error as truth. The agent codes a visit wrong. Billing acts on it. The EHR confirms it. The next agent reads the EHR and builds a care plan on the wrong code. Three systems are now wrong, and nobody has flagged it yet.

Corti calls this the reason 89% of healthcare AI agents never make it past pilot. I'd put it more bluntly. Pilot is the only environment where the spiral doesn't kill you, because in pilot, humans are watching every step.

Production is where you find out if your system can actually catch itself.

Why does open source matter for healthcare AI?

Because if you can't see the agent's reasoning, you can't catch the spiral. You can only react to it after the damage.

HANA is fully open source. Self-hosted, no dependency on OpenAI or any external LLM provider, full audit trail of every decision the agent makes. We did this for one reason. CMOs and CIOs need to read the code. Not just trust a SOC 2 badge. Read the actual logic.

When you self-host an AI agent the air gap is structural. The PHI never leaves the perimeter. The behaviour is reproducible. The audit is yours, not your vendor's. That's why our deployments across five countries include health systems that wouldn't touch a closed-box voice AI.

It's also why HANA has zero critical adverse events across more than a million patient interactions. (Which still amazes me, honestly.)

How does HANA stay safe at scale?

Three architectural choices. First, every clinically significant input has a deterministic guardrail behind it, not a model decision. If a patient says "chest pain," the system escalates. That's a rule. Not a probability.

Second, agents are scope limited. The follow-up agent can ask about symptoms and medication adherence. It cannot reschedule a surgery. It cannot order a refill. Limited scope means limited blast radius. If the agent is wrong, the wrongness has a ceiling.

Third, everything is logged in a way clinicians can read. Not a black box transcript. A clinical timeline with the agent's reasoning, the patient's responses, and every handoff to a human. Our docs walk through it if you want to see the structure.

The boring stuff is the safe stuff. That's the lesson.

What should health system execs ask their AI vendors?

Five questions. Ask them this week. They'll tell you everything you need to know.

One. "Can I read the source code?" If no, you're trusting a marketing deck.

Two. "Where does PHI live during inference?" If the answer involves a third-party LLM you've never audited, you have a black box problem.

Three. "Show me your last critical incident and how the system caught it." If they can't, the system isn't catching them. It's missing them.

Four. "What can the agent NOT do?" Good vendors have a list. Bad vendors say "anything."

Five. "Who owns the audit trail in two years if you go out of business?" If the answer is anything other than the customer, walk away.

Why does self-hosted infrastructure win this decade?

Because regulators are catching up. The ONC's HTI-1 Final Rule introduced transparency requirements for predictive AI in clinical workflows back in 2024, and the requirements have only tightened since. State boards are scrutinising AI-driven clinical communication. Insurers are starting to require audit trails for AI tools that touch patient care.

The clinics and health systems we work with don't want to be the test case that proves you needed on-premise. They want infrastructure they own, code they can read, and a vendor that's structurally incapable of seeing their data.

That's not a future-state opinion. That's how every HANA deployment ships today.

Key takeaways

The Corti research is the clearest articulation I've seen of why healthcare AI keeps stalling. It's not the models. It's the absence of governance. Pilots succeed because humans backstop every action. Production fails because there's nobody watching when the spiral starts.

Open source, scope-limited agents, deterministic guardrails, full audit trails, structural data sovereignty. None of that is glamorous. None of it gets written up in TechCrunch. But it's the entire reason HANA has gotten to a million interactions without a critical adverse event.

If you're a health system exec evaluating voice AI, the question isn't "how smart is the agent." It's "how does the agent fail." Read the answer in the code, not the deck.

FAQ

Why does open source matter if the vendor is HIPAA compliant? HIPAA compliance is the floor, not the ceiling. Open source code means your security team can validate behaviour, not just trust a checkbox. It also means you're not stranded if the vendor pivots, gets acquired, or shuts down.

What's the difference between voice AI and an AI agent? Voice AI is the interface. The agent is the reasoning layer behind it. HANA combines both: voice for the patient conversation, an agent for the clinical workflow underneath. The agent decides what to ask, what to log, when to escalate.

How long does a HANA deployment take? For a single clinic, typically two to four weeks. For a health system with multiple specialties and EHRs, six to twelve weeks. The integration work is mostly EHR and telephony plumbing, not modelling.

If you're a health system exec or platform builder evaluating voice AI infrastructure, come ask us the hard questions. The boring answers are the ones that hold up in production.