Included Health's Blueprint for Taming AI in Healthcare

SAN FRANCISCO, CA – June 17, 2026 – As the healthcare industry grapples with the immense promise and potential peril of generative artificial intelligence, one company is attempting to write the rulebook for its safe deployment. Included Health, a company delivering integrated healthcare services, today announced the publication of a peer-reviewed framework in NEJM Catalyst Innovations in Care Delivery that offers a detailed blueprint for implementing patient-facing AI with clinical safety at its core.

In an era where a quick search for symptoms can lead patients to unvetted, general-purpose AI tools, the potential for misinformation and harm is significant. This new paper, titled “Blueprint for Safety: Implementing a Clinically Governed AI Digital Assistant for Patient Guidance,” makes the case for a fundamentally different approach—one built not for speed alone, but for safety, transparency, and trust.

A Framework Built on Clinical Governance

The model detailed by Included Health is a direct response to the 'wild west' environment of public AI. It outlines how the company developed and piloted an AI assistant designed to handle routine health questions safely while intelligently escalating higher-risk situations to human clinicians. This approach moves beyond simple automation to create a system that understands its own limitations.

“At Included Health, we believe healthcare AI should be held to a higher standard than general-purpose tools,” said Ami Parekh, MD, the company's chief health officer. “People deserve timely support, but they also need a model they can trust to provide accurate, safe guidance.”

This higher standard is built upon four foundational principles:

1. Cross-Functional Governance: From the outset, the AI initiative was governed by a team including clinical, product, and executive leadership, ensuring that medical oversight was not an afterthought but a core component of development.
2. Proactive Risk Analysis: Before a single patient interacted with the system, the team conducted extensive pre-launch testing and 'red teaming' to identify potential failure modes and stress-test the AI's safety guardrails.
3. Three-Tier Risk Classification: The system was designed to classify user questions into standard, high-risk, or emergency categories. This stratification is crucial, allowing the AI to provide instant answers for low-risk queries while immediately routing more serious concerns to qualified professionals.
4. Continuous Clinician-in-the-Loop Review: During the pilot, every single clinical interaction was manually reviewed by clinicians daily. This continuous audit serves a dual purpose: it ensures immediate patient safety and provides invaluable data for refining and improving the system over time.

This level of oversight directly addresses a primary concern among medical AI experts, who consistently warn against the 'black box' problem where AI decisions are opaque. By embedding clinicians in the review process, the model ensures human judgment remains the ultimate authority.

From Theory to Practice: The Pilot's Promising Results

To validate its framework, Included Health deployed the risk-stratified AI assistant in a seven-week pilot study involving 50% of its patient population. The results suggest that a safety-first approach can coexist with, and even enhance, efficiency. The clinical review, which covered 100% of interactions, found that the AI maintained a high level of safety throughout the pilot.

Most notably, the system reduced unnecessary handoffs for standard-risk questions by an impressive 65%. In a healthcare system burdened by administrative tasks, this represents a significant improvement. It allows more patients to receive instant, clinically appropriate guidance for their routine needs, thereby freeing up human clinicians to focus their expertise on more complex and urgent cases. This demonstrates a tangible way technology can improve lives by optimizing the allocation of a health system's most valuable resource: its people. Patient experience, a critical metric for adoption and trust, also remained high and on par with the control group that did not use the AI assistant.

Charting a Course in a Complex Landscape

Included Health's publication is particularly timely. Regulatory bodies like the FDA and international consortiums are actively debating how to best regulate medical AI to ensure patient safety without stifling innovation. Recent studies have raised alarms, with some analyses showing that a significant number of FDA-authorized AI tools lacked comprehensive clinical validation data. The core challenge for the industry is to move beyond purely technical metrics and evaluate AI's real-world clinical impact, including its potential to introduce or exacerbate bias.

Against this backdrop, the company's decision to publish its methodology in a peer-reviewed journal is a strategic move toward transparency. It invites scrutiny and provides a tangible model for others to follow. “We believe the framework described in this paper offers a practical blueprint for how healthcare organizations can deploy patient-facing AI with safety, transparency, and human oversight built in from the start,” noted Ankoor Shah, MD, vice president of clinical excellence at Included Health.

This framework aligns with the growing consensus among ethics experts that human oversight is non-negotiable. By explicitly designing a system that knows when to ask for help, the model provides a crucial safety net that is often missing from more general AI applications.

The 'AI + EQ' Philosophy

This clinically governed AI is not a standalone product but a component of Included Health's broader 'AI + EQ' (Emotional Quotient) strategy. The company’s model is built on providing comprehensive care navigation, advocacy, and 24/7 support. The AI assistant functions as the first point of contact within this larger ecosystem, designed to seamlessly integrate with human-led services.

This integration is key to building patient trust. Members are not just interacting with a chatbot in a vacuum; they are engaging with a system that guarantees access to a human expert when the stakes are higher. This hybrid approach, combining the efficiency of AI with the empathy and expertise of clinicians, reflects a mature understanding of how technology can best serve human needs in healthcare. By being transparent about the AI’s role and its limitations, the model empowers patients while ensuring their safety.

By publishing their blueprint, Included Health is doing more than just showcasing its own innovation; it is issuing a challenge to the entire digital health industry to build toward a higher standard of care where technology serves, supports, and protects patients.