Catholic Health: A practical guide for your next AI deployment

With health system budgets tightening and cost pressures rising, the value of AI has never been clearer. By reducing costs and improving efficiency, AI enables organizations to reinvest in innovation and enhance the patient experience.

One health system embracing this opportunity is Catholic Health. In a recent conversation, Tushar Garg, Notable Health Product Lead, and Allyson Collins, VP of Digital Strategy at Catholic Health, discussed how patient-facing AI became central to Catholic Health’s digital strategy.

Catholic Health’s AI innovation journey

Based on Long Island, New York, Catholic Health is blazing a new trail of innovation in healthcare. Competing in one of the most dynamic healthcare markets in the country, the health system treats every patient interaction as an opportunity to strengthen relationships and improve access, leading them to explore AI as a new digital access point.

Catholic Health’s AI innovation journey focuses on four key initiatives:

1. Implementing SMS texting with live referral agents for referral scheduling
2. Deploying inbound Voice Agents for the MyChart helpdesk support
3. Building a collaborative, iterative development process with vendors
4. Executing strategic, phased go-lives with AI Voice and text Agents

But first, Catholic Health faced critical questions at the leadership level:  

• Brand implications: Would AI Agents alienate patients?
• Oversight: How do we ensure patient-facing AI remains accurate, empathetic, and aligned with our brand values and core mission?
• Reputation: Could a single misstep impact patient trust?

“All around the country, patients have a choice of where to go for care. This is why digital matters. Patients have choices and loyalty must be earned. It’s up to our team to make sure that our digital access experience can surprise and delight our patients, so that they know they can count on us, trust us, and keep returning to us for care,” Collins adds on the importance of prioritizing digital innovation.

Setting the stage for AI success

With an Epic-first mindset, Catholic Health went live with EHR AI texting for referral scheduling. To further speed the pace of innovation, Catholic Health then partnered with an external AI provider to design, test, and launch a responsive AI voice system that quickly adapts to real-world patient needs. 

An AI Voice Agent guides digital patient access throughout the entire journey, utilizing a combination of EHR and AI tools. 

A conversation with Allyson Collins

Collins shares her insights for a successful AI deployment.*

*Editor’s Note: Responses are lightly edited for clarity and length.

Garg: How did you decide to focus your AI efforts on contact centers, specifically MyChart?

Collins: We realized we were receiving approximately 5,000 calls a month to our MyChart Helpdesk, and each call was costing $10.

At the same time, we were hearing feedback from patients that they weren’t always getting the correct answers. We knew this was our digital front door. If a patient calls and can’t access MyChart, they might never return to it, or even to Catholic Health.

We focused our energy on an AI Voice Agent and decided to start small with password resets, one of the most common reasons patients called. But once we listened to the calls, we realized patients weren’t thinking in silos.

They might think they needed a password reset but they actually forgot their username, and then they wanted help scheduling an appointment. So we shifted gears to a big bang so we could provide a cohesive experience with a bigger value.

Garg: Tell us a little bit about what the patient experience was like for patients calling into the MyChart help desk before you started doing any AI work. 

Collins: The first 30 seconds of a call can make or break the call. Our patients were waiting an average of 30 to 60 seconds to speak with a representative. At that point, 10% of the calls were getting dropped. There was a simple solution right in front of us: having the AI Agent answer the phone immediately.

And now that the AI Agent is answering the phone, we're actually answering 500 more helpdesk calls a month.

Implementation insights

Garg: Walk us through the real work of starting patient-facing AI with MyChart helpdesk. 

Collins: We started at a high level by saying, “What are the outcomes that we're trying to achieve here?” Our goal was a 20 to 25% containment rate. And then, I pushed us to aim higher and we raised it to a 30% goal because we had a substantial cost savings opportunity here. We wanted 24/7 coverage, and for the AI Agent to send text message links after verbal consent for a better user experience.

To address call latency – the time between when the patient finishes asking the question and the Agent responds – we set a certain amount of time that we didn't want the latency to exceed, as that silent pause is a top indicator that you're talking to an AI Agent.

We then created comprehensive, step-by-step instructions for performing every task in MyChart that we wanted the AI Agent to assist with. For example:

• Username request
• Password reset 
• How to find your bill
• How to add a proxy

There were five criteria for grading each conversation. 

1. Was the AI Agent able to correctly identify the intent of the call that came in? 
2. Did she follow the instructions for correctly answering the caller's question? 
3. Did she follow the tone and voice guidelines that we gave her? 
4. Did she properly transfer to a live agent as needed? We wanted every mention of an emergency or urgency to be directed immediately to a live agent. 
5. Did she hallucinate? 

We spent a great deal of time listening to dozens of AI voices. We wanted the AI Agent to sound professional, but not overly formal. We wanted her to be conversational but not too casual.  

The biggest takeaway: testing is our favorite part of the process. We created a spreadsheet containing over a hundred different tests that we wanted to run with the AI Agent, and then we used AI to grade every single call that came in as a perfect, pass, or fail. 

Lessons learned

Garg: What were some of your initial learnings after launching the patient-facing AI Agent? 

Collins: For the first few days, my team monitored calls hourly. The tech worked predictably – exactly as we designed it. But patients were unpredictable in ways that reminded us why our training materials and customer service standards matter so much.

We learned very quickly that we missed a few steps. We realized that we didn't fully account for some of the minor nuances between the web and app for MyChart. We then shifted and rewrote the instructions to include a web-specific set and an app-specific set.

We’ve also gone back and forth on changes about how often the AI interjects when a patient is in the middle of a process. So, if the patient is getting an email and then needs to click to fill out a form, how often is the agent saying that they are here when the patient needs them. We don’t want to hurry the patient but we also want them to feel like we’re paying attention and engaged in the process.

Garg: What impact have you seen so far, for both patients and your operations? 

Collins: We expected a 30% containment rate, and we started with a 52% containment rate, which has since increased to 57%.

We've taken more than 25,000 calls. We're estimating $350,000 per year in savings, which has been incredibly beneficial for us. 

The patient experience is really the greatest impact here. One patient shared with us, “You are the best virtual assistant I've ever spoken to in about 60 years. So thank you very much. You're the best. God bless you.” 

Catholic Health’s future with AI

Garg: Now that we're live with the first patient-facing AI Voice Agent, what's next?

Collins: Expanding patient-facing AI into other areas, such as our access center, which answers primary care and specialty practice calls. They've taken hundreds of thousands of calls so far this year, so there’s potential for a truly high impact. Their focus is scheduling, but a huge percentage of patients are calling back their doctor or asking questions about test results or forms.

All of these activities are repeatable tasks that we can train the AI to perform. We want our human agents to focus on scheduling, which is where they can have the most impact at the top of their license.

Garg: Three to five years from now, a patient calls into Catholic Health. What is your ideal experience? 

Collins: I don't want a patient to have to think about which call center they're calling. I would love for a patient to think ‘I'm reaching out to Catholic Health,’ and that might be via phone, chat, or text message. But every experience that they have at any of those points is consistent with the same tone and voice, and the patient is really driving that interaction based on the method and timing that works best for them.

Garg: What can readers take away from your experience implementing patient-facing AI Agents? 

Collins: Two things:

1. Get live quickly after extensive AI-powered testing, 
2. Measure success to really build on this use case and move quickly to other call centers.

Replicating Catholic Health’s success

Catholic Health’s journey shows that patient-facing AI success comes from rapid learning, testing, and scaling with confidence. 

Their approach proves what’s possible when AI innovation meets a patient-first mindset.