One of the most wrenching revelations of healthcare during the covid pandemic has been the lack of providers and too many ICU patients to monitor them effectively. Before the onset of the pandemic, Techolution partnered with Brooklyn hospital to create an ML-driven approach to high-risk event detection and alerting in patient monitoring in ICUs and ERs. This would provide predictive, proactive care support to the most seriously ill patients.
The goal was to create a pre-trained AI that would be architected for deployment on the cloud to enable comprehensive dashboard access. The model would be trained on empirical real-world data garnered from historical code blue and high-risk events.
Process, Tools, and Results
Here, the Techolution team utilized Google cloud, Google Kubernetes engine, and Kafka for streaming data updates to a BigQuery data lake. Model building relied on R/Python pipelines for data processing, and H2O as well as TensorFlow for machine learning model building.
Unlike other teams tackling AI platform development, Techolution AutoAI took advantage of TensorFlow to serve a variety of other libraries, including Flask for API services. Using containerized solutions enables close integration of on-prem data and the cloud via Kubernetes services.
The team worked closely with Brooklyn to train the model on ICU event data from March to July 2020. The result would be defined by an expected 100 percent per-patient-per-day sensitivity to these high-risk events, which showed a 96 percent specificity. The model results were based on cross-validated training exercises after probabilistic threshold optimization.
The model continued to deliver high accuracy over the next three months. It showed an overall accuracy and specificity of 96 percent and an 80 percent sensitivity rate to high-risk ICU events on a per-patient-per-day basis.
As in all AI/ML algorithms, the ability of the model to perform its designated task (in this case, predicting high risk patient alerts for proactive care) completely depends on the accuracy of the data used to train the model. As time goes by it’s important to provide the best data and a highly organized and automated system to ensure that the latest model is the best and most accurate model. Techolution’s approach is focused on the reality that the first model is never the best and having multiple models would enable long-term accuracy, adaptability, and resiliency.
Importance of MLOps Champion/Challenger Models
There are three consistent challenges that stop many businesses and their AI models in their tracks:
- Access to data science expertise
- Access to huge, clean data sets
- The natural occurrence of model accuracy degradation.
The Techolution AutoAI approach has overcome these challenges via the use of MLOps Champion/Challenger models.
The purpose behind the Champion/Challenger method is to allow the best approaches to predictive model testing in a production environment to constantly rise to the forefront. This approach is like an A/B testing model found in marketing, but here, it requires multiple AI models to determine the most successful model at accurate predictions.
The original model is designated as the Champion. It is shadowed by the new or retrained models known as (Challengers). This approach pits the Challengers against each other and the Champion with the model having the greatest accuracy and success rate becoming the new Champion. The overall goal is to identify which variation is the most successful. By making the models compete, the MLOps engineer can make an accurate recommendation about the model with the highest accuracy.
This iterative process designates the best model based on predictive accuracy as the new champion. Because creating an AI model is a cyclical process, the Challengers are always being refined to take their place as the Champion model. This cyclical process is always taking place so that the best model can be essentially hot swapped in real time to ensure the best model is always being used.
The AI/ML Model is Only the Start
It doesn’t get discussed much among stakeholders, but every AI/ML model is custom, and few teams have the internal expertise on staff to reach a production grade model.
That doesn’t change the fact that there are many companies in the marketplace that purport to deliver customized AI models at a “reasonable cost.” Many businesses never leave the exploratory pilot stage when they find out the development of the AI model is only part of an AI platform that encompasses:
- Data management, storage, cleaning, and routing
- IoT integration (Bedside medical device monitoring systems in the case of Brooklyn)
- Fully automated end-to-end cloud platform,
- Integrated application CI/CD pipeline
- UX/UI design
While every business needs every element, they all vary as to the level of outside support they need to both envision their goals and then realize them. The structure of most organizations does not allow them to deal with the perpetual cycle of model refinement. The result is that the ongoing specialized staffing costs and AppDev team integration are unsustainable. Even if you have these engineers and developers on staff, they have limited time to devote to projects beyond their main strength.
Like most organizations, Brooklyn Hospital has limited resources to compete for the data scientist, MLOps, cloud and AppDev pipeline experts to handle a massive AI project like this on their own. It’s clear that the Champion/Challenger MLOps model delivers the competing models that are focused on delivering the best model as things change over time. But holding together such an interdisciplinary team under a single employer’s roof is both highly impractical and expensive, so what’s the solution?
Al/ML Platform Success Starts with A Focus on Very Specific Outcomes
Techolution’s partnership with Brooklyn Hospital shows how we can bring the team in any combination of disciplines and timeframes as fractional resources to create a bespoke AI model, In this case, one that can provide the predictive, high risk patient detection for all of their ICU beds and the multiple systems that deliver that patient data in real time.
We built the success of the partnership with Brooklyn Hospital on a foundation where very specific outcomes in patient care were determined before choosing any tool, technology, or method. This foundation has guided every decision. Getting to the business value meant drilling down beyond the problem of helping care providers use their time and technology more effectively to identify high-risk patients. On a granular level, Brooklyn Hospital and Techolution sought answers to questions like:
- How does this improve patient outcomes?
- How does this improve care delivery?
- How does this improve resource allocation?
- How does this improve the entire healthcare environment interdepartmentally?
- How does this improve the efficiency and effectiveness of our IT organization?
The answers to these intertwined questions make the risks and rewards very clear, so stakeholders can make sound, long-term decisions about what it takes to run an enterprise level AI/ML platform application. Every decision is based on whether a business partner wants to run the environment end-to-end or just parts of it and gaining an understanding of what that means. In this case, the cloud platform and the containers that would hold the application engine driven by the AI model would be built on a Techolution GCP instance group with Brooklyn Hospital internal team members having access where needed.
By collaborating with Techolution UI and UX specialists that truly understand their healthcare providers’ workflows, Brooklyn was able to partner on the design of a dashboard that provided:
- Active alerts
- Beds online
- Beds occupied
- People logged into the system
- Individual patient profiles with BP, pulse, O2, respiration, and last update time
- Total vital alerts with vitals changes that triggered each alert
- Predictive alerts that show predicted, false positive, and unpredicted
- High frequency alerts
- Predictive analytics dashboards to track patient changes over time along with alert accuracy
The success of the Brooklyn Hospital AI platform for ICU high risk event detection and predictions is one Techolution partnership among many across healthcare, finance, retail, manufacturing, and other sectors. We grounded the success of each of these projects in a strong focus on helping each partner to determine their real-world business outcomes need. We then collaboratively work backwards from that understanding to create each partner’s personal team, processes, and technologies that will continue to drive its evolution.
This approach becomes the ideal foundation where deep experience in cloud, AI, IoT, and AppDev shows that the best and only blueprint to success will be a customized one. Our Techolution team can then work with our partners without being hemmed in by preconceived notions about the best tools, platforms, cloud providers, or machine learning platforms.
An AI/ML platform must continually be capable of delivering accurate results that drive business or health outcomes where the stakes couldn’t be higher. By delivering the fractional team that you need today and tomorrow, we can consult, learn and educate in ways that enable your team to see the future possibilities that may have seemed hidden just yesterday.