Artificial intelligence (AI) has immense potential to empower clinicians by synthesizing patient data and medical knowledge to provide valuable insights. At Google, we understand how AI can benefit the clinician community. We collaborated with Apollo 24|7, the largest multi-channel digital healthcare platform in India, to build their Clinical Intelligence Engine (CIE). This CIE can be used to generate evidence-based insights from the wealth of knowledge captured in their millions of de-identified clinical interaction data points. This solution is designed as an assistive tool to empower clinicians by augmenting their abilities, thereby providing better patient care.

Apollo 24|7’s solution includes multiple approaches. It uses Med-PaLM 2, which is now part of our newly released MedLM and clinical knowledge graph-based models.

Dataset used

We used the MIMIC IV dataset, which is a publicly available electronic health record dataset that includes free-text discharge notes.

Evaluation metrics

Below are the three key metrics that were used for evaluating the experiments:

1. Modified MAP@K: Mean Average Precision (MAP) is calculated for the top K candidates, in order to judge a recommender-like system. We calculate the average precision for each value up to K in an iterative fashion.
2. Precision Score: The precision score helps us determine how many false positives are present in the predictions, as compared to the true positives.
3. Capture Rate: The capture rate metric helps us empirically determine how many of the actually present values in the ground truth were identified correctly by the model. In a way, it is similar to recall.

Experimentation

We experimented with multiple models, ranging from Google Research’s large language model (LLM) called MedLM, to graph models like personalized pagerank, and traditional models like kNN (K-nearest neighbor) and two-tower models.

In this blog, we explain our approach using Personalized Pagerank augmented by MedLM.

Step 1: Build a clinical knowledge graph

• We built a comprehensive clinical knowledge graph in Neo4j, which is a graph database.
• First, we use Google’s PaLM 2 model and the Vertex Healthcare Natural Language API to extract medical concepts from the unstructured medical text in the form of discharge notes.
• These clinical entities include concepts like medicines, dosage, medicine frequency, medicine duration, and symptoms. We link them to create the right relations. The extracted entities are ingested into Neo4j as the graph database.

• We create a bipartite knowledge graph with only two types of nodes - Admissions & UMLS. The UMLS (Unified Medical Language System) codes are generated by the response of the Healthcare NL API.
• In order to retain the context of the entities, we extract the sections (like Symptoms, Allergies, Past Medical History and so on) from the discharge summaries. The medicine, Paracetamol, present in the Medication or Allergy sections means two different things even though the UMLS code is the same. Hence, we create an intermediate node SUMLS, which is a combination of UMLS+Section.

Once we created this knowledge graph, we experimented with the different graph algorithms.

Step 2: Build personalized Pagerank model using GraphSage and Optuna optimization

• Based on the clinical knowledge graph created in Neo4j, the baseline Pagerank model was built.
• In order to consider the node attributes, the node embeddings were used to represent the similar nodes with nearby vectors and the kNN algorithm was used to connect the top 10 most similar admissions.
• Lower weights were given to the edges of popular nodes based on their degree in order to reduce their popularity effect.
• Next, a GraphSage model was trained to create the neighborhood embeddings of the nodes present in the bipartite knowledge graph.
• For optimizing the weights of the edges, Optuna was used.

Step 3: MedLM augmented results

MedLM harnesses the power of Google’s MedLM, and is aligned to the medical domain to more accurately answer medical questions. It can be used to facilitate rich, informative discussions, answer complex medical questions, and find insights in complicated and unstructured medical texts. It is also used to help draft short- and long-form responses and summarize documentation and insights from internal data sets and bodies of scientific knowledge.

In our experiments, we fed the output of the Personalized Pagerank model to MedLM as context and generated the final response, which had a higher accuracy.

We observed that the final responses generated by this approach using MedLM and clinical knowledge graphs were grounded in factuality and are far more accurate by reducing the false positives and boosting the true positives.

“This solution built on MedLM augmented with a clinical knowledge graph can analyze a patient's medical records and generate insights on relevant medications, laboratory evaluations, medical procedures, and potential diagnoses for the clinician to review. By generating these evidence-based insights, this gen AI solution aims to enhance the clinical workflows, reduce errors, and improve patient outcomes. And it is super important to understand that this is just the tip of the iceberg in terms of the AI’s capabilities where it is so powerful, but yet always assistive to the clinicians.” - Abdussamad M, Engineering Lead at Apollo 24|7.

This solution is not intended to replace the clinician's expertise but rather to augment the clinician’s skills and experience.

Fast track end-to-end deployment with Google Cloud Consulting (GCC)

The partnership between Google Cloud and Apollo 24|7 is just one of the latest examples of how we’re providing AI-powered solutions to solve complex problems to help organizations drive the desired outcomes. With Google Cloud Consulting (GCC), Apollo was able to perform repeated iterations and experiments to build the final solution, thereby empowering the business. Apollo entrusted GCC to collaborate with their teams to build the state of the workflows for their business requirements. The GCC portfolio provides a unified services capability, bringing together offerings across multiple specializations, into a single place. This includes services from learning to technical account management to professional services and customer success. See Google Cloud Consulting’s full portfolio of offerings.

^{Disclaimer: MedLM is still in the preview phase in India and it is not approved for production use}