MLCommons introduce MedPerf: An open-source platform for benchmarking AI models to deliver clinical efficacy

Evaluating the efficacy of AI models on large and varied real-world datasets is essential for the clinical translation of Medical AI. MedPerf, an open benchmarking platform, has been announced by MLCommons, an open worldwide engineering community, to effectively evaluate AI models on a wide variety of real-world medical data to provide clinical efficacy while protecting patient privacy and minimizing legal and regulatory concerns.

Medical AI models can develop an unintentional bias against certain patient populations if trained on data from a small subset of possible clinical settings. Because of its inability to generalize, Medical AI may have less effect in the real world. However, due to privacy, legal, and regulatory considerations, data owners are reluctant to grant access to larger, more diverse datasets for training models. MedPerf enhances medical AI by eliminating bias and increasing generalizability and clinical impact by making data from around the world conveniently and safely accessible to AI researchers. 

Without access to patient data, MedPerf allows healthcare organizations to evaluate and validate AI models in a streamlined, human-supervised manner. Medical AI models are remotely installed and reviewed on-premises by data suppliers, a feature made possible by the platform’s reliance on federated assessment. Concerns about the privacy of patients’ information are alleviated, and trust is bolstered, all of which contribute to better cooperation among healthcare stakeholders.

MedPerf’s ability to orchestrate the evaluation of numerous AI models with the same collaborators allows us to do so in hours rather than months. This effectiveness was shown in the largest federated experiment on glioblastoma, the Federated Tumor Segmentation (FeTS) Challenge. The FeTS Challenge used MedPerf to benchmark 41 distinct models across 32 sites on 6 continents.

In addition, a series of pilot trials reflective of academic medical research confirmed MedPerf’s efficacy. Segmentation of brain tumors, the pancreas, and the phases of a surgical workflow were just some of the topics covered in these on-premise and cloud-based investigations. The findings confirm that federated evaluation benchmarks help move toward more accessible AI-enabled medical care for everybody.

MedPerf promotes fast.ai and other widely used ML libraries for their usability, adaptability, and performance to facilitate wider adoption. Microsoft Azure OpenAI Services, Epic Cognitive Computing, and HF inference points are only a few of the supported API-only and private AI models.

MedPerf was originally designed for radiography, but it is a general-purpose platform that may be applied to any field of biomedicine. MedPerf can support various activities, including digital pathology and omics, thanks to its sister project, GaNDLF, which is dedicated to simplifying constructing ML pipelines. To bridge the data engineering gap and give developers access to state-of-the-art pre-trained CV and NLP models, MedPerf is creating examples for the specialized low-code libraries in computational pathologies, such as PathML or SlideFlow, Spark NLP, and MONAI.

The team hopes their work will boost confidence in medical AI, speed up the spread of ML in clinical settings, and eventually let medical AI tailor care to each patient, cut healthcare costs, and enhance the quality of life for doctors and patients alike.

Check out the Reference Article and Project Page. Don’t forget to join our 26k+ ML SubReddit, Discord Channel, and Email Newsletter, where we share the latest AI research news, cool AI projects, and more. If you have any questions regarding the above article or if we missed anything, feel free to email us at Asif@marktechpost.com

🚀 Check Out 800+ AI Tools in AI Tools Club