COLUMBIA, MARYLAND - Multimodal artificial intelligence derives its name from the multimodal datasets used to train and engage the artificial intelligence (AI). Those modes include images (still images and video), language (spoken and written words), and numerical data, primarily from databases, but also labeled values extracted from documents and images. 

Numerical data itself comes in different modes, e.g., analog data (non-digital readings on devices, perhaps recorded manually in logs), time series, tabular data, map data (points, lines, polygons), and chart data (distributions, histograms). 

Qualitative data, which can be any of those three modes just mentioned, also exists, but qualitative data itself is multimodal - collected and presented in different ways, such as words, multiple-choice responses to survey questions, sentiment, emotion, intent, and context.

In healthcare, a number of multimodal data collections can be used in AI applications, including medical imaging, patient photos, verbal and written notes from physicians, lab results, patient surveys, health diagnostics, electronic health records (EHR), electronic medical records (EMR), and much more. 

Emerging AI applications that require ingestion, processing, and decisioning on such multiple diverse data types will necessarily be more complex than unimodal (single data format) AI applications, such as a medical imaging diagnosis or a conversational AI chatbot for answering patients’ questions. Those complexity challenges are outweighed by the significant benefits that multimodal AI applications can bring to healthcare, including clinical care, diagnoses, treatments, and anomaly discovery. 

This article discusses multimodal AI, and includes a short tutorial, examples, and some healthcare applications.

Sensational Systems