TEHRAN - Modern deep learning models (DLMs) learn to solve a task by using the ground truth labels given for each data sample by experts. In the case of a chest X-ray image presenting a pneumonia case, the label ‘pneumonia’ will be provided by radiologists. The developers of DLMs do not control the low-level details of what image features the model is relying on to make the right diagnosis. This is one of the most important differences between modern deep learning and classical machine learning models, which required handcrafted and manually chosen features from images or other data types to perform a task.

In addition to making the training pipeline significantly simpler, this ability to learn from only the data and the provided label without the need for low-level control over the features chosen by the model enables new opportunities when investigating the features that the model uses to make its predictions. By inspecting these, one can find novel markers in medical images - e.g., X-rays or pathology slides - that reliably predict some important outcome or diagnosis; such novel features were not previously known to be related to the desired outcome. Now, however, with access to such features and the ability to learn about them, human experts can contribute their own additional input to make certain challenging clinical predictions more feasible.

More than meets the eye

In a recent study by Google researchers published in The Lancet, the authors attempted to predict some systemic parameters of the human body using only the external eye photographs.¹ They collected photographs of the external part of the eye from clinics in the United States and gathered clinical and laboratory measurements for each patient. The measurements include parameters related to the liver like albumin and aspartate aminotransferase (AST), kidney parameter