Best of Both Worlds: Multimodal Contrastive Learning with Tabular and Imaging Data

By Paul H. et al

Published on June 10, 2023

Read the original document by opening this link in a new tab.

Table of Contents

Abstract
1. Introduction
2. Related Work
3. Methods
3.1. Contrastive Framework for Tabular and Imaging Data
3.2. Explainability using Integrated Gradients
3.3. Contrastive Learning with Labels
4. Experiments and Results
4.1. Datasets
4.2. Experimental Setup

Summary

This paper introduces a contrastive learning framework that leverages both imaging and tabular data for pretraining unimodal encoders. The proposed framework combines SimCLR and SCARF, demonstrating its effectiveness in predicting risks of myocardial infarction and coronary artery disease using medical images and clinical features. The study showcases the importance of morphometric tabular features in the contrastive learning process. Additionally, a novel supervised contrastive learning method, LaaF, is introduced by appending the ground truth label as a tabular feature. Experimental results on cardiac health prediction and car model prediction tasks show superior performance of the multimodal pretrained model compared to other models.