[Paper Review] OLIVES Dataset: Ophthalmic Labels for Investigating Visual Eye Semantics

Mohit Prabhushankar, Kiran Kokilepersaud+4 more|arXiv (Cornell University)|Jun 9, 2022

Retinal Imaging and Analysis17 citations

TL;DR

The OLIVES dataset introduces a comprehensive, multi-modal ophthalmic dataset combining near-infrared fundus images, OCT scans, clinical labels, biomarker annotations, disease diagnoses (DR/DME), and longitudinal treatment data from 96 patients over 2+ years. It enables novel research in multi-modal learning, biomarker interpretation, and time-series treatment modeling, offering the first such dataset with structured, curated labels across all key ophthalmic data modalities for clinical and machine learning research.

ABSTRACT

The webpage associated with this dataset can be found here. Clinical diagnosis of the eye is performed over multifarious data modalities including scalar clinical labels, vectorized biomarkers, two-dimensional fundus images, and three-dimensional Optical Coherence Tomography (OCT) scans. While the clinical labels, fundus images and OCT scans are instrumental measurements, the vectorized biomarkers are interpreted attributes from the other measurements. Clinical practitioners use all these data modalities for diagnosing and treating eye diseases like Diabetic Retinopathy (DR) or Diabetic Macular Edema (DME). Enabling usage of machine learning algorithms within the ophthalmic medical domain requires research into the relationships and interactions between these relevant data modalities. Existing datasets are limited in that: (i) they view the problem as disease prediction without assessing biomarkers, and (ii) they do not consider the explicit relationship among all four data modalities over the treatment period. In this paper, we introduce the Ophthalmic Labels for Investigating Visual Eye Semantics (OLIVES) dataset that addresses the above limitations. This is the first OCT and fundus dataset that includes clinical labels, biomarker labels, and time-series patient treatment information from associated clinical trials. The dataset consists of $1268$ fundus eye images each with 49 OCT scans, and 16 biomarkers, along with 3 clinical labels and a disease diagnosis of DR or DME. In total, there are 96 eyes' data averaged over a period of at least two years with each eye treated for an average of 66 weeks and 7 injections. OLIVES dataset has advantages in other fields of machine learning research including self-supervised learning as it provides alternate augmentation schemes that are medically grounded.

Motivation & Objective

• To address the lack of integrated, multi-modal ophthalmic datasets that link clinical, imaging, biomarker, and treatment data across time.
• To support holistic machine learning research in ophthalmology by providing structured, curated labels across fundus images, OCT scans, and time-series treatment records.
• To enable research on biomarker interpretation and their relationship to disease states and treatment outcomes.
• To facilitate the development of self-supervised and multi-modal learning models for visual eye semantics and disease progression prediction.

Proposed method

• The dataset was constructed from two prospective randomized clinical trials (PRIME and TREX-DME) conducted between 2013 and 2021 at Retina Consultants of Texas.
• It includes 1268 near-infrared fundus images and at least 49 OCT scans per eye, totaling 78,185 images across 96 eyes.
• Clinical labels (e.g., BCVA, patient ID) and disease diagnoses (DR or DME) were extracted from de-identified EMRs.
• Biomarkers (e.g., intraretinal fluid, cystoid changes) were retrospectively annotated by experienced graders through open adjudication.
• The dataset includes time-series treatment data, including 7 injections per patient on average over 66 weeks of treatment.
• All data is structured and curated to support multi-modal, temporal, and biomarker-aware machine learning research.

Experimental results

Research questions

• RQ1 How can multi-modal learning models effectively integrate clinical labels, biomarkers, and imaging data (fundus and OCT) for improved ophthalmic diagnosis?
• RQ2 What is the relationship between longitudinal treatment patterns and biomarker changes in DME and DR progression?
• RQ3 How do biomarkers extracted from OCT scans correlate with clinical outcomes and disease severity over time?
• RQ4 Can self-supervised learning methods leverage the temporal structure of OCT scans and treatment history to improve representation learning in ophthalmic imaging?
• RQ5 To what extent do model predictions improve when incorporating both image data and biomarker annotations?

Key findings

• The OLIVES dataset comprises 96 eyes with 78,185 images, including at least 49 OCT scans per eye and 16 distinct biomarkers per scan.
• The dataset includes 4 clinical labels (e.g., BCVA) and disease diagnosis labels for DR or DME, with longitudinal treatment data spanning an average of 66 weeks and 7 injections per patient.
• Biomarker annotations were validated through open adjudication by experienced graders, ensuring high reliability and semantic consistency.
• The dataset enables benchmarking of multi-modal models that integrate image data, biomarkers, and time-series treatment patterns.
• OLIVES supports novel research directions in self-supervised learning, biomarker interpretation, and treatment prediction, filling a critical gap in existing ophthalmic datasets.
• The dataset is the first to provide a unified, curated, and structured resource combining all key ophthalmic data modalities for longitudinal clinical research.