[Paper Review] A dataset of medication images with instance segmentation masks for preventing adverse drug events

W. I. Chu, S. Hirani+2 more|arXiv (Cornell University)|Mar 11, 2026

Cold Fusion and Nuclear Reactions0 citations

TL;DR

MEDISEG provides 8,262 pill images with 32 classes and pixel-accurate instance masks to train robust pill recognition models; baseline YOLOv8/YOLOv9 show strong performance and few-shot transfer benefits.

ABSTRACT

Medication errors and adverse drug events (ADEs) pose significant risks to patient safety, often arising from difficulties in reliably identifying pharmaceuticals in real-world settings. AI-based pill recognition models offer a promising solution, but the lack of comprehensive datasets hinders their development. Existing pill image datasets rarely capture real-world complexities such as overlapping pills, varied lighting, and occlusions. MEDISEG addresses this gap by providing instance segmentation annotations for 32 distinct pill types across 8262 images, encompassing diverse conditions from individual pill images to cluttered dosette boxes. We trained YOLOv8 and YOLOv9 on MEDISEG to demonstrate their usability, achieving mean average precision at IoU 0.5 of 99.5 percent on the 3-Pills subset and 80.1 percent on the 32-Pills subset. We further evaluate MEDISEG under a few-shot detection protocol, demonstrating that base training on MEDISEG significantly improves recognition of unseen pill classes in occluded multi-pill scenarios compared to existing datasets. These results highlight the dataset's ability not only to support robust supervised training but also to promote transferable representations under limited supervision, making it a valuable resource for developing and benchmarking AI-driven systems for medication safety.

Motivation & Objective

• Motivate the need for realistic, multi-pill datasets that reflect real-world medication handling scenarios.
• Create MEDISEG with detailed instance segmentation masks across diverse lighting, occlusion, and dosette-box contexts.
• Evaluate state-of-the-art detectors (YOLOv8/YOLOv9) on MEDISEG to establish baseline performance.
• Demonstrate few-shot learning transfer when adapting to unseen pill classes.
• Provide a data resource and guidance to support research in medication safety through AI.

Proposed method

• Capture and preprocess real-world pill images in dosette-box setups with varied lighting using an iPhone 12 Pro Max.
• Annotate every pill with COCO-format instance segmentation masks, ensuring accuracy in overlaps and occlusions.
• Create MEDISEG (3-Pills) and MEDISEG (32-Pills) subsets with 3 and 32 pill classes, respectively.
• Train YOLOv8 and YOLOv9 on MEDISEG with 70/20/10 train/val/test splits to establish baselines.
• Conduct few-shot detection experiments using the FsDet framework (base/novel class split) to assess transfer under occlusion and overlap.
• Hyperparameter tuning via a genetic algorithm to optimize learning rate, momentum, and related settings for improved localization.

Figure 1: (A) Trends in AEMT events from 1980 to 2014 and (B) the distribution of these events across age groups [ 10 ] .

Experimental results

Research questions

• RQ1 Can MEDISEG realistic multi-pill imagery improve object detection and segmentation performance for pill recognition tasks?
• RQ2 Do detectors pretrained on MEDISEG transfer better to unseen pill classes under occlusion or overlap compared to models trained on controlled, single-pill datasets?
• RQ3 How do YOLOv8 and YOLOv9 compare on complex, multi-pill scenes with varying lighting and overlaps?
• RQ4 What is the impact of few-shot learning when adapting to new pill classes using MEDISEG as the base dataset?

Key findings

• YOLOv8 and YOLOv9 achieve high mAP@50 on the 3-Pills subset (up to 99.5%) and substantial but lower mAP on the 32-Pills subset (80.1%).
• In few-shot tests, models base-trained on MEDISEG outperform those trained on a more controlled dataset when recognizing unseen, occluded pills.
• YOLOv9 generally achieves higher mAP@50-95 on the 32-Pills subset, indicating stronger localisation under challenging conditions.
• Base training on MEDISEG yields lower detection losses and more stable localisation in visually complex scenes across 1-, 5-, and 10-shot setups.
• MEDISEG-based pretraining improves generalisation to unseen pill classes in overlap-heavy test sets, especially under heavy occlusion.

Figure 2: Examples of images taken from (A) the dataset by Lee et al. [ 4 ] , (B) the NIH Pillbox dataset [ 15 ] , (C) the CURE dataset [ 5 ] , (D) the dataset by Wong et al. [ 14 ] , and (E) the dataset by Tan et al. [ 11 ] .