[Paper Review] LOST: A flexible framework for semi-automatic image annotation

Jonas Jäger, Gereon Reus+3 more|arXiv (Cornell University)|Oct 16, 2019

Advanced Image and Video Retrieval Techniques26 references4 citations

TL;DR

LOST is a flexible, open-source framework for semi-automatic image annotation that enables researchers to design and execute customizable annotation pipelines combining multiple tools and machine learning models. By integrating object detection proposals and clustering-based label assignment, it reduces annotation time by up to a factor of two and maintains high annotation quality (mAP ~80%) over iterative refinement, outperforming single-stage annotation in later iterations.

ABSTRACT

State-of-the-art computer vision approaches rely on huge amounts of annotated data. The collection of such data is a time consuming process since it is mainly performed by humans. The literature shows that semi-automatic annotation approaches can significantly speed up the annotation process by the automatic generation of annotation proposals to support the annotator. In this paper we present a framework that allows for a quick and flexible design of semi-automatic annotation pipelines. We show that a good design of the process will speed up the collection of annotations. Our contribution is a new approach to image annotation that allows for the combination of different annotation tools and machine learning algorithms in one process. We further present potential applications of our approach. The source code of our framework called LOST (Label Objects and Save Time) is available at: https://github.com/l3p-cv/lost.

Motivation & Objective

• To address the high time cost of manual image annotation in computer vision research.
• To enable flexible, composable workflows that combine diverse annotation tools and machine learning models.
• To reduce expert annotation time by splitting tasks into simple and expert phases.
• To support iterative, self-improving annotation pipelines that refine models over time.
• To provide a reusable, open-source framework for scalable and collaborative dataset creation.

Proposed method

• LOST models annotation as a configurable pipeline using a web-based interface with modular components.
• It integrates object detection (RetinaNet) and feature embedding (ResNet50) for generating bounding box and class proposals.
• A two-stage loop is used: first, bounding boxes are proposed and drawn; second, clusters of boxes are labeled via the MIA (multi-image annotation) interface.
• The framework supports iterative refinement by fine-tuning models (RetinaNet, ResNet50) with newly annotated data after each loop iteration.
• LOST allows custom Python scripts to be injected for full control over the pipeline logic and data flow.
• The system can be deployed locally via Docker or as a cloud-based collaborative platform.

Experimental results

Research questions

• RQ1 Can a flexible, composable framework reduce annotation time while maintaining high-quality labels?
• RQ2 How does iterative, model-guided annotation compare to single-stage manual annotation in terms of time and accuracy?
• RQ3 To what extent can clustering and batch annotation (via MIA) reduce expert annotation effort?
• RQ4 Can self-improving pipelines with feedback loops maintain or improve annotation quality over time?
• RQ5 How effective is the separation of labor into simple (proposal verification) and expert (label assignment) tasks?

Key findings

• The looped two-stage annotation approach reduced average annotation time per box by a factor of two compared to single-stage annotation in later iterations.
• Annotation quality remained stable at approximately 80% mAP with a standard deviation of 4.3%, reflecting annotator attention and image difficulty.
• RetinaNet's detection performance improved over iterations, reaching a mAP of 58% on Pascal VOC 2007, approaching the 66% mAP of fully supervised models.
• The MIA interface enabled cluster-level label assignment, cutting time for class labeling by up to 50% compared to per-box labeling.
• The framework allowed effective separation of labor: simple box drawing and expert label assignment, reducing reliance on expert time.
• Training the models during rest periods (e.g., overnight) made computational overhead negligible in practice.