[Paper Review] Pragmatic Image Compression for Human-in-the-Loop Decision-Making

Siddharth Reddy, Anca D. Dragan+1 more|arXiv (Cornell University)|Jul 7, 2021

Simulation Techniques and Applications4 citations

TL;DR

This paper proposes Pragmatic Image Compression (PICO), a human-in-the-loop learning approach that compresses images to lower bitrates by preserving only the visual features necessary for users to perform specific tasks. By training a compression model to induce the same user actions on compressed images as on originals—using an adversarial discriminator to distinguish original from compressed image responses—PICO achieves 2–4× bitrate reduction across diverse tasks while maintaining functional similarity.

ABSTRACT

Standard lossy image compression algorithms aim to preserve an image's appearance, while minimizing the number of bits needed to transmit it. However, the amount of information actually needed by a user for downstream tasks -- e.g., deciding which product to click on in a shopping website -- is likely much lower. To achieve this lower bitrate, we would ideally only transmit the visual features that drive user behavior, while discarding details irrelevant to the user's decisions. We approach this problem by training a compression model through human-in-the-loop learning as the user performs tasks with the compressed images. The key insight is to train the model to produce a compressed image that induces the user to take the same action that they would have taken had they seen the original image. To approximate the loss function for this model, we train a discriminator that tries to distinguish whether a user's action was taken in response to the compressed image or the original. We evaluate our method through experiments with human participants on four tasks: reading handwritten digits, verifying photos of faces, browsing an online shopping catalogue, and playing a car racing video game. The results show that our method learns to match the user's actions with and without compression at lower bitrates than baseline methods, and adapts the compression model to the user's behavior: it preserves the digit number and randomizes handwriting style in the digit reading task, preserves hats and eyeglasses while randomizing faces in the photo verification task, preserves the perceived price of an item while randomizing its color and background in the online shopping task, and preserves upcoming bends in the road in the car racing game.

Motivation & Objective

• To address the inefficiency of standard lossy compression, which optimizes for perceptual fidelity rather than task-specific utility.
• To reduce image transmission bitrates by focusing on preserving only the visual information that drives user decisions in downstream tasks.
• To develop a compression method that adapts to individual user behaviors without prior knowledge of their tasks or action labels.
• To validate that functional similarity—inducing identical user actions—can be learned through human-in-the-loop feedback and adversarial training.
• To demonstrate the method's generalization across diverse real-world tasks such as digit recognition, photo verification, online shopping, and video gaming.

Proposed method

• Train a neural network-based encoder-decoder compression model to generate compressed images from original inputs.
• Use human-in-the-loop interaction: users perform tasks (e.g., click, steer, identify) on compressed images, and their actions are logged.
• Train a discriminator to classify whether a user’s action was triggered by the original or compressed image.
• Optimize the compression model to maximize the discriminator’s confusion—i.e., to make actions on compressed images indistinguishable from those on originals.
• Use adversarial loss to implicitly define a functional similarity objective, avoiding the need for ground-truth action labels or task-specific loss functions.
• Apply the method across multiple tasks using real human participants on Amazon Mechanical Turk, with model training on pilot data and evaluation on new users.

Experimental results

Research questions

• RQ1 Can image compression be optimized for functional similarity—inducing identical user actions—rather than perceptual fidelity?
• RQ2 To what extent can a compression model learn task-relevant features through human-in-the-loop feedback without explicit task labels?
• RQ3 How much bitrate reduction can be achieved with pragmatic compression compared to non-adaptive and perceptual similarity baselines across diverse tasks?
• RQ4 Does PICO adaptively preserve task-critical features (e.g., digit number, eyeglasses, road bends) while discarding irrelevant details (e.g., handwriting, background, distant road)?
• RQ5 Can PICO maintain user performance and situational awareness in real-time decision-making tasks like video gaming under low-bitrate constraints?

Key findings

• PICO achieved 2–4× lower bitrates than non-adaptive and perceptual similarity baselines while matching user actions on compressed and original images.
• In digit recognition, PICO preserved the digit number but randomized handwriting style, demonstrating task-specific feature prioritization.
• In photo verification, PICO preserved hats and eyeglasses while randomizing facial features, aligning with the task of detecting occlusions.
• In online shopping, PICO preserved perceived price and car shape while discarding color and background, enabling accurate price-based selection at lower bitrates.
• In the car racing game, PICO preserved road bends and angles critical for steering decisions, leading to significantly higher road progress (F(1,11) = 176.32, p < .0001) compared to non-adaptive baselines.
• Users reported higher situational awareness and control with PICO compared to non-adaptive compression, indicating improved usability under bandwidth constraints.