[Paper Review] MediaPipe: A Framework for Building Perception Pipelines

Camillo Lugaresi, Jiuqiang Tang+12 more|arXiv (Cornell University)|Jun 14, 2019

Embedded Systems Design Techniques6 references219 citations

TL;DR

MediaPipe provides a graph-based framework to build, evaluate, and deploy perception pipelines with reusable components, cross-platform support, and performance tools.

ABSTRACT

Building applications that perceive the world around them is challenging. A developer needs to (a) select and develop corresponding machine learning algorithms and models, (b) build a series of prototypes and demos, (c) balance resource consumption against the quality of the solutions, and finally (d) identify and mitigate problematic cases. The MediaPipe framework addresses all of these challenges. A developer can use MediaPipe to build prototypes by combining existing perception components, to advance them to polished cross-platform applications and measure system performance and resource consumption on target platforms. We show that these features enable a developer to focus on the algorithm or model development and use MediaPipe as an environment for iteratively improving their application with results reproducible across different devices and platforms. MediaPipe will be open-sourced at https://github.com/google/mediapipe.

Motivation & Objective

• Enable rapid prototyping of perception pipelines by composing reusable components (calculators) into graphs.
• Provide a cross-platform deployment environment that preserves behavior and performance across devices.
• Offer tooling for performance evaluation, synchronization, and resource management to guide iterative improvements.
• Support GPU acceleration and multi-platform graphics APIs to optimize perception workloads.
• Facilitate dissemination and reuse through an open architecture with subgraphs and configurable execution.

Proposed method

• Define pipelines as graphs of modular calculators connected by time-stamped data streams.
• Use GraphConfig protocol buffers to describe topology and node options.
• Support side packets for constant data and streams for time-varying data.
• Implement a scheduling system with per-node readiness, timestamps, and executors for parallelism.
• Provide GPU integration with opaque buffers and cross-context synchronization of OpenGL/Metal workflows.
• Offer performance tools (Tracer and Visualizer) to analyze packet flow and graph topology.

Experimental results

Research questions

• RQ1 How can perception pipelines be efficiently prototyped as modular graphs of calculators?
• RQ2 How does MediaPipe manage scheduling, synchronization, and determinism to support real-time pipelines?
• RQ3 What mechanisms enable cross-platform deployment and GPU acceleration without sacrificing performance?
• RQ4 What tooling supports performance evaluation and debugging for perception graphs?

Key findings

• MediaPipe enables rapid prototyping by composing reusable calculator components into configurable graphs.
• The framework provides deterministic yet flexible synchronization based on per-stream timestamps and settled timestamps.
• GPU support is integrated with cross-context synchronization, enabling GPU-accelerated pipelines without CPU bottlenecks.
• Performance tooling (Tracer and Visualizer) facilitates tracking packet flow, latency, and graph topology for tuning.
• Subgraphs and modular calculators promote reuse and cross-platform consistency across development and deployment environments.