[Paper Review] You Really Need A Good Ruler to Measure Caching Performance in Information-Centric Networks

Liang Wang, Jussi Kangasharju+1 more|arXiv (Cornell University)|Mar 18, 2016

Caching and Content Delivery13 references3 citations

TL;DR

This paper critiques existing caching performance metrics in Information-Centric Networking (ICN), arguing they are ill-suited for ICN's network-of-caches model. It proposes the coupling factor—a novel metric linking content popularity and network topology—and advocates for footprint reduction over traditional metrics like byte hit rate or average hops, demonstrating that metric choice critically affects performance evaluation outcomes.

ABSTRACT

Information-centric networks are an interesting new paradigm for distributing content on the Internet. They bring up many research challenges, such as addressing content by name, securing content, and wide-spread caching of content. Caching has caught a lot of attention in the research community, but a lot of the work suffers from a poor understanding of the different metrics with which caching performance can be measured. In this paper we not only present a comprehensive overview of different caching metrics that have been proposed for information-centric networks, but also propose the coupling factor as a new metric to capture the relation- ship between content popularity and network topology. As we show, many commonly used metrics have several failure modes which are largely ignored in literature. We identify these problems and propose remedies and new metrics to address these failures. Our work highlights the fundamental differences between information-centric caches and "traditional" cache networks and we demonstrate the need for a systematic understanding of the metrics for information- centric caching. We also discuss how experimental work should be done when evaluating networks of caches.

Motivation & Objective

• To identify fundamental flaws in commonly used caching metrics for ICN, such as byte hit rate and average hops, which fail to capture network-level effects.
• To highlight the critical differences between ICN caching and traditional web or processor caching, especially in how hits are distributed across the network.
• To propose the coupling factor as a new metric that quantifies the relationship between content popularity and network topology to better characterize caching behavior.
• To advocate for footprint reduction as a more meaningful metric by measuring reduction in intra-ISP traffic, reflecting real cost savings.
• To guide experimental design in ICN caching research by emphasizing baseline independence and proper metric selection.

Proposed method

• Proposes the coupling factor as a metric that quantifies how well content popularity aligns with network topology, using a dimensionality-based analysis of hit distribution across the network.
• Introduces footprint reduction as a superior alternative to byte hit rate and average hops, defined as 1 - (traffic with caching) / (traffic without caching), measuring actual intra-ISP traffic savings.
• Analyzes the mathematical invariance of footprint reduction under different baseline strategies, proving that rankings remain unchanged under affine transformation.
• Uses theoretical analysis and simulation to compare metric behaviors, showing that metrics like average hops are insensitive and poorly discriminative.
• Employs a dimensionality notion to visualize where hits occur in the network, distinguishing between local, regional, and global hit patterns.
• Demonstrates that using no caching as baseline ensures metric consistency and avoids bias in performance comparison.

Experimental results

Research questions

• RQ1 Why do traditional caching metrics like byte hit rate and average hops fail to accurately reflect caching performance in ICN?
• RQ2 How does the coupling factor improve the characterization of caching behavior by integrating content popularity and network topology?
• RQ3 What are the implications of baseline selection in performance evaluation, and is the ranking of caching strategies independent of the baseline?
• RQ4 How does footprint reduction outperform other metrics in measuring real network cost savings in ICN?
• RQ5 What are the key design principles for conducting valid and meaningful experimental evaluations of ICN caching systems?

Key findings

• The footprint reduction metric is invariant to baseline strategy choice, as proven by affine transformation invariance, ensuring consistent ranking of caching strategies.
• Traditional metrics like byte hit rate and average hops suffer from critical failure modes, such as poor discrimination and misrepresentation of traffic cost savings.
• The coupling factor effectively captures the interplay between content popularity and network topology, offering deeper insight into caching behavior than isolated metrics.
• Footprint reduction provides more actionable insight than byte hit rate because it directly measures reduction in intra-ISP traffic, which correlates with operator cost savings.
• The paper demonstrates that experimental evaluation in ICN must carefully control baseline selection and metric choice to avoid misleading conclusions.
• The proposed metrics and framework reveal that ICN caching is fundamentally different from web or processor caching, necessitating a new measurement paradigm.