[Paper Review] Is there a secular decline in disruptive patents? Correcting for measurement bias

Jeffrey T. Macher, Christian Rutzer+1 more|arXiv (Cornell University)|Jun 19, 2023

Innovation Diffusion and ForecastingDecision Sciences3 citations

TL;DR

This paper challenges the methodology and conclusions of Park et al. (2023), which claimed a secular decline in disruptive patents. By correcting for truncation bias—excluding backward citations before 1976—and accounting for a 2012 U.S. patent law change allowing citations to pending applications—it demonstrates that the apparent decline is an artifact of measurement error, not a real trend. The study finds that highly disruptive patents, especially in IT, have increased since 2005.

ABSTRACT

Despite tremendous growth in the volume of new scientific and technological knowledge, the popular press has recently raised concerns that disruptive innovation is slowing. These dire prognoses were driven in part by Park et al. (2023), a Nature publication that uses decades of data and millions of observations coupled with a novel quantitative metric (the CD index) that characterizes innovation in science and technology as either consolidating or disruptive. We challenge the Park et al. (2023) patent findings, principally around concerns of truncation bias and exclusion bias. We show that 88 percent of the decrease in the average CD index over 1980-2010 reported by the authors can be explained by their truncation of all backward patent citations before 1976. We also show that this truncation bias varies by technology class. We further account for a change in U.S. patent law that allows for citations to patent applications in addition to patent grants-something ignored by the authors in their analysis--and update the analysis to 2016. We show that the number of highly disruptive patents has increased since 1980--particularly since 2008. Our results suggest caution in using the Park et al. (2023) patent findings and conclusions as a basis for research and decision-making in public policy, industry restructuring or firm reorganization aimed at altering the current innovation landscape.

Motivation & Objective

• To challenge the validity of Park et al. (2023)’s claim of a secular decline in disruptive innovation based on the CD index.
• To identify and correct for truncation bias in the Park et al. methodology, particularly the exclusion of backward citations before 1976.
• To account for a 2012 U.S. patent law change that allows citations to pending patent applications, which Park et al. ignored.
• To re-evaluate the trend in disruptive innovation using updated data through 2016 and corrected citation practices.
• To caution researchers and policymakers against relying on flawed metrics for public policy and strategic decision-making.

Proposed method

• Reconstructs the CD index using all backward citations, including those predating 1976, to correct for truncation bias.
• Applies a four-stage methodology: (I) standard truncation (as in Park et al.), (II) full backward citations, (III) inclusion of citations to granted applications, and (IV) inclusion of citations to non-granted applications.
• Uses the CD5 index, defined as the ratio of citations to a patent’s predecessors that do not cite the patent itself, to measure disruptive potential.
• Applies the corrected methodology to 3.66 million U.S. patents from 1980 to 2016, with technology classification via WIPO field codes.
• Compares the distribution of CD5 values under truncated vs. non-truncated methods to quantify bias effects.
• Updates the analysis to 2016 and incorporates citations to patent applications, which were excluded in Park et al.’s analysis.

Experimental results

Research questions

• RQ1 To what extent does excluding backward citations before 1976 distort the measurement of disruptive innovation using the CD index?
• RQ2 How does the inclusion of citations to pending patent applications—enabled by a 2012 U.S. patent law change—affect the CD index and perceived trends in disruptive innovation?
• RQ3 Does the observed decline in disruptive patents from 1980 to 2010 persist when measurement biases are corrected?
• RQ4 Are there technology-specific variations in the impact of truncation and exclusion bias on the CD index?
• RQ5 To what extent does the corrected methodology alter the conclusion that disruptive innovation is declining?

Key findings

• 88% of the reported decline in disruptive patents from 1980 to 2010 in Park et al. (2023) is attributable to the truncation of backward citations before 1976.
• After correcting for truncation bias, the number of highly disruptive patents (CD5 > 0.75) increased from 2005 onward, particularly in IT technologies.
• The proportion of patents classified as highly disruptive (CD5 > 0.75) rose from 15% under Park et al.’s method to 22% when all backward citations were included in 1980.
• When citations to non-granted patent applications were included, the CD5 index declined slightly due to citation inflation, but this did not reflect a real decline in disruptive innovation.
• The distribution of CD5 values under the corrected methodology (no truncation) shows a significant shift: only 15% of patents with CD5 > 0.75 under Park et al.’s method actually belong to that group when all citations are considered.
• The study finds that the number of highly disruptive patents has increased since 2005, contradicting the claim of a secular decline.