[Paper Review] Citations in Software Engineering -- Paper-related, Journal-related, and Author-related Factors

Mika Mäntylä, Vahid Garousi|arXiv (Cornell University)|Aug 12, 2019

Software Engineering Research50 references4 citations

TL;DR

This study investigates factors influencing citation counts in software engineering (SE) journals using negative binomial and quantile regression on 25,113 SE papers (1970–2018). Key findings show that publication venue, author team’s past citations, paper length, reference count, and recency of references are the most influential predictors of citations, with high-impact venues and teams with few but highly cited past papers yielding higher citation counts.

ABSTRACT

Many factors could affect the number of citations to a paper. Citations have an important role in research policy and in measuring the excellence of research and researchers. This work is the first study in software engineering (SE) to assess multiple factors affecting the number of citations to SE papers. We use (a) negative binomial regression and (b) quantile regression to study arithmetic mean and median expected citations of a paper. Our dataset includes all the 25,113 papers which have been published in a set of 16 main SE journals, between 1970 and 2018. Our results indicate that publication venue, author team's past citations, paper length, the number of references, and the recency of references are the most influential factors on the number of citations to SE papers. From our empirical findings, we present several implications and advice to researchers for getting higher citations on their papers, which are in addition to the obvious case of conducting high-quality technical research, e.g. (1) Aim for high-profile venues, (2) Build a high-quality author team with highly cited past papers, and (3) Aim for high-quality work that has comprehensive content (thus longer paper length and reference list).

Motivation & Objective

• To identify and quantify the impact of paper-related, journal-related, and author-related factors on citation counts in software engineering.
• To address the gap in SE bibliometrics by extending prior work with robust multivariate modeling and a comprehensive dataset.
• To provide actionable insights for researchers aiming to increase citation impact beyond technical quality alone.
• To examine whether citation patterns in SE align with broader scientific trends or exhibit unique characteristics.

Proposed method

• Collected data from Scopus for 25,113 SE journal papers published between 1970 and 2018.
• Applied negative binomial regression to model mean citation counts, accounting for overdispersion in citation data.
• Used quantile regression to analyze median citation expectations, reducing the influence of extreme outliers.
• Incorporated predictors including paper age, length (pages), title length, reference count, reference recency (Price Index), journal impact factor, and author team metrics (past citations, publication count, team size).
• Conducted model diagnostics and error analysis, particularly for journals with missing data (e.g., TOSEM).
• Validated findings against prior studies in other scientific fields to assess consistency and uniqueness in SE.

Experimental results

Research questions

• RQ1 Which paper-related, journal-related, and author-related factors significantly predict citation counts in SE journals?
• RQ2 How do the effects of these factors differ when modeling mean versus median citations?
• RQ3 To what extent do author team characteristics—such as past citation performance and publication volume—predict citation outcomes?
• RQ4 How does the recency of references influence citation counts, and what does it indicate about author awareness of prior work?
• RQ5 Are the predictors of citations in SE consistent with findings from other scientific disciplines?

Key findings

• Publication venue and journal impact factor are among the strongest predictors of citation counts, with high-impact journals significantly increasing expected citations.
• Author team past performance—measured by citations to prior papers—is a major driver of citation counts, with teams having few but highly cited past papers outperforming those with high publication volume but low impact.
• Paper length (number of pages) and reference count are positively associated with higher citation counts, suggesting comprehensive, well-documented work receives greater recognition.
• Recency of references (Price Index) is a significant predictor, indicating that citing recent work correlates with higher citation potential.
• Title length negatively affects mean citations, suggesting shorter, more concise titles may improve visibility and citation potential.
• The number of authors is not statistically significant in predicting mean citations, but author affiliation count becomes relevant when modeling median citations.