[Paper Review] Citation Counting, Citation Ranking, and h-Index of Human-Computer Interaction Researchers: A Comparison between Scopus and Web of Science

Lokman I. Meho, Yvonne Rogers|ArXiv.org|Mar 12, 2008

scientometrics and bibliometrics research40 references7 citations

TL;DR

This study compares Scopus and Web of Science in measuring citation impact for 22 top HCI researchers, finding Scopus provides significantly broader coverage—especially of conference proceedings—leading to more nuanced h-index and citation rankings. Despite similar journal citation counts, Scopus offers more detailed and distinct citation network visualizations and is recommended as a standalone source for HCI research evaluation when conference citations are essential.

ABSTRACT

This study examines the differences between Scopus and Web of Science in the citation counting, citation ranking, and h-index of 22 top human-computer interaction (HCI) researchers from EQUATOR--a large British Interdisciplinary Research Collaboration project. Results indicate that Scopus provides significantly more coverage of HCI literature than Web of Science, primarily due to coverage of relevant ACM and IEEE peer-reviewed conference proceedings. No significant differences exist between the two databases if citations in journals only are compared. Although broader coverage of the literature does not significantly alter the relative citation ranking of individual researchers, Scopus helps distinguish between the researchers in a more nuanced fashion than Web of Science in both citation counting and h-index. Scopus also generates significantly different maps of citation networks of individual scholars than those generated by Web of Science. The study also presents a comparison of h-index scores based on Google Scholar with those based on the union of Scopus and Web of Science. The study concludes that Scopus can be used as a sole data source for citation-based research and evaluation in HCI, especially if citations in conference proceedings are sought and that h scores should be manually calculated instead of relying on system calculations.

Motivation & Objective

• To evaluate and compare citation counting, citation ranking, and h-index between Scopus and Web of Science for top HCI researchers.
• To assess whether broader database coverage in Scopus leads to meaningful differences in researcher impact metrics.
• To examine the reliability and consistency of system-generated h-index values versus manually calculated ones.
• To investigate how citation network maps differ between the two databases for individual scholars.
• To compare h-index scores from Scopus and Web of Science with those from Google Scholar using a union of both databases as a benchmark.

Proposed method

• Selected 22 leading HCI researchers from the EQUATOR project for analysis.
• Collected citation counts and h-index values from Scopus, Web of Science, and Google Scholar.
• Compared citation data focusing on journal-only citations to isolate the effect of conference proceedings.
• Generated and compared citation network visualizations for individual researchers using both databases.
• Calculated h-index manually and compared it with system-reported values to assess accuracy.
• Used the union of Scopus and Web of Science data as a reference standard for Google Scholar comparison.

Experimental results

Research questions

• RQ1 How do citation counts and h-indices for HCI researchers compare between Scopus and Web of Science?
• RQ2 Does Scopus’s broader coverage of conference proceedings significantly alter citation rankings and h-index scores compared to Web of Science?
• RQ3 Are there notable differences in citation network maps generated by Scopus versus Web of Science for individual scholars?
• RQ4 How do system-reported h-index values from Scopus and Web of Science compare with manually calculated ones?
• RQ5 How do h-index scores from Google Scholar compare with those derived from the union of Scopus and Web of Science?

Key findings

• Scopus provides significantly more coverage of HCI literature than Web of Science, primarily due to inclusion of relevant ACM and IEEE conference proceedings.
• No significant differences in citation counts or h-indices were found when comparing only journal citations between the two databases.
• Scopus produces more nuanced and distinct citation rankings and h-index values than Web of Science due to broader coverage.
• Scopus generates significantly different citation network maps for individual scholars compared to Web of Science, indicating divergent network structures.
• System-reported h-index values in Scopus and Web of Science differ from manually calculated values, suggesting potential inaccuracies in automated calculations.
• The h-index based on the union of Scopus and Web of Science data serves as a reliable benchmark, and Scopus alone is sufficient for citation-based evaluation in HCI when conference citations are important.