[Paper Review] Unpacking Security Scanners for GitHub Actions Workflows

Madjda Fares, Yogya Gamage+1 more|arXiv (Cornell University)|Jan 20, 2026

Scientific Computing and Data Management0 citations

TL;DR

Systematically compares 9 state-of-the-art static analyzers for GitHub Actions workflows, mapping their rules to a common taxonomy of 10 weaknesses and evaluating coverage, consistency, and performance on a dataset of 596 real-world workflows.

ABSTRACT

GitHub Actions is a widely used platform to automate the build and deployment of software projects through configurable workflows. As the platform's popularity grows, it also becomes a target of choice for software supply chain attacks. These attacks exploit excessive permissions, ambiguous versions or the absence of artifact integrity checks to compromise the workflows. In response to these attacks, several security scanners have emerged to help developers harden their workflows. In this paper, we perform the first systematic comparison of 9 GitHub Actions Workflows security scanners. We compare them regarding scope (which security weaknesses they target), detection capabilities (how many weaknesses they detect), and performance (how long they take to scan a workflow). In order to compare the scanners on a common ground, we first establish a classification of 10 common security weaknesses that can be found in GitHub Actions Workflows. Then, we run the scanners against a curated set of 2722 workflows. Our study reveals that the landscape of GitHub Actions Workflows security scanners is very diverse, with both general purpose and focused scanners. More importantly, we provide evidence that these scanners implement fundamentally different analysis strategies, leading to major gaps regarding the nature and the number of reported security weaknesses. Based on these empirical evidence we make actionable recommendations for developers to harden their GitHub Actions Workflows.

Motivation & Objective

• Define a taxonomy of 10 high-level security weaknesses in GitHub Actions Workflows to enable fair cross-tool comparison.
• Curate a set of actively maintained, locally executable scanners for benchmarking.
• Empirically compare scanners on coverage, consistency, and runtime using a large real-world workflow dataset.
• Provide actionable guidance to developers for hardening GitHub Actions workflows.

Proposed method

• Curate 9 state-of-the-art, locally executable GitHub Actions workflow scanners from an initial pool of 30 candidates.
• Create a 10-item weakness taxonomy by aggregating and clustering 84 rules across tools.
• Map each tool's detection rules to the taxonomy to assess coverage of weakness classes.
• Benchmark scanners on 596 real-world workflows from 77 repositories to study detection consistency and runtime.
• Measure runtime using Unix time per workflow file, repeated twice for stability.

Experimental results

Research questions

• RQ1 RQ1: How similarly do scanners cover the same workflow weakness classes across tools?
• RQ2 RQ2: How consistently do scanners detect weaknesses on the same workflows?
• RQ3 RQ3: What are the runtime characteristics and reporting features of the scanners?

Key findings

• No single scanner covers all identified weakness classes; some are highly specialized while others are broad in scope.
• Specialized and general-purpose scanners are complementary for comprehensive security coverage.
• Most scanners are fast enough for CI integration, with a few outliers adding modest delays.
• Zizmor and poutine offer broad weakness coverage among the evaluated tools.
• Scanner outputs vary in interpretation of weaknesses, leading to differences in reported findings.
• An open science repository provides access to all data and results.