[Paper Review] TypeEvalPy: A Micro-benchmarking Framework for Python Type Inference Tools

Ashwin Prasad Shivarpatna Venkatesh, Samkutty Sabu+4 more|arXiv (Cornell University)|Dec 28, 2023

Software Engineering Research4 citations

TL;DR

TypeEvalPy is a standardized micro-benchmarking framework for evaluating Python type inference tools, featuring 154 code snippets across 18 categories with 845 type annotations. It containerizes tools, normalizes output, and measures performance using metrics like exact match rate and precision, revealing that hybrid approaches like HiTyper-DL outperform pure ML methods but soundness remains limited, with top tools achieving only 44% soundness.

ABSTRACT

GitHub link: https://github.com/secure-software-engineering/TypeEvalPy A Micro-benchmarking Framework for Python Type Inference Tools 📌 Features: 📜 Contains 154 code snippets to test and benchmark. 🏷 Offers 845 type annotations across a diverse set of Python functionalities. 📂 Organized into 18 distinct categories targeting various Python features. 🚢 Seamlessly manages the execution of containerized tools. 🔄 Efficiently transforms inferred types into a standardized format. 📊 Automatically produces meaningful metrics for in-depth assessment and comparison. 🛠️ Supported Tools Supported ✅ In-progress 🔧 Planned 💡 HeaderGen Intellij PSI Llama 2 Jedi Pyre ChatGPT Pyright PySonar2 HiTyper Pytype Scalpel TypeT5 Type4Py 🏆 TypeEvalPy Leaderboard Below is a comparison showcasing exact matches across different tools, coupled with top_n predictions for ML-based tools. Rank 🛠️ Tool Top-n Function Return Type Function Parameter Type Local Variable Type Total 1 HeaderGen 1 186 56 322 564 2 Jedi 1 122 0 293 415 3 Pyright 1 100 8 297 405 4 HiTyper 135 163173175 273737 179225229 369435441 5 HiTyper (static) 1 141 7 102 250 6 Scalpel 1 155 32 6 193 7 Type4Py 135 39103109 193131 99167174 157301314 (Auto-generated based on the the analysis run on 20-10-23 14:51) 🐳 Running with Docker 1️⃣ Clone the repo git clone https://github.com/ashwinprasadme/TypeEvalPy.git 2️⃣ Build Docker image docker build -t typeevalpy . 3️⃣ Run TypeEvalPy 🕒 Takes about 30mins on first run to build Docker containers. 📂 Results will be generated in the results folder within the root directory of the repository. Each results folder will have a timestamp, allowing you to easily track and compare different runs. docker run \ -v /var/run/docker.sock:/var/run/docker.sock \ -v ./results:/app/results \ typeevalpy 🔧 Optionally, run analysis on specific tools: docker run \ -v /var/run/docker.sock:/var/run/docker.sock \ -v ./results:/app/results \ typeevalpy --runners headergen scalpel 🛠️ Available options: headergen, pyright, scalpel, jedi, hityper, type4py, hityperdl

Motivation & Objective

• To address the lack of a unified, reproducible evaluation framework for Python type inference tools.
• To enable direct, fair comparison of type inference tools by standardizing input, execution, and output formats.
• To identify strengths and limitations of state-of-the-art tools across diverse Python language constructs.
• To highlight the impact of external type stubs and hybrid techniques on inference quality.
• To provide a foundation for future research in improving soundness and completeness in Python type inference.

Proposed method

• TypeEvalPy integrates 154 micro-benchmarks covering 18 Python 3.10 language features, each with targeted type annotations.
• It containerizes type inference tools to ensure isolated, reproducible execution environments.
• A translator module normalizes tool outputs into a standardized type format for consistent comparison.
• The framework computes key metrics including exact match rate, precision, recall, and soundness for quantitative evaluation.
• It supports both pure static analysis and machine learning-based tools, including top- $n$ predictions for ML models.
• The system enables detailed mismatch analysis to diagnose specific inference errors, such as incorrect callable type inference.

Experimental results

Research questions

• RQ1 How do different type inference tools perform across a comprehensive, standardized set of Python language constructs?
• RQ2 To what extent do external type stubs and hybrid approaches improve inference accuracy and completeness?
• RQ3 Why do even top-performing tools exhibit low soundness, and what are the primary sources of type inference errors?
• RQ4 How do static analysis and machine learning-based tools compare in handling complex constructs like function parameters and local variables?
• RQ5 What are the key failure patterns in type inference, particularly regarding callable types and function references?

Key findings

• HeaderGen achieved the highest performance in exact match rate, soundness, and completeness among all evaluated tools.
• Pyright and Jedi ranked second and third, respectively, with strong performance in built-ins and external library handling.
• HiTyper-DL, a hybrid model combining deep learning and static analysis, outperformed pure ML-based Type4Py but still achieved only 44% soundness.
• The deep learning component of HiTyper contributed only 15 of 154 sound results, significantly reducing completeness by 103 code snippets.
• Both Jedi and Pyright incorrectly inferred callable types as return types when functions were passed as arguments, a recurring flaw in 57 and 18 cases, respectively.
• Type4Py underperformed due to its reliance on training data and inability to generalize beyond seen patterns, especially for classes and exceptions.