[Paper Review] TACO: Topics in Algorithmic COde generation dataset

Rongni Li, Jie Fu+7 more|arXiv (Cornell University)|Dec 22, 2023

Software Engineering Research4 citations

TL;DR

TACO is a large-scale, open-source code generation dataset comprising 26,443 programming problems with fine-grained annotations for algorithms, skills, and difficulty levels, designed to improve and benchmark code generation models on complex, competition-level challenges. It enables significant performance gains in model fine-tuning, especially when specialized to specific algorithmic topics like sorting or dynamic programming.

ABSTRACT

We introduce TACO, an open-source, large-scale code generation dataset, with a focus on the optics of algorithms, designed to provide a more challenging training dataset and evaluation benchmark in the field of code generation models. TACO includes competition-level programming questions that are more challenging, to enhance or evaluate problem understanding and reasoning abilities in real-world programming scenarios. There are 25433 and 1000 coding problems in training and test set, as well as up to 1.55 million diverse solution answers. Moreover, each TACO problem includes several fine-grained labels such as task topics, algorithms, programming skills, and difficulty levels, providing a more precise reference for the training and evaluation of code generation models. The dataset and evaluation scripts are available on Hugging Face Hub (https://huggingface.co/datasets/BAAI/TACO) and Github (https://github.com/FlagOpen/TACO).

Motivation & Objective

• To address the limitations of existing code generation benchmarks, which are too easy, lack test set quality, and lack fine-grained annotations.
• To provide a more challenging and higher-quality evaluation benchmark for code generation models, especially for real-world programming reasoning and problem-solving.
• To enhance model performance by enabling fine-tuning on specific algorithmic topics using detailed annotations.
• To support diverse applications such as code comprehension, education, algorithm recommendation, and LLM enhancement through structured algorithmic metadata.

Proposed method

• Curated 26,443 programming problems from public datasets and open-source repositories, focusing on Python 3 solutions for cleaner syntax.
• Systematically categorized raw tags into 36 primary algorithmic topics (e.g., Dynamic Programming, Sorting) and added programming skill annotations.
• Annotated each problem with multiple labels: algorithm topic, programming skill, difficulty level, and time complexity.
• Constructed a training set of 25,433 problems and a test set of 1,000 problems, with up to 1.55 million diverse solution answers.
• Provided evaluation scripts and hosted the dataset on Hugging Face Hub and GitHub for open access and reproducibility.
• Enabled model fine-tuning using topic-specific subsets to improve performance on targeted algorithmic challenges.

Experimental results

Research questions

• RQ1 Can a large-scale, fine-grained code generation dataset improve model performance on complex, real-world programming tasks?
• RQ2 How does fine-tuning on topic-specific subsets of TACO affect code generation performance compared to full-dataset fine-tuning?
• RQ3 To what extent do fine-grained annotations (e.g., algorithm type, skill, difficulty) enhance model reasoning and code quality?
• RQ4 Can TACO serve as a reliable benchmark for evaluating code generation models beyond standard benchmarks like HumanEval or MBPP?
• RQ5 How effective is TACO in supporting downstream applications such as algorithm recommendation and code comprehension?

Key findings

• GPT-4 achieved a pass@1 score of 7% on Leetcode (hard) tasks, highlighting the limitations of current benchmarks and the need for more challenging evaluation sets like TACO.
• Fine-tuning models like StarCoder-1B on specific algorithmic skills (e.g., Data Structures, Greedy Algorithms, Sorting) using TACO’s training set led to measurable performance improvements over full-dataset fine-tuning.
• The TACO dataset’s fine-grained labels—such as algorithm topics, skills, and difficulty—enable more precise model training and evaluation, reducing ambiguity in performance assessment.
• The dataset’s rich annotation schema supports advanced applications, including automated code documentation, algorithm recommendation, and educational tooling.
• TACO outperforms existing benchmarks in capturing algorithmic diversity and complexity, offering a more realistic evaluation of model capabilities.
• Using TACO for evaluation is recommended over APPS or CodeContest due to potential false positives in test case coverage, which TACO mitigates through rigorous curation and manual verification.