[Paper Review] audino: A Modern Annotation Tool for Audio and Speech

Manraj Singh Grover, Pakhi Bamdev+4 more|arXiv (Cornell University)|Jun 9, 2020

Speech Recognition and Synthesis15 references4 citations

TL;DR

audino is a modern, web-based, open-source annotation tool for audio and speech that enables collaborative, server-side temporal segmentation, transcription, and labeling with centralized project and user management. It supports diverse NLP and speech tasks like ASR, VAD, and emotion recognition, and enables scalable, secure, and efficient dataset creation via Docker deployment and API integration, with successful use in transcribing 65 hours of L2 English speech for automated scoring systems.

ABSTRACT

In this paper, we introduce a collaborative and modern annotation tool for audio and speech: audino. The tool allows annotators to define and describe temporal segmentation in audios. These segments can be labelled and transcribed easily using a dynamically generated form. An admin can centrally control user roles and project assignment through the admin dashboard. The dashboard also enables describing labels and their values. The annotations can easily be exported in JSON format for further analysis. The tool allows audio data and their corresponding annotations to be uploaded and assigned to a user through a key-based API. The flexibility available in the annotation tool enables annotation for Speech Scoring, Voice Activity Detection (VAD), Speaker Diarisation, Speaker Identification, Speech Recognition, Emotion Recognition tasks and more. The MIT open source license allows it to be used for academic and commercial projects.

Motivation & Objective

• To address the lack of modern, collaborative, server-side audio annotation tools that support scalable, secure, and centralized management of speech datasets.
• To enable efficient and accurate temporal segmentation, transcription, and labeling of audio data for diverse speech and NLP tasks.
• To provide a production-ready, accessible, and extensible platform for both academic and commercial use through a permissive MIT license.
• To streamline large-scale dataset creation by supporting API-based data upload, pre-labeled data import, and automated quality tracking via inter-annotator agreement.
• To enhance data security and privacy through token-based authentication, hashed filenames, and role-based access control.

Proposed method

• The tool is built as a full-stack web application using a RESTful API and a React-based frontend, hosted on a server with centralized data storage.
• Audio files and annotations are stored in a central database, with user access controlled via JSON Web Tokens (JWT) and role-based permissions.
• Annotators interact with audio waveforms through a browser-based interface, using playback controls, zoom, and segment selection to define time-bound regions for labeling.
• A dynamic form is generated per annotation task, allowing input of transcriptions and selection of predefined labels (single or multi-choice) for each segment.
• Projects are managed via an admin dashboard that enables creation of users, roles, projects, labels, and API keys for secure data upload and integration.
• The system supports JSON export of annotations, enabling downstream use in machine learning pipelines, and integrates with pre-trained models for initial labeling via API.

Experimental results

Research questions

• RQ1 How can a modern, collaborative, and secure web-based annotation tool improve the scalability and efficiency of audio and speech dataset creation?
• RQ2 To what extent can centralized project and user management reduce errors and improve consistency in large-scale speech annotation projects?
• RQ3 Can a web-based tool with API integration and pre-labeled data import significantly reduce manual annotation effort in speech recognition and scoring tasks?
• RQ4 How effective is the tool in supporting diverse speech annotation tasks such as speaker diarization, emotion recognition, and ASR error correction?
• RQ5 What role does a centralized dashboard and secure authentication play in enhancing data privacy and access control during collaborative annotation?

Key findings

• audino successfully transcribed 65 hours of L2 English speech from simulated oral proficiency interviews, enabling fine-tuning of a pre-trained ASR model.
• The tool enabled two annotators to transcribe overlapping 20% of the data, allowing automated Word Error Rate (WER) tracking and error flagging through a CRON job.
• Inter-annotator agreement was monitored via automated WER calculation, and major transcription errors were resolved through discussion, improving data quality.
• The use of audino reduced manual effort in dataset preparation by allowing pre-labeled data import and iterative correction, accelerating the development of an automated oral proficiency scoring system.
• The tool's centralized architecture and role-based access control significantly improved data security and project management compared to client-side or offline tools.
• The project has been actively maintained and used in production for over a year, demonstrating reliability and extensibility for large-scale speech annotation tasks.