[Paper Review] Open Brain AI. Automatic Language Assessment

Charalambos Themistocleous|arXiv (Cornell University)|Jun 11, 2023

Artificial Intelligence in Healthcare and Education4 citations

TL;DR

Open Brain AI is a free, automated computational platform that leverages machine learning, natural language processing, and large language models to analyze multilingual spoken and written language productions for clinical language assessment. It enables fast, accurate, and scalable evaluation of discourse macro- and micro-structure, reducing clinician workload and improving efficiency in diagnosing and treating neurogenic communication disorders.

ABSTRACT

Language assessment plays a crucial role in diagnosing and treating individuals with speech, language, and communication disorders caused by neurogenic conditions, whether developmental or acquired. However, current assessment methods are manual, laborious, and time-consuming to administer and score, causing additional patient stress. To address these challenges, we developed Open Brain AI (https://openbrainai.com). This computational platform harnesses innovative AI techniques, namely machine learning, natural language processing, large language models, and automatic speech-to-text transcription, to automatically analyze multilingual spoken and written speech productions. This paper discusses the development of Open Brain AI, the AI language processing modules, and the linguistic measurements of discourse macro-structure and micro-structure. The fast and automatic analysis of language alleviates the burden on clinicians, enabling them to streamline their workflow and allocate more time and resources to direct patient care. Open Brain AI is freely accessible, empowering clinicians to conduct critical data analyses and give more attention and resources to other critical aspects of therapy and treatment.

Motivation & Objective

• To address the limitations of manual, time-consuming language assessments in clinical settings.
• To develop an automated, scalable, and accessible platform for assessing language production in individuals with neurogenic speech and language disorders.
• To reduce clinician burden by enabling fast, accurate, and objective analysis of linguistic features in spoken and written language.
• To support clinicians in focusing more on direct patient care by automating data analysis and scoring.
• To provide a freely accessible tool that enhances clinical workflow and supports evidence-based therapy decisions.

Proposed method

• The platform integrates automatic speech recognition (ASR) for transcribing spoken language into text.
• It employs natural language processing (NLP) pipelines to extract linguistic features from transcribed speech and written text.
• Machine learning models are trained to assess discourse macro-structure (e.g., coherence, organization) and micro-structure (e.g., syntax, lexical diversity).
• Large language models (LLMs) are utilized to analyze semantic content, fluency, and complexity of language productions.
• The system processes multilingual inputs, supporting diverse clinical populations.
• A web application interface enables clinicians to upload and analyze language samples with real-time feedback and structured output.

Experimental results

Research questions

• RQ1 Can automated AI systems reliably assess discourse macro-structure in multilingual spoken and written language samples from individuals with neurogenic language disorders?
• RQ2 How accurately can machine learning and NLP techniques detect linguistic impairments in clinical language production compared to manual assessment?
• RQ3 To what extent does integrating LLMs improve the sensitivity and specificity of automated language assessment in clinical settings?
• RQ4 Can an open-access, automated platform reduce clinician workload while maintaining diagnostic reliability?
• RQ5 How does the platform perform across diverse linguistic and cultural backgrounds in multilingual populations?

Key findings

• The Open Brain AI platform enables fully automatic analysis of language samples, significantly reducing time and effort required for clinical assessment.
• The system successfully extracts and evaluates both discourse macro-structure and micro-structure features from spoken and written language with high consistency.
• Clinicians can now access objective, data-driven linguistic measurements in real time, improving diagnostic accuracy and treatment planning.
• The platform is freely accessible and publicly available via a web application, promoting equitable access to advanced language assessment tools.
• By automating labor-intensive scoring tasks, the system allows clinicians to redirect time and attention toward direct patient care and therapeutic interventions.
• The integration of ASR, NLP, and LLMs enables robust, multilingual language analysis suitable for diverse clinical populations.