[Paper Review] Using Natural Language Processing to Develop an Automated Orthodontic Diagnostic System

Tomoyuki Kajiwara, Chihiro Tanikawa+4 more|arXiv (Cornell University)|May 31, 2019

Biomedical Text Mining and OntologiesBiochemistry, Genetics and Molecular Biology6 references3 citations

TL;DR

This paper proposes an NLP-based automated orthodontic diagnostic system that extracts orthodontic problems from free-text dentist certificates and ranks them by treatment priority. Using a combination of bag-of-words, one-of-k, and Universal Sentence Encoder embeddings with SVM-rank, the system achieves an F1-score of 0.585 for problem detection and a Spearman correlation of 0.584 with human rankings, demonstrating feasibility for clinical decision support.

ABSTRACT

We work on the task of automatically designing a treatment plan from the findings included in the medical certificate written by the dentist. To develop an artificial intelligence system that deals with free-form certificates written by dentists, we annotate the findings and utilized the natural language processing approach. As a result of the experiment using 990 certificates, 0.585 F1-score was achieved for the task of extracting orthodontic problems from findings, and 0.584 correlation coefficient with the human ranking was achieved for the treatment prioritization task.

Motivation & Objective

• To develop an AI system that automates orthodontic diagnosis and treatment planning from free-form clinical notes written by dentists.
• To address the challenge of extracting meaningful orthodontic problems from unstructured, natural language descriptions in medical certificates.
• To prioritize treatment plans based on clinical severity, mimicking expert dentist reasoning.
• To reduce diagnostic errors and workload in orthodontic clinics by automating summarization and decision support.
• To explore the use of NLP and machine learning for clinical decision-making in dentistry.

Proposed method

• The system uses NLP to extract features from free-text medical certificates using bag-of-words (BoW), one-of-k (OoK), and Universal Sentence Encoder (USE) embeddings.
• Multi-label classification is performed using SVM and deep learning models to identify orthodontic problems from text, with 400+ medical condition labels.
• A learning-to-rank model based on SVM-rank is applied to prioritize extracted problems using vectorized problem descriptions.
• The system evaluates model performance using F1-score for problem detection and Spearman’s rank correlation for treatment prioritization.
• Sentence alignment and translation techniques are used to generate simplified language for patient communication, though text simplification is left for future work.
• Hyperparameters such as hidden layer size and regularization (C) are tuned on validation data for optimal performance.

Experimental results

Research questions

• RQ1 Can NLP techniques effectively extract orthodontic problems from unstructured, free-form clinical notes written by dentists?
• RQ2 How well can an automated system rank orthodontic problems by clinical priority compared to human experts?
• RQ3 Which embedding method—BoW, OoK, or USE—yields the best performance in diagnosing and prioritizing orthodontic conditions?
• RQ4 To what extent can machine learning models replicate the diagnostic reasoning of experienced orthodontists using only text data?
• RQ5 Can NLP-based systems reduce diagnostic workload and support less experienced dentists in clinical decision-making?

Key findings

• The system achieved an F1-score of 0.585 in detecting orthodontic problems from free-text findings, indicating moderate to strong performance in multi-label classification.
• The treatment prioritization model achieved a Spearman’s rank correlation coefficient of 0.584 with human rankings, demonstrating a strong positive correlation with expert judgment.
• The Universal Sentence Encoder (USE) model outperformed both BoW and OoK in treatment prioritization, achieving the highest correlation (0.584), likely due to richer semantic representation.
• The BoW model performed best in problem detection, possibly because it is less sensitive to sentence structure and incomplete phrasing common in clinical notes.
• The OoK model showed higher performance than BoW in prioritization (0.566 vs. 0.513), suggesting label-based features may better reflect clinical expertise.
• The results indicate that NLP-based systems can partially replicate expert diagnostic reasoning, supporting their use in clinical decision support.