[Paper Review] Evidences of the mismatch between industry and academy on modelling language quality evaluation

Fáber D. Giraldo, Sergio España+1 more|arXiv (Cornell University)|Jun 7, 2016

Model-Driven Software Engineering Techniques22 references6 citations

TL;DR

This paper identifies a significant mismatch between industrial and academic perspectives on modeling language quality evaluation in Model-Driven Engineering (MDE). By analyzing 38 industrial and 14 academic sources, it reveals that industry prioritizes practical concerns like tool maturity, code quality, and integration complexity, while academia emphasizes theoretical frameworks, formal semantics, and scalability—highlighting a critical gap in alignment that hinders MDE adoption.

ABSTRACT

Quality is an implicit property of models and modelling languages by their condition of engineering artifacts. However, the quality property is affected by the diversity of conceptions around the model-driven paradigm. In this document is presented a report of quality issues on modelling languages and models. These issues result from an analysis about quality evidences obtained from industrial and academic/scientific contexts.

Motivation & Objective

• To investigate the disconnect between industrial and academic perspectives on modeling language quality evaluation in MDE.
• To identify and categorize key quality issues reported in industrial practice versus academic research.
• To analyze how differing priorities—such as tool maturity and code quality in industry versus formal semantics and scalability in academia—affect MDE adoption.
• To provide evidence-based insights into the practical and theoretical challenges impeding effective modeling language evaluation.
• To support the development of more aligned, realistic, and industry-relevant evaluation frameworks for modeling languages.

Proposed method

• Systematic extraction of quality issues from 38 industrial sources (journal papers, technical reports, web pages) and 14 academic sources (journal, conference, workshop papers).
• Categorization of issues into themes such as tooling complexity, model-code alignment, and scalability.
• Identification of explicit statements from each source that support the detected quality issues.
• Comparison of industrial and academic concerns through thematic analysis of cited works and their contextual statements.
• Use of citation analysis to trace the evolution of quality concerns across industrial and academic contexts.
• Mapping of issues to specific MDE challenges such as model evolution, tool immaturity, and lack of interoperability.

Experimental results

Research questions

• RQ1 What are the primary quality issues reported by industry practitioners in MDE adoption?
• RQ2 How do academic research communities frame and address modeling language quality issues compared to industrial practice?
• RQ3 What are the key discrepancies between industrial and academic priorities in evaluating modeling language quality?
• RQ4 To what extent do industrial concerns such as tool complexity and code quality align with academic research on model semantics and formal verification?
• RQ5 How do issues like model obsolescence, tool immaturity, and lack of interoperability impact MDE adoption across contexts?

Key findings

• Industrial practitioners report that MDE adoption is hindered by tool immaturity, high learning curves, and poor integration with existing development practices.
• A major industrial concern is the lack of alignment between models and generated code, with models frequently becoming obsolete or inconsistent with the implementation.
• Industry perceives modeling tools as 'heavyweight'—complex to install, learn, configure, and use—contributing to resistance in adoption.
• Academic research emphasizes formal semantics, behavioral modeling, and scalability for large-scale systems, but these concerns are underrepresented in industrial reports.
• There is a notable gap in addressing the operationalization of model quality frameworks, with academia calling for better support of time and behavioral semantics in metamodels.
• Several once-promising MDE tools, such as ArcStyler and OptimalJ, have declined due to technical and business challenges, underscoring the instability of tooling ecosystems.