[Paper Review] MetaAID: A Flexible Framework for Developing Metaverse Applications via AI Technology and Human Editing

Hongyin Zhu|arXiv (Cornell University)|Apr 4, 2022

Virtual Reality Applications and Impacts23 citations

TL;DR

MetaAID presents a lightweight, three-layer framework that combines AI technologies and human editing to support collaborative metaverse application development across multiple industries, demonstrated via five applications in entertainment, education, and consumption.

ABSTRACT

Achieving the expansion of domestic demand and the economic internal circulation requires balanced and coordinated support from multiple industries (domains) such as consumption, education, entertainment, engineering infrastructure, etc., which is indispensable for maintaining economic development. Metaverse applications may help with this task and can make many industries more interesting, more efficient, and provide a better user experience. The first challenge is that metaverse application development inevitably requires the support of various artificial intelligence (AI) technologies such as natural language processing (NLP), knowledge graph (KG), computer vision (CV), and machine learning (ML), etc. However, existing metaverse application development lacks a lightweight AI technology framework. This paper proposes a flexible metaverse AI technology framework metaAID that aims to support language and semantic technologies in the development of digital twins and virtual humans. The second challenge is that the development process of metaverse applications involves both technical development tasks and manual editing work, and often becomes a heavyweight multi-team collaboration project, not to mention the development of metaverse applications in multiple industries. Our framework summarizes common AI technologies and application development templates with common functional modules and interfaces. Based on this framework, we have designed 5 applications for 3 industries around the expansion of domestic demand and economic internal circulation. Experimental results show that our framework can support AI technologies when developing metaverse applications in different industries.

Motivation & Objective

• Motivate the need for a lightweight, collaborative AI stack to support metaverse applications spanning multiple industries.
• Propose a three-layer MetaAID framework (app development, AI technology, and human editing) to enable multi-team collaboration and reuse of templates.
• Create a repository of human–machine collaboration templates to accelerate agile content creation.
• Demonstrate the framework by developing five applications across three industries to show cross-domain applicability and efficacy.

Proposed method

• Define a three-layer framework architecture: app development layer (frontend/backend templates, data storage, deployment, and maintenance), AI technology layer (NLP, KG, algorithms, and data pipelines), and human editing layer (editing tools, standards, crawlers, and software).
• Assemble a technology stack by aggregating eight years of AI frameworks and solutions to address the accumulation challenge of the AI tech stack.
• Develop a repository of reusable human–machine templates and content templates to support rapid, high-quality editing and production.
• Implement five applications across three industries (entertainment, education, consumption) on multiple platforms (website, iOS, WeChat mini-program) to evaluate interoperability and practicality.
• Use digital twins and virtual humans as focal metaverse components, leveraging NLP, KG, CV/ML tools, and cloud services for deployment and O&M.

Experimental results

Research questions

• RQ1 How can a lightweight, flexible framework integrate AI technologies with human editing to support cross-industry metaverse application development?
• RQ2 Can a shared repository of templates and modular components streamline collaboration and reduce development weight across multiple industries?
• RQ3 Do five prototype applications built with MetaAID demonstrate the framework's adaptability across entertainment, education, and consumption sectors?

Key findings

• The framework enables AI-assisted development for metaverse applications across three industries.
• Five applications across three platforms were developed to validate the framework.
• NLP, KG, and ML components were successfully integrated with a human-editing layer to produce interactive metaverse content.
• A reusable template repository supports agile human–machine collaboration and content creation.
• Experimental setup and results indicate the framework can support multi-industry metaverse development with federated data analysis and inter-application collaboration.