[Paper Review] Understanding and representing the semantics of large structured documents

Muhammad Mahbubur Rahman, Tim Finin|arXiv (Cornell University)|Jul 24, 2018

Semantic Web and Ontologies7 references3 citations

TL;DR

This paper proposes a deep learning-based document ontology to automatically extract general-purpose semantic structures and domain-specific concepts from large structured documents like academic papers and RFPs. Using Variational and Convolutional Autoencoders for general structure and LDA with unigram, bigram, and phrase-based models for semantic concept extraction, the approach achieves high-quality, unsupervised semantic annotations that correlate strongly with document categories.

ABSTRACT

Understanding large, structured documents like scholarly articles, requests for proposals or business reports is a complex and difficult task. It involves discovering a document's overall purpose and subject(s), understanding the function and meaning of its sections and subsections, and extracting low level entities and facts about them. In this research, we present a deep learning based document ontology to capture the general purpose semantic structure and domain specific semantic concepts from a large number of academic articles and business documents. The ontology is able to describe different functional parts of a document, which can be used to enhance semantic indexing for a better understanding by human beings and machines. We evaluate our models through extensive experiments on datasets of scholarly articles from arXiv and Request for Proposal documents.

Motivation & Objective

• To develop an automated, unsupervised method for discovering the semantic structure of large, multi-themed documents such as scholarly articles and RFPs.
• To create a document ontology that captures both general-purpose functional sections (e.g., introduction, results) and domain-specific semantic concepts.
• To enable improved semantic indexing, content-based question answering, and machine understanding of complex documents through automated annotation.
• To overcome challenges in manual ontology construction by leveraging deep learning on large-scale document corpora.
• To evaluate the effectiveness of different NLP techniques—VAE, CAE, and LDA variants—for semantic representation and concept discovery.

Proposed method

• Trained Variational Autoencoders (VAE) and Convolutional Autoencoders (CAE) on section-level embeddings to learn general-purpose semantic structure of documents.
• Applied Latent Dirichlet Allocation (LDA) with three dictionary types: unigram, bigram, and phrase-based, to extract domain-specific semantic concepts.
• Used a filtering strategy to retain terms appearing in 20–10% of training sections, resulting in a final dictionary of 100,000 terms.
• Generated topic models with varying numbers of topics and passes, selecting top-probability terms as semantic concepts for each section.
• Visualized learned embeddings using t-SNE to assess clustering quality and manifold learning capability.
• Evaluated models on arXiv articles and RFP documents, comparing performance across different LDA configurations and document types.

Experimental results

Research questions

• RQ1 Can deep autoencoders effectively learn the general semantic structure of large, structured documents such as research papers and RFPs?
• RQ2 Which LDA variant—unigram, bigram, or phrase-based—produces more meaningful and coherent domain-specific semantic concepts for document sections?
• RQ3 To what extent do the extracted semantic concepts correlate with known document categories, such as arXiv subject classifications?
• RQ4 How well do the learned embeddings from VAE and CAE models capture semantic similarity and functional relationships between document sections?
• RQ5 Can the proposed ontology enable effective semantic annotation and improve downstream tasks like content-based retrieval and question answering?

Key findings

• The bigram-based LDA model outperformed unigram and phrase-based models in generating semantically meaningful and coherent topic terms for document sections.
• A strong correlation was observed between the top semantic terms extracted by LDA and the original arXiv subject categories, indicating high relevance and accuracy.
• VAE and CAE models successfully learned a low-dimensional manifold in the embedding space, enabling effective clustering of similar document sections.
• The system demonstrated robustness across diverse domains, including computer science, mathematics, nuclear theory, and astrophysics, with consistent semantic concept discovery.
• Embedding visualizations revealed that section headers in RFP documents, though sparse, still formed discernible clusters, indicating that semantic structure is learnable even with limited headers.
• The ontology and annotation pipeline were effective in capturing functional roles (e.g., introduction, results) and domain-specific concepts (e.g., 'phase', 'magnetic reconnection', 'deep learning') in an unsupervised manner.