[Paper Review] Taming the zoo - about algorithms implementation in the ecosystem of Apache Hadoop

Piotr Jan Dendek, Artur Czeczko+4 more|arXiv (Cornell University)|Mar 21, 2013

Data Quality and Management20 references3 citations

TL;DR

This paper presents CoAnSys, a Hadoop-based framework for large-scale text mining of scientific publications, implementing MapReduce-friendly algorithms for citation matching, document similarity, and classification. It demonstrates efficient big data processing through optimized workflows using HDFS, HBase, Pig, and Oozie, achieving scalable and modular knowledge extraction from scientific literature.

ABSTRACT

Content Analysis System (CoAnSys) is a research framework for mining scientific publications using Apache Hadoop. This article describes the algorithms currently implemented in CoAnSys including classification, categorization and citation matching of scientific publications. The size of the input data classifies these algorithms in the range of big data problems, which can be efficiently solved on Hadoop clusters.

Motivation & Objective

• To address the challenge of processing massive scientific publication datasets using scalable data mining techniques.
• To design a modular, reusable framework for implementing big data text mining algorithms within the Apache Hadoop ecosystem.
• To optimize performance and resource utilization in Hadoop workflows through efficient data serialization, storage, and workflow management.
• To enable researchers to implement and chain complex data transformations using standardized I/O interfaces and workflow orchestration.
• To share practical implementation patterns and best practices for MapReduce-based text mining in real-world scientific data pipelines.

Proposed method

• Employing Apache Hadoop and the MapReduce paradigm to distribute and parallelize text mining workloads across clusters.
• Using Protocol Buffers for compact, extensible serialization of data between modules and storage layers.
• Storing input data as HDFS SequenceFiles for high-performance I/O and output results in HBase via REST for persistent access.
• Implementing algorithms such as citation matching, document similarity, and classification using MapReduce jobs in Java, Pig, or Scala.
• Optimizing data transfer and memory usage by precomputing TF-IDF, using replicated, skewed, or merge joins in Pig, and minimizing intermediate data shuffling.
• Chaining data processing pipelines using Apache Oozie for workflow orchestration, ensuring reproducibility and execution tracking.

Experimental results

Research questions

• RQ1 How can citation matching be efficiently implemented at scale using the MapReduce paradigm in a Hadoop cluster?
• RQ2 What are the performance bottlenecks in document similarity and classification pipelines, and how can they be mitigated in a big data environment?
• RQ3 Which data storage and serialization strategies (HDFS vs. HBase, Protocol Buffers) yield optimal performance for large-scale text mining workflows?
• RQ4 How can workflow management tools like Apache Oozie improve the reliability and maintainability of complex data processing pipelines in Hadoop?
• RQ5 What are the key implementation patterns and optimizations that enable efficient execution of text mining algorithms on Hadoop?

Key findings

• The CoAnSys framework successfully enables scalable processing of scientific publication data using Hadoop, with algorithms like citation matching and document classification achieving high throughput on large datasets.
• Using HDFS SequenceFiles instead of HBase for intermediate data storage significantly improves performance and avoids task failure chains due to timeout under high load.
• Protocol Buffer serialization enables efficient, extensible, and type-safe data exchange between modules, improving data integrity and reducing I/O overhead.
• Optimizing the order of operations—such as precomputing TF-IDF and grouping data transformations early—reduces data transfer and improves overall workflow efficiency.
• Apache Oozie outperforms scripting alternatives like Bash or Python for workflow management due to built-in execution tracking, persistence, and integration with Hadoop.
• Specialized join types in Pig (e.g., replicated, skewed, merge joins) significantly enhance performance when joining large datasets with smaller reference sets.