[Paper Review] Crawling Twitter data through API: A technical/legal perspective

Shahab Saquib Sohail, Mohammad Muzammil Khan+5 more|arXiv (Cornell University)|May 22, 2021

Privacy, Security, and Data Protection31 references4 citations

TL;DR

This paper proposes a privacy-preserving framework for crawling and utilizing Twitter API data to enable personalized recommendations without compromising user privacy. By anonymizing Personally Identifiable Information (PII) through unique user codes and classifying data via affective computing, the system allows third-party services to access only aggregated, non-identifiable data, ensuring both effective recommendation systems and robust data protection—validated through 30 million tweets crawled in 8 days.

ABSTRACT

The popularity of the online media-driven social network relation is proven in today's digital era. The many challenges that these emergence has created include a huge growing network of social relations, and the large amount of data which is continuously been generated via the different platform of social networking sites, viz. Facebook, Twitter, LinkedIn, Instagram, etc. These data are Personally Identifiable Information (PII) of the users which are also publicly available for some platform, and others allow with some restricted permission to download it for research purposes. The users' accessible data help in providing with better recommendation services to users, however, the PII can be used to embezzle the users and cause severe detriment to them. Hence, it is crucial to maintain the users' privacy while providing their PII accessible for various services. Therefore, it is a burning issue to come up with an approach that can help the users in getting better recommendation services without their privacy being harmed. In this paper, a framework is suggested for the same. Further, how data through Twitter API can be crawled and used has been extensively discussed. In addition to this, various security and legal perspectives regarding PII while crawling the data is highlighted. We believe the presented approach in this paper can serve as a benchmark for future research in the field of data privacy.

Motivation & Objective

• To address the dual challenge of enabling effective recommendation systems while protecting user privacy in social media data.
• To establish a technical and legal framework for responsible Twitter API data crawling that respects user PII.
• To propose a model that balances data utility for research and services with stringent privacy safeguards.
• To provide a benchmark for future research in privacy-aware recommender systems using social media data.

Proposed method

• Crawling Twitter data using the Search API to collect 30 million tweets over 8 days, focusing on publicly available user content.
• Anonymizing user data by assigning unique codes to each user, replacing direct identifiers while preserving data utility.
• Classifying user data into categories (e.g., e-commerce, travel, social preferences) using affective computing techniques.
• Enabling third-party services to access only classified, non-identifiable data based on application-specific needs.
• Returning recommendations to users by re-mapping the anonymized code back to the original user, ensuring end-to-end privacy.
• Incorporating legal safeguards such as transparency, user consent, data deletion rights, and breach notification protocols.

Experimental results

Research questions

• RQ1 How can Twitter API data be collected and utilized for recommendation systems without violating user privacy?
• RQ2 What technical mechanisms can anonymize PII while preserving data utility for personalized services?
• RQ3 What legal and ethical guidelines are necessary to ensure responsible data access and usage in social media research?
• RQ4 How can a balance be achieved between detecting malicious users and protecting legitimate users’ privacy?
• RQ5 What framework can serve as a benchmark for privacy-aware data crawling and recommendation systems?

Key findings

• The framework successfully enables recommendation services using anonymized Twitter data without exposing direct user identities.
• A total of 30 million tweets were collected over 8 days using the Twitter Search API, demonstrating the feasibility of large-scale data crawling.
• Anonymization via user-specific codes allows data classification and third-party access while ensuring that no PII is exposed to external systems.
• The integration of affective computing enables the extraction of behavioral traits and preferences from public tweets for targeted recommendation purposes.
• Legal safeguards such as transparency, consent, data deletion rights, and breach notification were formally proposed as essential components of the system.
• The proposed model offers a balanced approach to distinguish between legitimate and malicious users without compromising user privacy or data utility.