[Paper Review] Interpreting Cloud Computer Vision Pain-Points: A Mining Study of Stack Overflow

Alex Cummaudo, Rajesh Vasa+3 more|arXiv (Cornell University)|Jan 27, 2020

Software Engineering Research4 citations

TL;DR

This study analyzes developer pain points in cloud-based computer vision APIs by mining Stack Overflow questions, using classification taxonomies to reveal that developers struggle with shallow understanding of underlying AI mechanics and inadequate documentation. Unlike mature domains, questions reflect basic debugging issues rather than deep technical engagement, indicating a need for improved education and documentation in AI-powered services.

ABSTRACT

Intelligent services are becoming increasingly more pervasive; application developers want to leverage the latest advances in areas such as computer vision to provide new services and products to users, and large technology firms enable this via RESTful APIs. While such APIs promise an easy-to-integrate on-demand machine intelligence, their current design, documentation and developer interface hides much of the underlying machine learning techniques that power them. Such APIs look and feel like conventional APIs but abstract away data-driven probabilistic behaviour - the implications of a developer treating these APIs in the same way as other, traditional cloud services, such as cloud storage, is of concern. The objective of this study is to determine the various pain-points developers face when implementing systems that rely on the most mature of these intelligent services, specifically those that provide computer vision. We use Stack Overflow to mine indications of the frustrations that developers appear to face when using computer vision services, classifying their questions against two recent classification taxonomies (documentation-related and general questions). We find that, unlike mature fields like mobile development, there is a contrast in the types of questions asked by developers. These indicate a shallow understanding of the underlying technology that empower such systems. We discuss several implications of these findings via the lens of learning taxonomies to suggest how the software engineering community can improve these services and comment on the nature by which developers use them.

Motivation & Objective

• To identify and categorize the types of challenges developers face when using cloud-based computer vision APIs.
• To investigate whether existing classification taxonomies for Stack Overflow posts can be applied to understand pain points in intelligent services.
• To compare the nature of questions about computer vision APIs with those from more mature domains like mobile development to highlight differences in developer understanding.
• To explore how learning taxonomies (SOLO and Bloom’s) can provide insight into the cognitive depth of developer questions about AI-powered APIs.
• To inform improvements in API design, documentation, and software engineering education for intelligent services.

Proposed method

• The study mined 1,024 Stack Overflow posts related to cloud computer vision services using targeted keywords and filtering criteria.
• Posts were classified using two existing taxonomies: a 2018 taxonomy for documentation-related vs. general questions, and a 2019 taxonomy based on SOLO and Bloom’s learning levels.
• Classification was performed by the primary author, with a rigorous filtering and reliability assessment process to mitigate subjectivity.
• The analysis focused on identifying patterns in question types, particularly the cognitive level of inquiry (e.g., recall vs. evaluation) using Bloom’s taxonomy.
• The study compared the distribution of question types in computer vision with those in established domains like mobile development to highlight disparities.
• A qualitative lens was applied to interpret findings through the frameworks of learning taxonomies, aiming to understand the depth of developer understanding.

Experimental results

Research questions

• RQ1 What types of pain points do developers commonly express when using cloud-based computer vision APIs on Stack Overflow?
• RQ2 How do the cognitive levels of developer questions about computer vision APIs compare to those about more mature software domains like mobile development?
• RQ3 To what extent do existing classification taxonomies for Stack Overflow posts apply to questions about intelligent services such as computer vision?
• RQ4 What are the dominant themes in developer complaints regarding documentation quality and completeness for AI-powered APIs?
• RQ5 How can learning taxonomies (SOLO and Bloom’s) help interpret the depth of understanding reflected in developer questions about intelligent services?

Key findings

• Developers frequently ask shallow, basic debugging questions about computer vision APIs, indicating limited understanding of the underlying machine learning mechanisms.
• A significant proportion of questions are classified as 'documentation-related', suggesting that clarity, completeness, and correctness of API documentation are major pain points.
• The distribution of question types in computer vision differs markedly from that in mature domains like mobile development, where higher-order cognitive questions (e.g., analysis, evaluation) are more common.
• The application of Bloom’s taxonomy reveals that most questions are at the 'remember' and 'understand' levels, with few reaching 'apply', 'analyze', or 'evaluate'—indicating a lack of deep technical engagement.
• The study identifies a mismatch between the perceived simplicity of RESTful AI APIs and the actual complexity of their probabilistic, data-driven behavior, which developers often fail to grasp.
• The authors suggest that current documentation and developer education fail to address the non-deterministic and context-sensitive nature of AI-powered services, leading to misuse and frustration.