[Paper Review] To Walk or Not to Walk: Crowdsourced Assessment of External Vehicle-to-Pedestrian Displays

Lex Fridman, Bruce Mehler+4 more|arXiv (Cornell University)|Jul 10, 2017

Human-Automation Interaction and Safety10 references90 citations

TL;DR

The paper presents a scalable crowdsourced online method (via Amazon Mechanical Turk) to evaluate 30 external vehicle-to-pedestrian display concepts, using 200 participants to assess whether each design communicates a safe-to-cross intent. It demonstrates the method’s efficiency for early-stage design evaluation and discusses interpretation variability and safety implications.

ABSTRACT

Researchers, technology reviewers, and governmental agencies have expressed concern that automation may necessitate the introduction of added displays to indicate vehicle intent in vehicle-to-pedestrian interactions. An automated online methodology for obtaining communication intent perceptions for 30 external vehicle-to-pedestrian display concepts was implemented and tested using Amazon Mechanic Turk. Data from 200 qualified participants was quickly obtained and processed. In addition to producing a useful early-stage evaluation of these specific design concepts, the test demonstrated that the methodology is scalable so that a large number of design elements or minor variations can be assessed through a series of runs even on much larger samples in a matter of hours. Using this approach, designers should be able to refine concepts both more quickly and in more depth than available development resources typically allow. Some concerns and questions about common assumptions related to the implementation of vehicle-to-pedestrian displays are posed.

Motivation & Objective

• Assess whether external vehicle-to-pedestrian displays effectively communicate crossing intent to pedestrians.
• Demonstrate a scalable online methodology for evaluating many design concepts quickly and cost-effectively.
• Identify design elements that reliably convey walk/don’t-walk signals to pedestrians.
• Highlight limitations and safety concerns associated with external displays in autonomous or semi-autonomous vehicles.

Proposed method

• Use MTurk to recruit 200 experienced workers to view 30 animated display concepts over a base vehicle image.
• Employ catch stimuli to filter for attentive, reliable responses and ensure data quality.
• Present stimuli as 1280x720 images with randomized ordering and indefinite animation until response.
• Require participants to judge if it is safe to cross (Yes/Not Sure/No) from a pedestrian perspective.
• Analyze proportion of respondents who interpret each design as safe to walk and compare to designer intent.
• Provide a scalable Python/HTML/Ajax framework with PostgreSQL backend for rapid data collection.

Experimental results

Research questions

• RQ1 Can crowdsourced online testing reliably evaluate external vehicle-to-pedestrian display concepts?
• RQ2 Which designs clearly communicate the intent to walk or not walk to pedestrians?
• RQ3 How much do interpretations align with designers’ intended messages across a large set of concepts?
• RQ4 What are the limitations of using online crowdsourcing for safety-critical vehicle-pedestrian signals?

Key findings

• Some designs (e.g., designs 2 and 10) matched the intended not-to-walk signal well across participants.
• Several walk-oriented designs (e.g., designs 1 and 7) were interpreted as safe to walk by a high portion of participants, but universal agreement was not observed.
• Half or more of participants found eight of the twenty walk-oriented designs unclear or misinterpreted as don’t walk.
• The two don’t-walk designs fared better but still did not achieve universal unambiguous interpretation.
• The method demonstrates rapid, cost-effective testing of many design variations within hours, enabling early-stage refinement.
• The study cautions that misinterpretation and mixed populations could impact safety during a transition to external displays.