Careless participants are essential for our phishing study: Understanding the impact of screening methods

Tenga Matsuura, Ayako A. Hasegawa, Mitsuaki Akiyama, Tatsuya Mori

Research output: Chapter in Book/Report/Conference proceedingConference contribution

3 Citations (Scopus)


Online surveys using crowdsourcing services have been widely adopted in academic research projects aimed at understanding human perception and behavior. Because there is a concern that online surveys may include dishonest or careless responses by crowdworkers who perform a large number of tasks, or responses by bots, several screening methods have been proposed to discard such low-quality responses. However, in security research, especially in phishing research where the attention of participants is considered to influence the results, the elimination of careless responses may lead to the removal of participants who should be included in the research. In this study, we address the following research question: "Does the adoption of existing screening methods bias the results of security surveys?"Using Amazon Mechanical Turk and Prolific Academic, two popular crowdsourcing platforms used in online surveys, we conducted online user studies (N = 600) on security knowledge, security behavior, and phishing email detection performance to elucidate the influence of screening methods on the results. The obtained results indicate that the adoption of the instructional manipulation check (IMC) screening method triggers bias in the demographics of the participants, as well as differences in the results of phishing email detection performance. In addition, the degree of these differences depends on the crowdsourcing platform. We also demonstrated that it is non-trivial to determine the correlation between screening methods and factors that can influence the results of a survey on security behavior. These findings suggest that caution should be exercised when applying screening methods such as attention checks and IMC in studies where the extent of user attention could have a significant impact on the results.

Original languageEnglish
Title of host publicationProceedings - EuroUSEC 2021
Subtitle of host publication2021 European Symposium on Usable Security
PublisherAssociation for Computing Machinery
Number of pages12
ISBN (Electronic)9781450384230
Publication statusPublished - 2021 Oct 11
Event2021 European Symposium on Usable Security, EuroUSEC 2021 - Virtual, Online, Germany
Duration: 2021 Oct 112021 Oct 12

Publication series

NameACM International Conference Proceeding Series


Conference2021 European Symposium on Usable Security, EuroUSEC 2021
CityVirtual, Online


  • Attention check
  • Crowdsourcing
  • Instruction manipulation check
  • Phishing

ASJC Scopus subject areas

  • Software
  • Human-Computer Interaction
  • Computer Vision and Pattern Recognition
  • Computer Networks and Communications


Dive into the research topics of 'Careless participants are essential for our phishing study: Understanding the impact of screening methods'. Together they form a unique fingerprint.

Cite this