TY - GEN
T1 - Stay on-topic
T2 - 23rd European Symposium on Research in Computer Security, ESORICS 2018
AU - Juuti, Mika
AU - Sun, Bo
AU - Mori, Tatsuya
AU - Asokan, N.
N1 - Publisher Copyright:
© Springer Nature Switzerland AG 2018.
PY - 2018
Y1 - 2018
N2 - Automatically generated fake restaurant reviews are a threat to online review systems. Recent research has shown that users have difficulties in detecting machine-generated fake reviews hiding among real restaurant reviews. The method used in this work (char-LSTM) has one drawback: it has difficulties staying in context, i.e. when it generates a review for specific target entity, the resulting review may contain phrases that are unrelated to the target, thus increasing its detectability. In this work, we present and evaluate a more sophisticated technique based on neural machine translation (NMT) with which we can generate reviews that stay on-topic. We test multiple variants of our technique using native English speakers on Amazon Mechanical Turk. We demonstrate that reviews generated by the best variant have almost optimal undetectability (class-averaged F-score 47%). We conduct a user study with experienced users and show that our method evades detection more frequently compared to the state-of-the-art (average evasion 3.2/4 vs 1.5/4) with statistical significance, at level α1% (Sect. 4.3). We develop very effective detection tools and reach average F-score of 97% in classifying these. Although fake reviews are very effective in fooling people, effective automatic detection is still feasible.
AB - Automatically generated fake restaurant reviews are a threat to online review systems. Recent research has shown that users have difficulties in detecting machine-generated fake reviews hiding among real restaurant reviews. The method used in this work (char-LSTM) has one drawback: it has difficulties staying in context, i.e. when it generates a review for specific target entity, the resulting review may contain phrases that are unrelated to the target, thus increasing its detectability. In this work, we present and evaluate a more sophisticated technique based on neural machine translation (NMT) with which we can generate reviews that stay on-topic. We test multiple variants of our technique using native English speakers on Amazon Mechanical Turk. We demonstrate that reviews generated by the best variant have almost optimal undetectability (class-averaged F-score 47%). We conduct a user study with experienced users and show that our method evades detection more frequently compared to the state-of-the-art (average evasion 3.2/4 vs 1.5/4) with statistical significance, at level α1% (Sect. 4.3). We develop very effective detection tools and reach average F-score of 97% in classifying these. Although fake reviews are very effective in fooling people, effective automatic detection is still feasible.
UR - http://www.scopus.com/inward/record.url?scp=85052238662&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85052238662&partnerID=8YFLogxK
U2 - 10.1007/978-3-319-99073-6_7
DO - 10.1007/978-3-319-99073-6_7
M3 - Conference contribution
AN - SCOPUS:85052238662
SN - 9783319990729
T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
SP - 132
EP - 151
BT - Computer Security - 23rd European Symposium on Research in Computer Security, ESORICS 2018, Proceedings
A2 - Lopez, Javier
A2 - Zhou, Jianying
A2 - Soriano, Miguel
PB - Springer Verlag
Y2 - 3 September 2018 through 7 September 2018
ER -