Investigating EFL students' production of speech acts: A comparison of production questionnaires and role plays

The present study compares two popular measures of second language pragmatic competence: production questionnaires and role plays. Twelve Japanese university students representing three different English proficiency levels responded to both measures for the same four request and four refusal situations. Response length, range and content of the expressions, and native speaker evaluations of these responses were analyzed. The production questionnaire and role play elicited somewhat different production samples from the students. Role plays induced longer responses, and a larger number and greater variety of strategies/formulas, than production questionnaires. These differences appear to be caused by the interactive nature of role plays. Students often switched strategies for the same situations across different methods. Such intra-participant variations could have been missed if different participants had responded to different methods as in many previous studies. In addition, the correlation between the appropriateness scores of the two methods was not high enough to support the claim that they measured exactly the same trait. The low correlation probably resulted not only because the two methods produced different responses, but also because the role play responses provided additional audio-visual information, which might have affected the raters' evaluations. These findings suggest that production questionnaire scores cannot be simply substituted for role play scores.