Carol A. Chapelle - The Concise Encyclopedia of Applied Linguistics

From a test design perspective, it is also important to know what makes items difficult so they can be targeted at test takers at different ability levels. This is a challenge for many pragmatics tests, which tend to not have sufficient numbers of difficult items, and it is true for tests in the speech act tradition and assessing interactional competence. For example, Roever et al.'s (2014) battery was overall easy for test takers, and so were Youn's (2013, 2015) and Ikeda's (2017) instruments. We know relatively little about what makes items or tasks difficult, though Roever (2004) put forward some suggestions for pragmalinguistically oriented tests. For measures of interactional competence, it might be worth trying interactional tasks that require orientation to conflicting social norms, for example, managing status incongruent talk as a student interacting with a professor under institutional expectations of initiative (Bardovi‐Harlig & Hartford, 1993), or in a workplace situation persuading one's boss to remove his son from one's project team (Ross, 2017). However, much more research is needed here as well.

A challenge specific to tests using sociopragmatic judgment is establishing a baseline. Put simply, testers need a reliable way to determine correct and incorrect test taker responses. The usual way to do so is to use a native‐speaker standard and this has been shown to work well for binary judgments of correct/incorrect, appropriate/inappropriate, and so on (Bardovi‐Harlig & Dörnyei, 1998; Schauer, 2006). However, native‐speaker benchmarking is much more problematic when it comes to preference judgments. For example, in Matsumura's (2001) benchmarking of his multiple‐choice items on the appropriateness of advice, there was not a single item where 70% of a native‐speaker benchmarking group (N = 71) agreed on the correct response, and only 2 items (out of a pretest and posttest total of 24) where more than 60% of native speakers agreed. On 10 items, the most popular response option was chosen by less than half the native‐speaker group. Roever et al. (2014) found stronger NS agreements with all their items showing at least 50% agreement among NS, and they assigned 2 points for test taker responses that were chosen by the largest NS group and 1 point for responses chosen by the next 2 largest groups, provided they were at least 10% of the NS sample. This scoring approach tried to take into account NS preference but the point distribution is essentially a tester decision with little empirical basis.

Finally, tests of sociopragmatics have often been designed contrastively for a pair of languages, for example, native Japanese speakers learning English (Hudson et al., 1995), native English speakers learning Japanese (Yamashita, 1996), native English speakers learning Korean (Ahn, 2005), or native Chinese speakers learning English (Liu, 2006). This necessarily lowers the practicality of tests, as well as the likelihood that they will eventually become part of large‐scale international test batteries (like TOEFL or IELTS). Roever (2005) did not limit his test taker population to a specific L1, and used differential item functioning to show that there were some L1 effects but that they were generally minor (Roever, 2007), indicating that limiting pragmatics tests to a specific population is not a necessity.

Tests of L2 pragmatics have seen a great deal of development and focused research in the last two decades. They offer a promising addition to the traditional language tests, which tend to focus on grammar, vocabulary, and skills. However, they pose significant challenges for test design if a complex construct like pragmatics is to be assessed comprehensively and reliably. There is still a great deal of research required.

SEE ALSO:Assessment of Speaking; Paired and Group Oral Assessment

1 Ahn, R. C. (2005). Five measures of interlanguage pragmatics in KFL (Korean as a foreign language) learners (Unpublished PhD thesis). University of Hawai'i at Manoa.

2 Austin, J. L. (1962). How to do things with words. Oxford, England: Oxford University Press.

3 Bachman, L. F., & Palmer, A. S. (1996). Language testing in practice: Designing and developing useful language tests. Oxford, England: Oxford University Press.

4 Bardovi‐Harlig, K., & Dörnyei, Z. (1998). Do language learners recognize pragmatic violations? Pragmatic versus grammatical awareness in instructed L2 learning. TESOL Quarterly, 32, 233–62.

5 Bardovi‐Harlig, K., & Hartford, B. (1993). Learning the rules of academic talk: A longitudinal study of pragmatic development. Studies in Second Language Acquisition, 15, 279–304.

6 Bouton, L. F. (1988). A cross‐cultural study of ability to interpret implicatures in English. World Englishes, 17, 183–96.

7 Bouton, L. F. (1994). Conversational implicature in the second language: Learned slowly when not deliberately taught. Journal of Pragmatics, 22, 157–67.

8 Bouton, L. F. (1999). Developing non‐native speaker skills in interpreting conversational implicatures in English: Explicit teaching can ease the process. In E. Hinkel (Ed.), Culture in second language teaching and learning (pp. 47–70). Cambridge, England: Cambridge University Press.

9 Brown, J. D. (2001). Six types of pragmatics tests in two different contexts. In K. Rose & G. Kasper (Eds.), Pragmatics in language teaching (pp. 301–25). New York, NY: Cambridge University Press.

10 Brown, J. D. (2008). Raters, functions, item types and the dependability of L2 pragmatics tests. In E. Alcón Soler & A. Martínez‐Flor (Eds.), Investigating pragmatics in foreign language learning, teaching and testing (pp. 224–48). Clevedon, England: Multilingual Matters.

11 Brown, J. D., & Ahn, R. C. (2011). Variables that affect the dependability of L2 pragmatics tests. Journal of Pragmatics, 43(1), 198–217.

12 Brown, P., & Levinson, S. D. (1987). Politeness: Some universals in language usage. Cambridge, England: Cambridge University Press.

13 Byram, M. (1997). Teaching and assessing intercultural communicative competence. Philadelphia, PA: Multilingual Matters.

14 Canale, M. (1983). From communicative competence to communicative language pedagogy. In J. Richards & R. Schmidt (Eds.), Language and communication (pp. 2–27). London, England: Longman.

15 Canale, M., & Swain, M. (1980). Theoretical bases of communicative approaches to second language teaching and testing. Applied Linguistics, 1, 1–47.

16 Chapelle, C. A., Enright, M. K., & Jamieson, J. (2010). Does an argument‐based approach to validity make a difference? Educational Measurement: Issues and Practice, 29(1), 3–13.

17 Clift, R. (2016). Conversation analysis. Cambridge, England: Cambridge University Press.

18 Council of Europe. (2001). Common European framework of reference for languages: Learning, teaching, assessment. Cambridge, England: Cambridge University Press.

19 Crystal, D. (1997). A dictionary of linguistics and phonetics. Oxford, England: Blackwell.

20 Ewald, J. D. (2012). “Can you tell me how to get there?”: Naturally occurring versus role‐play data in direction giving. Pragmatics, 22(1), 79–102.

21 Galaczi, E. (2014). Interactional competence across proficiency levels: How do learners manage interaction in paired speaking tests? Applied Linguistics, 35(5), 553–74.

22 Galaczi, E., & Taylor, L. (2018). Interactional competence: Conceptualisations, operationalisations, and outstanding questions. Language Assessment Quarterly, 15(3), 219–36.

23 Garfinkel, H. (1967). Studies in ethnomethodology. Englewood Cliffs, NJ: Prentice Hall.

24 Golato, A. (2003). Studying compliment responses: A comparison of DCTs and recordings of naturally occurring talk. Applied Linguistics, 24(1), 90–121.