Carol A. Chapelle - The Concise Encyclopedia of Applied Linguistics

The instruments discussed above represent a significant step in testing pragmatics by demonstrating that aspects of learners' pragmatic ability can be assessed practically and with satisfactory reliability. However, the speech act framework underlying these tests has come under severe criticism (Kasper, 2006) as it was based strongly on the discourse‐external context factors identified by Brown and Levinson (1987), atomized speech acts rather than considering them in their discursive context, and used DCTs, which have been shown to be highly problematic (Golato, 2003). This has led to the emergence of tests taking an interactional view.

In an early attempt to measure interactional abilities, Walters (2007, 2009) worked in a conversation analytic framework and attempted to measure test takers' receptive and productive knowledge of features of sequence organization and responses to social actions. He attained only very low reliabilities, illustrating the difficulty of measuring minute features of interaction.

In a groundbreaking study, Youn (2013, 2015) took a different approach. Employing an interactional competence perspective, she had 102 test takers of different proficiency levels perform two role plays with a trained interlocutor as well as a monologue. She scored performances on the following criteria:

content delivery: smooth and fluid turn taking;

language use: deployment of pragmalinguistic tools;

sensitivity to the situation: tailoring contributions to the recipient;

engagement with the interaction: displaying understanding of interlocutor talk;

turn organization: providing responses without excessive pausing.

A Rasch analysis showed that the test spread test takers out well and that the criteria functioned independently and were easy for raters to implement. Youn's study was a significant step forward as it was the first that clearly demonstrated the feasibility of assessing interactional competence.

Ikeda (2017) also investigated measurement of interactional competence but employed three role plays and three monologues with six rating criteria. Similar to Youn, he found a good spread of test takers and high inter‐rater reliability. There was significant overlap between scores on the monologic and dialogic tasks, raising the possibility of capturing a large amount of variance attributable to interactional competence with monologue tasks, which would greatly increase practicality.

Focusing on another aspect of interaction, Galaczi (2014) described differences in topic management, listener contributions, and turn‐taking management between learners at different levels of the Common European Framework of Reference (Council of Europe, 2001). She found that these interactional abilities improved with increasing proficiency and argued for their greater inclusion in rating scales. It must be noted that a feature like “topic management” was more likely to figure prominently in Galaczi's data, which involved test taker dyads discussing a set topic, than in Youn's and Ikeda's work, where interactions were based around requests.

Two other interaction‐focused assessment studies have been conducted which did not situate themselves in an interactional competence framework. Grabowski (2009, 2013) employed role plays and rated test taker performance based on criteria derived from Purpura's (2004) model of communicative language ability. Timpe (2013) employed Skype‐delivered role plays as part of a larger testing battery of intercultural competence (Byram, 1997). She scored test taker performance on two large holistic criteria, discourse management and pragmatic competence.

Fundamentally, tests of L2 pragmatics have the same requirements and pose the same development challenges as other language tests. They must be standardized to allow comparisons between test takers, they must be reliable to ensure precise measurement, they must be practical so that they do not overtax resources, and, above all, they must allow defensible inferences to be drawn from scores that can inform real‐world decisions (Messick, 1989; Kane, 2006). Some of these requirements are particularly difficult to meet for tests of pragmatics, which probably accounts for their very limited uptake.

Most importantly, practicality is a serious challenge for testing pragmatics. While some instruments in the speech act tradition were designed to be administered online and to allow automatic scoring (Roever, 2005; Itomitsu, 2009; Roever et al., 2014), tests under the interactional competence construct by their very nature include interaction and therefore currently require time and resource‐intensive involvement of a live interlocutor and scoring by raters. Work is underway to assess interaction through the use of intelligent agents backed by automatic speech recognition engines (Suendermann‐Oeft et al., 2017; Litman, Strik, & Lim, 2018) but this work is still in its infancy and requires nothing short of modeling language users' commonsense members' knowledge (Garfinkel, 1967), which is a daunting prospect. While other aspects of pragmatics, especially some pragmalinguistic abilities, are more easily measurable, it would be a case of serious construct underrepresentation to only include them and then argue that “pragmatics” as a whole is being measured. However, it would be much more feasible for tests that already include face‐to‐face speaking components, such as the International English Language Testing System (IELTS) or the American Council on the Teaching of Foreign Languages (ACTFL) oral proficiency interviews (OPI), to alter their tasks, procedures, and rating scales to measure interactional aspects of pragmatics.

The issue of practicality is further complicated by different types of interactional activities making different abilities visible. For example, two test takers discussing a set topic, as in Galaczi's (2014) study, will by necessity demonstrate their management of topical talk and allow conclusions as to relevant abilities, such as extending interlocutor contributions and managing topic changes. However, these abilities are much less transparent in role plays such as Youn's (2013, 2015), which are more suitable for making test takers' ability to do preference organization visible. This raises the specter that a test would need to involve several different interactional activities, compounding the practicality problem, though research will need to show whether conducting separate measurements of different interactional abilities is necessary.

However, even if the practicality issue can be resolved, measuring of interactional aspects of pragmatic competence is not an easy endeavor. Two related challenges are the co‐constructed nature of interaction (Jacoby & Ochs, 1995) and the standardization of the test. While tests need to be standardized to allow comparison between test taker performances, this is chronically difficult for spoken interactions, which have their own dynamic (Heritage, 1984; Kasper, 2006) and can unfold in unpredictable ways. Youn (2013, 2015) was the only one trying to address this problem by providing both the interlocutor and the test taker with an outline of the conversation. This makes the interaction somewhat more predictable and allows better comparison between different test takers, but it arguably distorts the construct since real‐world interactions are not usually scripted.

A significant amount of research is still necessary to understand how generalizable specific instances of role play performances in testing situations are across all possible performances, and to what extent they can be extrapolated to real‐world performances (Kane, 2006; Chapelle, Enright, & Jamieson, 2010). Findings like Ikeda's (2017) about the large degree of overlap between dialogic role play performances and monologue tasks are promising, and so is Okada's (2010) argument that abilities elicited through role plays are also relevant in real‐world interaction (though see Ewald, 2012, and Stokoe, 2013, for differences between role plays and real‐world talk). Still, comprehensive measurement of a complex construct such as interactional competence is one of the big challenges facing testing of L2 pragmatics.