Carol A. Chapelle - The Concise Encyclopedia of Applied Linguistics

Selecting, training, and monitoring raters is a central aspect of writing assessment. Useful procedures for rater training and monitoring can be found in White (1994), Weigle (2002), and Shaw and Weir (2007). Recent research has focused on the effects of rater background and training on scores; see Lumley (2005), Barkaoui (2007), and Eckes (2008) for summaries. This research suggests that training can mitigate but not eliminate differences in rater severity and consistency due to background variables. For this reason, many programs have begun using test analysis tools such as multifaceted Rasch measurement (MFRM) to adjust scores for differences between raters (see McNamara, 1996, for an introduction to MFRM, and Schaefer, 2008, for a review of studies in writing assessment that have used this approach).

Another important development with regards to scoring is the use of automated essay scoring (AES) systems such as e‐rater® , developed by Educational Testing Service (Attali & Burstein, 2006) and IntelliMetric™ and MY Access!® developed by Vantage Learning Systems (Elliott, 2003), in part to contain the costs and time involved in scoring large‐scale writing assessments. Research demonstrates that automated systems are at least as reliable in scoring standard essay tests as humans (see Shermis & Burstein, 2003; Dikli, 2006; and Shermis, 2014, for overviews of automated essay scoring). However, the use of AES systems is controversial; many writing instructors, in particular, are opposed to any machine scoring of writing, while others highlight the speed and reliability of AES systems as advantages. A recent review of the arguments in this area can be found in Ockey (2009).

Assessment specialists have focused on designing tasks that reflect real‐world writing more accurately while maintaining rigorous standards for scoring. It has long been recognized that a single timed impromptu essay is inadequate for assessing writing (see, for example, Behizadeh, 2014). The trend in large‐scale writing tests is toward multiple tasks and new genres. For example, the TOEFL iBT® includes both an independent task and an integrated writing task that is based on listening and reading (Chapelle, Enright, & Jamieson, 2008), while the Cambridge suite of exams has introduced e‐mail tasks in some tests (Shaw & Weir, 2007). These less traditional tasks are bound to stimulate new validity research as they become more widely used.

At the same time the introduction of more tasks and the need to maintain rigorous standards in scoring add to the human and financial costs of producing and scoring writing assessments. AES systems are an important way to help contain such costs; thus, the ongoing debate about the use of automated scoring is another important issue that is certain to be a focus of debate in the foreseeable future.

The importance of writing in the business world and the growth of instant global communication may also mean that new assessments for business writing will need to be developed that are specifically tailored to the genres and tasks of international business communities. This is an additional area of potential growth in writing assessment (see Katz, Haras, & Blaszczynski, 2010, for an overview of the role of writing in business). Similarly, globalization and the extensive use of different varieties of English make it imperative for language testing to grapple with issues of World Englishes in writing assessment (Brown, 2014).

This entry has demonstrated the complexity of assessing writing as well as the opportunities that have emerged in an age of globalized electronic communication.

SEE ALSO:Assessment in the Classroom; Assessment of Integrated Skills; English for Academic Purposes; Rating Scales and Rubrics in Language Assessment; Task‐Based Language Assessment; Uses of Language Assessments; Washback in Language Assessment; Writing and Language for Specific Purposes

1 Attali, Y., & Burstein, J. (2006). Automated essay scoring with e‐rater version 2.0. Journal of Technology, Learning, and Assessment, 4(3).

2 Barkaoui, K. (2007). Participants, texts, and processes in second language writing assessment: A narrative review of the literature. The Canadian Modern Language Review, 64, 97–132.

3 Behizadeh, N. (2014). Mitigating the dangers of a single story: Creating large‐scale writing assessments aligned with sociocultural theory. Educational Researcher, 43(3), 125–36.

4 Brown, J. D. (2014). The future of world Englishes in language testing. Language Assessment Quarterly, 11(1), 5–26.

5 Chapelle, C., Enright, M., & Jamieson, J. (Eds.). (2008). Building a validity argument for the Test of English as a Foreign Language. New York, NY: Routledge.

6 Dikli, S. (2006). An overview of automated scoring of essays. Journal of Technology, Learning, and Assessment, 5(1).

7 Eckes, T. (2008). Rater types in writing performance assessments: A classification approach to rater variability. Language Testing, 25, 155–85.

8 Educational Testing Service. (2004). TOEFL iBT Test: Integrated writing rubrics. Retrieved April 25, 2019 from https://www.ets.org/Media/Tests/TOEFL/pdf/Writing_Rubrics.pdf

9 Elliott, S. (2003). IntelliMetric: From here to validity. In M. D. Shermis & J. C. Burstein (Eds.), Automated essay scoring: A cross‐disciplinary perspective (pp. 67–81). Mahwah, NJ: Erlbaum.

10 Green, A. (2014). Exploring language assessment and testing: Language in action. London, England: Routledge.

11 Hamp‐Lyons, L. (1990). Second language writing: Assessment issues. In B. Kroll (Ed.), Second language writing: Research insights for the classroom (pp. 69–87). Cambridge, England: Cambridge University Press.

12 Hamp‐Lyons, L. (1991). Basic concepts. In L. Hamp‐Lyons (Ed.), Assessing second language writing in academic contexts (pp. 5–15). Norwood, NJ: Ablex.

13 Hamp‐Lyons, L., & Condon, W. (2000). Assessing the portfolio: Principles for practice, theory, and research. Cresskill, NJ: Hampton Press.

14 Hughes, A. (2003). Testing for language teachers (2nd ed.). Cambridge, England: Cambridge University Press.

15 Huot, B. (1990). Reliability, validity and holistic scoring: What we know and what we need to know. College Composition and Communication, 41(2), 201–13.

16 Jacobs, H. L., Zinkgraf, D. R., Wormuth, V. F., Hartfiel, V. F., & Hughey, J. B. (1981). Testing ESL composition: A practical approach. Rowley, MA: Newbury House.

17 Katz, I., Haras, C., & Blaszczynski, C. (2010). Does business writing require information literacy? Business Communication Quarterly, 73(2), 135–49.

18 Knoch, U. (2009). Diagnostic assessment of writing: A comparison of two rating scales. Language Testing, 26(2), 275–304.

19 Lumley, T. (2002). Assessment criteria in a large‐scale writing test: What do they really mean to the raters? Language Testing, 19(3), 246–76.

20 Lumley, T. (2005). Assessing second language writing: The rater's perspective Frankfurt am Main, Germany: Peter Lang.

21 McNamara, T. (1996). Measuring second language performance. Harlow, England: Addison‐Wesley.

22 Ockey, G. J. (2009). Developments and challenges in the use of computer‐based testing (CBT) for assessing second language ability. The Modern Language Journal, 93(s1), 836–47.

23 Pennington, M. C. (2003). The impact of the computer in second language writing. In B. Kroll (Ed.), Exploring the dynamics of second language writing (pp. 287–310). Cambridge, England: Cambridge University Press.

24 Powers, D. E., Fowles, M. E., Farnum, M., & Ramsey, P. (1994). Will they think less of my handwritten essay if others word process theirs? Effects on essay scores of intermingling handwritten and word‐processed essays. Journal of Educational Measurement, 31(3), 220–33.