Investigating cultural variability in rater judgements of oral proficiency interviews
Abstract
The article is concerned if the cultural background of non-native raters could lead to substantial differences in the assessment of students’ oral performances during the national examination in the English language in Estonia. The discussion involves the analysis of the ratings of twenty video-recordings of oral proficiency interviews by two rater groups of the Estonian and Russian origin, and a questionnaire study investigating rater perceptions of the national examination. Despite the lack of any marked cultural bias being displayed in the raters’ behaviour, the results of the study reveal a number of significant differences in their perceptions of various aspects of the national examination as well as their own behaviour during the rating process.
DOI: http://dx.doi.org/10.5128/ERYa10.17
Keywords
Full Text:
PDFReferences
Abedi, Jamal 2011. Assessing English language learners: Critical issues. – Maria del Rosario Basterra, Elise Trumbull, Guillermo Solano-Flores (Eds.). Cultural Validity in Assessment: Addressing Linguistic and Cultural Diversity. New York: Routledge, 49–71.
Alas, Ene 2010. The English Language National Examination Validity Defined by Its Oral Proficiency Interview Interlocutor Behaviour. Tallinn University Dissertations on Humanities 22. Tallinn: Tallinn University Press.
Bachman, Lyle F.; Lynch, Brian K.; Mason, Maureen 1995. Investigating variability in tasks and rater judgements in a performance test of foreign language speaking. – Language Testing, 12 (2), 238–257. http://dx.doi.org/10.1177/026553229501200206
Brown, Anne 1995. The effect of rater variables in the development of an occupation-specific language performance test. – Language Testing, 12 (1), 1–15. http://dx.doi.org/10.1177/026553229501200101
FitzGerald, Helen Gay 2003. How Different Are We? Spoken Discourse in Intercultural Communication: The Significance of the Situational Context. Languages for Intercultural Communication and Education 4. Clevedon: Multilingual Matters.
Fulcher, Glenn 2003. Testing Second Language Speaking. London: Longman/Pearson Education.
Hofstede, Geert 1997. Culture and Organisations: Software of the Mind. New York: McGraw-Hill.
Kriisa, Kristel 2012. 2012. aasta inglise keele riigieksami lühianalüüs. [Analysis of the National Examination in the English Language of 2012.] http://ekk.edu.ee/valdkonnad/uldhsridusvalishindamine/riigieksamite-materjalid-2012/inglise-keel (10.3.2013).
Lumley, Tom; McNamara, Tim F. 1995. Rater characteristics and rater bias: Implications for training. – Language Testing, 12 (1), 54–71. http://dx.doi.org/10.1177/026553229501200104
Luoma, Sari 2004. Assessing Speaking. Cambridge: Cambridge University Press. http://dx.doi.org/10.1017/CBO9780511733017
McNamara, Tim F. 2000. Language Testing. Oxford: Oxford University Press.
Pajupuu, Hille 2001. Kuidas kohaneda võõras kultuuris: käsiraamat. [How to Adapt Oneself to a Foreign Culture: A Handbook.] Tallinn: Eesti Keele Sihtasutus.
Pajupuu, Hille 2007. Kuidas hinnata suure panusega testide hindajaid. [How to assess the raters of high-stakes tests.] – Eesti Rakenduslingvistika Ühingu aastaraamat. Estonian Papers in Applied Linguistics, 3, 221–233. http://dx.doi.org/10.5128/ERYa3.15
Pollitt, Alastair; Murray, Neil L. 1996. What raters really pay attention to. – Performance Testing, Cognition and Assessment. Selected Papers from the 15th Language Testing Research Colloquium, Cambridge and Arnhem. Cambridge: Cambridge University Press, 74–91.
REKK 2008 = National Examinations and Qualification Centre of Estonia 2008. Marking scale for speaking. www.ekk.edu.ee/valdkonnad/uldharidusvalishindamine/riigieksamite-materjalid-2008/inglise-keel (26.12.2012).
Tannen, Deborah 1984. Conversational Style: Analysing Talk Among Friends. Norwood, NJ: Albex.
Trompenaars, Fons 1994. Riding the Waves of Culture: Understanding Cultural Diversity in Business. London: Nicholas Brealey.
Wigglesworth, Gillian 1993. Exploring bias analysis as a tool for improving rater consistency in assessing oral interaction. – Language Testing, 10 (3), 305–319. http://dx.doi.org/10.1177/026553229301000306
Winke, Paula; Gass, Susan; Myford, Carol 2012. Raters’ L2 background as a potential source of bias in rating oral performance. – Language Testing, 30 (2), 231–252. http://dx.doi.org/10.1177/0265532212456968
DOI: http://dx.doi.org/10.5128/ERYa10.17
Refbacks
- There are currently no refbacks.
Copyright (c) 2014 Irina Stassenko, Liljana Skopinskaja, Suliko Liiv
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
ISSN 1736-2563 (print)
ISSN 2228-0677 (online)
DOI 10.5128/ERYa.1736-2563