Developing an adaptive, diagnostic test of. English writing skills

Maat: px

Weergave met pagina beginnen:

Download "Developing an adaptive, diagnostic test of. English writing skills"

Joris Jansen
6 jaren geleden
Aantal bezoeken:

1 Developing an adaptive, diagnostic test of English writing skills

construction Assessment delivery Construction groups External committee Standard setting groups

2 Development of the DET Objectives Consultation IT Student model Consultation External committee Research Student models Psychometric Automatic scoring Assessment design Assessment creation Assessment construction Assessment delivery Construction groups External committee Standard setting groups Assessment report Survey Consultation Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) Intervention Co-creation 2

3 Architecture of assessment design Almond, Steinberg & Mislevy (2002) Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 3

4 Student model of English writing skills I Social proficiency 1. Can use the appropriate tone of voice/register 2. Can use the appropriate conventions for the text II III IV Structuring Phrasing Transcribing 1. Can structure a text 2. Can use appropriate structuring words (referers) 1. Can use appropriate words/word combinations 2. Can vary vocabulary where appropriate 1. Can use correct word order and grammar 2. Can use correct spelling and punctuation Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 4

5 Task model English writing skills, approx.1,000 items in the database Item types Multiple Choice: Matrix-item MC-item Multiple response-item Dropdown-item Hotspot-item Open questions Short answer-item (open) Drag and Drop Matching-item Sorting-item Order-item Types of tasks Contexts (from CEFR): s and letters Forms Messages on social media Chats Reports Adverts Essays Other Correction-item Marking-item Paragraph-item Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 5

6 Examples of test items Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 6

7 Phase 1: the construction process 1. Construction group develops test items 2. Construction group gets feedback from test assessor 3. Construction group improves test item 4. Test assessor finalises test item 5. Test item is imported into digital template 6. Test item is ready for screening Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 7

8 Phase 2: screening process Type of screening explanation done by 1 content and technical screening screening content accuracy and technical quality aspects assessor who has supervised construction group member 2 screening in QB screening for correct input, keys, coding etc and again screening for content accuracy and technical quality aspects team of 2 assessors 3 Screening in Facet Screening in Facet running and final check team of 2 different assessors 4 screening for usage of mother tongue screening for accurate, simple and clear use of mother tongue native speaker assessor Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 8

Phase 3: Examination Board Approval Examination Board approval required for every item After pre-test: Evaluation of psychometric data for all

9 Phase 3: Examination Board Approval Examination Board approval required for every item After pre-test: Evaluation of psychometric data for all items with Examination Board Committee Approve (or reject) for adaptive test Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 9

10 Behind the scenes Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 10

11 Overview of the adaptive test Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 11

12 Group Report Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 12

13 Student Report Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 13

14 Assessment of writing skills In the classroom Diagnostic testing a unique assignment complex: several writing problems one single assessment difficult to score and evaluate focus on 1 writing problem Several contexts and writing goals in a short period of time Several and repeated assessments Objective / automatic scoring possible Suitable for classroom and single summative assessment Suitable for formative, diagnostic assessment (o.a. Pullens, Den Ouden, Herrlitz & Van den Bergh, 2013; De Glopper & Willemsen, 2014) Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 14

15 Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 15

Vergelijkbare documenten

University of Groningen Educational value of digital examination

University of Groningen Educational value of digital examination Benefits Digital Examination HANDWRITING CORRECTING 1 2 3 Do you remember the Correcting the essay exams in handwriting from your students