You May Start Typing Now e-Assessment of Essay Style Questions Dr. Tom Mitchell Intelligent Assessment Technologies Ltd.

Perceived advantages of e-assessment e-Assessment Automatic marking Immediate feedback Increased flexibility Question types : atomic, item-banked, closed form, very often MCQ

But when we look at higher level examinations: Higher Level Assessment Examinations will generally consist of essay / extended answer / short answer questions; Some questions will require drawings and calculations to be assessed as part of the process; Examinations will require detailed human marking, often involving multiple markers.

What can we conclude ? Assessing Higher Levels Discursive (essay / extended response) questions likely to continue to be the bedrock of much higher level assessment; Examiners can probe students’ understanding rather than just their ability to retain atoms of knowledge; Little imminent likelihood of a mass migration to on-screen machine marked tests. Is e-assessment relevant here ?

Not Machine Marked ? Then Why e-Assessment Assessing Higher Levels Digital technology is pervasive, students are increasingly ‘digital natives’ (Prensky, 2001); Much of teaching and learning is technology based / supported; Digital technology permeates the professional workplace; Pen-and-paper examinations are becoming increasingly anachronistic.

How valid are hand written examinations? Assessing Higher Levels We write reports, papers, articles, presentations, etc. on computers Type, then cut and paste to organise our arguments, repeat… Very different process from writing on paper… Could you still write an essay on paper ? What if it was really high stakes ?

MSc in Railway Systems Engineering and Integration (RSEI) A Development Case Study University of Birmingham Postgraduate programme with MBA flavour: Engineering mixed with economics and strategy 8 taught Modules of one week’s duration, each Module summatively assessed Paper and pen tests with 30 questions to be answered in 75 minutes

Extended answer, with occasional requirement for hand drawn sketches RSEI Assessments What is the relationship between static train routes and the tracks and junctions of a rail­way network? What is the difference between a train path and a dynamic train route and how are the two related? Use a sketch to identify the equilibrium point between the supply and demand curves. How and why does the market reach equilibrium?

Drivers for e-Assessment RSEI Assessments Blind marking : ‘Reducing or removing the more undesirable aspects of subjective marking’. ‘Removing the (possibly subconscious) influence on marking of increasingly poor handwriting and spelling.’

How to do it ? E-Assessment of Essays Few e-assessment systems are fit for purpose for extended answers / essays Custom application : Extegrity (US) used by Edinburgh University Microsoft Word running on a locked down desktop University of Southampton

Preferred Option E-Assessment of Essays In browser, as part of e-assessment system Flexibility of delivery : cross platform No client install Include other question types in assessment Utilise work-flow of e-assessment system

Candidate Design Specification Provide simple word processor interface Support cut and paste and simple formatting Keep it simple and familiar looking Marker Support blind on-screen marking by question Support existing marking practices (i.e., multiple markers); Support insertion of comments for feedback

End of Module Assessments V1.0 The Pilot Small numbers initially (25 – 30 candidates) Invigilated summative, high stakes, examinations Delivered in University IT suite Paper version ready as back-up

A Blended Approach Hand Drawn Sketches Draw a generic V-life-cycle model for a major project. On-screen drawing tools too constrained, candidates would need to be familiar with them, etc So: Candidates hand draw diagrams + sketches We have devised a scheme to associate the paper drawings with the electronic candidate responses during marking

The Process Hand Drawn Sketches Invigilator provides candidates with pre-printed sheets of paper, with 4 ID numbers to be filled in Candidates click on a button in the interface to insert a sketch, and the system tells them the ID numbers to write on the paper – highly redundant The ID numbers are also embedded in the candidate’s answer, cannot be edited/deleted, and is subsequently visible to the human marker

Hand Drawn Sketches

Hand Drawn Sketches

A Successful Pilot… V1.0 The Pilot A number of end of Module tests delivered successfully Word processor interface worked well, and allowed candidates to organise and structure their arguments Candidates in general liked on-screen (more detail later) But…

System Vulnerable to IT Problems Robustness Intermittent network problems + old slow server Particular problem where a candidate spends a long time on a single answer →Complete rewrite of test delivery mechanism

V2.0 : Fault Tolerance

Supports V2.0 : Fault Tolerance Autosaving of candidate answers as they type No single point of failure Tolerant to intermittent or prolonged network / server outages, and recovers transparently – this functionality has come into play many times ! Can recover from prolonged network / server failure by saving to USB file All from within a standard web browser, no client install required

The Student Experience (1) Results Initially cautious, but subsequently ‘just got on with it’; The less computer literate requested a longer time period within which to compete the on-screen examination, compared to the paper version; Students readily took to re-visiting and re-editing responses to questions – definite advantage compared to paper-based system; Accessibility control had to be improved (font / background colours, font size, etc)

The Student Experience (2) Results Digital natives ? MSc course, candidates have a range of ages and technical abilities In the transition year, some trepidation about typing rather than writing Now that course fully e-assessed, does not seem to arise as an issue

The Examiners’ Experience Results On-screen marking was a (surprising) success. Reading the answers is much easier than when having to decipher poor handwriting. Much ambivalence / subjective interpretation is removed; Marking is fairer since the system allows marking question by question thus eliminating both positive and negative influences from poorly / excellently answered questions before and after.

From All Paper to All On-Screen Conclusions ExamOnline has minimal impact on pedagogy, and is very simple to use; Crucially is has been specifically designed and developed for discursive examinations; Supports hand-drawn sketches All module tests and now all final summative exams The move from paper to screen for discursive examinations has been successful and interesting;

Dr. Tom Mitchell