Set the scene of the context within which Aos had to develop their assessment concepts. Throw-away flowers – unless there are rules on how unwanted ideas are treated / weeded out, people will gravitate to ideas that are cheap to try.

Transactional tasks can demanding on background knowledge / raise issues of equality and diversity Meaningful and engaging is a challenge if your target audience range from 12 to 92, let alone 12 to 19. Demands of language and single level, criterion referenced approach lend themselves to e-assessment and the support they can bring. Inventive nature makes heavy demand of the authors – need to be creative requires time and space, opportunity to explore dead-ends Combines ‘rigour’ of timed written exams with engagement and practicality of vocational assessments – takes many awarding organisations some way from their comfort zones.

Immediacy of results is key to current programmes Survey work for Becta et al show that e-assessment is widespread and liked in FE (all sectors of) By using the interactivity of the computer, making a range of tasks more authentic – for example interacting with graphs and data within a spreadsheet, creating and editing documents within a word processor.

School cohort have very different needs to ACBL cohort – places pressure for very different test approaches to be seen to be equivalent In technology, you have to be very clear what you want or the chances are it will not work. Otherwise people will sit on the sidelines and watch.

In this example every item of text can be changed – yet it is 100% closed in that it is 100% computer markable. Could be part of convincing scenario – extended problem, i.e. Proof draft letter from colleague before completing and sending.

School cohort have very different needs to ACBL cohort – places pressure for very different test approaches to be seen to be equivalent

In this example every item of text can be changed – yet it is 100% closed in that it is 100% computer markable. Could be part of convincing scenario – extended problem, i.e. Proof draft letter from colleague before completing and sending.

So what demands does this place on the environment that the candidate operates in? Requires a lot of information to be available – search and selection is important Requires tools to be available and must record method May include options for free text entry?

Example of a possible screen layout

Example of Excel spreadsheet with formatting provided (before and after?) and discussion of possible heuristic rules.

School cohort have very different needs to ACBL cohort – places pressure for very different test approaches to be seen to be equivalent In technology, you have to be very clear what you want or the chances are it will not work. Otherwise people will sit on the sidelines and watch.

School cohort have very different needs to ACBL cohort – places pressure for very different test approaches to be seen to be equivalent In technology, you have to be very clear what you want or the chances are it will not work. Otherwise people will sit on the sidelines and watch.

School cohort have very different needs to ACBL cohort – places pressure for very different test approaches to be seen to be equivalent In technology, you have to be very clear what you want or the chances are it will not work. Otherwise people will sit on the sidelines and watch.

School cohort have very different needs to ACBL cohort – places pressure for very different test approaches to be seen to be equivalent In technology, you have to be very clear what you want or the chances are it will not work. Otherwise people will sit on the sidelines and watch.

E-Assessment & Functional Skills 10th November 2010 John Winkley, Tom Mitchell AlphaPlus

Introduction Who are AlphaPlus? What are Functional Skills? Why is e-assessment important to Functional Skills? Why are Functional Skills important to e-assessment? E-assessment challenges to be overcome for Functional Skills

About AlphaPlus Experienced educational consultants with a large and varied portfolio education and training: Research and evaluation including quantitative and qualitative projects Assessment and public examinations, UK and international Educational development work, particularly in general skills Educational Policy including technology policy We have strategic partnerships with: CDELL Institute at the University of Nottingham Cito Group, Netherlands BTL Group Ltd

About AlphaPlus A sample of our projects Evaluation of GCSE and A Level Reform QCDA Evaluation of new Mathematics GCSEs QCDA Evaluation of Functional Skills QCDA Evaluation and Development of Diplomas QCDA/Ofqual And other applied GCSE/A Levels National Survey of Skills for Life BIS Qualification & Standards Development AOs, TDA, DFE, etc. Assessment Performance Management Driving Standards Mozambique Examination Reform Moz. MoE

Functional Skills Overview Qualifications in English, Mathematics and ICT, covering Entry Level 1 to Level 2 Key element of 14-19 reform with Functional Skills as standalone qualification and forming part of apprenticeships and Diplomas. Replacement for Key Skills Likely future replacement of Skills for Life qualifications, up to 1m tests each year ‘Functionality’ also being designed into new specification GCSEs

Functional Skills Context 3 Year pilot with accredited assessments commencing September 2010 ~10 Awarding Organisations ~2,000 centres ~100,000+ candidates, twice a year (KS3 to adult) Initial concept of link to GCSE grade C pass Development approach ‘let 1,000 flowers bloom’ Convergence process based on revised criteria and accreditation based on iterative Sample Assessment Material

Challenges in Functional Skills assessment Assessment should be of an ‘effective application of ... in purposeful contexts and scenarios that reflect real-life situations’ Activity should be meaningful and engaging Test should be level specific and criterion-referenced Limits are placed on open and closed items Emphasis on process skills; selection of method, application of method, checking of results / working Demand of language should be one level below subject demand in Mathematics and ICT

Why e-assessment is important to Functional Skills Centres and learners rely on on-demand on-screen testing: Of Key Skills / Skills for Life (using MCQ tests) For roll-on / roll-off programmes and progression Centres find e-assessment brings many efficiency benefits: Flexibility in registrations and entries Immediate Results Many learners find e-assessment engaging Economies of scale in a volume qualification for AOs

Challenges facing e-assessment Growth of high stakes e-assessment slower than many expected particularly in schools: Difficulty of offering examinations on a single instance Challenge of high validity/authenticity questions Item banked-tests under developed for high stakes use Other issues – cost of startup implementation and production of tests Regulatory ambiguity

Why Functional Skills is important for e-assessment Functional Skills presents an important test bed for e-assessment: High volume, publicly significant qualification Huge public investment Sets a high bar for innovation Smart computer marking Innovation in on-demand human marking process Requires innovative items and tools

Examples of e-assessment Sample assessment items, with thanks to

E-assessment and Functional English In English open items are linked to 100% of the marks available at Level 1 and Level 2 in writing and minimum 50% in reading. 40-45% of marks in writing are allocated to SPAG Definition of a closed item from the criteria includes: ‘assessment that presents items where the candidate is confined to either a single or limited number of correct responses predetermined by the assessment, and where process is not explicitly credited.’

Examples of e-assessment

E-Assessment and Functional Mathematics In mathematics 75% of marks must be allocated to open response items Definition of open response includes: ‘Open response assessment is defined as task-based assessment based on real-life contexts that require candidates to apply their skills, knowledge and understanding in order to resolve problem/s or produce effective outcome/s. Open response assessment presents purposeful tasks and problems embedded in realistic scenarios but does not prescribe the process/es or method/s by which the candidate responds.’

Examples of e-assessment

Example of ICT marking Example ICT paper has 45 marks 15 marks on numerical task 3 marks on format Graded for clarity and appropriate use Human quick to judge aesthetics Could be computer marked

Functional Skills Challenges for e-assessment FS English Reasonable to expect candidates to type text? Requirement for human marking for at least some of the test. E-marking well established, but on-demand? FS Maths Flexibility of process means candidates must have access to a basket of tools Entering maths working on computer is not natural Marking process is hard. Some human marking likely.

Functional Skills Challenges for e-assessment FS ICT Open tasks clearly possible and acceptable onscreen Computer marking possible, but human marking likely for the most open/high level items. May not be so time-consuming for markers though. Other issues Avoiding assessment contamination is a challenge for on-screen as assessments become richer. Accidental/unintentional aspects of specifications mitigate against e-assessment.

Key lessons for supportive E-Assessment environment Level of up-front investment means e-assessment is unlikely to be first option for a pilot qualification unless “designed in”. Complex interaction exists between specific test technology, each subject and the criteria. Requires method of creating and sharing ‘case law’. Up-front costs of e-assessment makes ‘single submission for approval’ model high stakes for all concerned. A transparent and collaborative process is required.

Suggested Practical Approaches For the substantial benefits to be achieved, e-assessment needs to be recognised in both high level policy and flowed down to subject specifications. To encourage the required levels of investment in a pilot assessment a transparent and collaborative process is required.

17th March 2010 John Winkley Thank you for listening. Questions?