e-Assessment: making IT work - Recently Updated Pages

e-Assessment: making IT work Home

chrisbatt — Mon, 23 Nov 2009 10:25:28 CST

e-Assessment - making IT work

Workshop Wiki Site

Welcome to the wiki site for the UCISA 'e-Assessment - making IT work' event.

This site is intended to serve as a forum for discussion and experience sharing on e-assessment practices both before and after the workshop.
On the left you will find a menu with links to the plenary and parallel sessions. Each session has its own page where you will find the session outline, presentation files, links to additional reading resources and a space to add your own thoughts and raise questions for the speakers to address. The speakers are keen to hear from you and foster discussion both before and after the event. After the event we will put up summary reports for each parallel session and there will be an opportunity to continue the discussion.

23rd November 2009

FEEDBACK SURVEY STILL OPEN!

(The survey remains available until 27th November at http://www.surveymonkey.com/s.aspx?sm=fwVSRGZEVIEVUC7RFWCjfQ_3d_3d if you have not yet submitted your views.). If you have already tried to access it following the latest email reminder from UCISA, but found it closed, we apologise for that but please do try again. Thank you again to everyone who has contributed so far.

20th November 2009

Many thanks for attending the event and for your active participation and experience sharing in the parallel sessions. We hope that you found the day useful, with ideas to take away to your institution.
Audio files for the keynote presentations and for some of the parallel sessions are now available. Reports on the parallel sessions are also now available, and we invite you to comment on them through the discussion spaces on this wiki site.

Please keep coming back to this site, and apologies again for the advertising!

Word processing for examination

jgramp — Fri, 20 Nov 2009 06:19:14 CST

Many of today's students (and indeed professionals) are more used to using keyboards than pen and paper - indeed some will very seldom write on paper. Their lack of practice with handwriting presents genuine physical challenges when they are required to write long essay answers in conventional written exams - they may experience cramp, have trouble writing sufficiently quickly, or have difficulty writing sufficiently legibly.
The use of word processing packages to outline and develop essays and reports allows students to develop their ideas in a very flexible way - readily modifying and moving text as new ideas occur to them rather than having to present an argument in a linear start-to-finish flow. A traditional exam removes this flexibility which means that at this critical moment of assessment students are forced to work in an unfamiliar way. Some institutions are now reported to be taking this issue seriously, presenting students with the option of word processing their exam answers; others make no such provision apart from for students with special needs.
This session will be led by Nora Mogey, Head of Learning Services at the University of Edinburgh, and will provide an opportunity for participants to find out about the practicalities of delivering computer-based traditional exams, and to share their own experiences.

You can listen to a recording of the session:
Nora Mogey parallel session

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Notes from ‘Word processing for examinations’ workshop

by Jessica Gramp, UCL

Some of the reasons for considering word processing for examinations include:

Investigating peer assessment and the creation of artifacts (not just traditional quiz style questions)
Multiple Choice Questions (MCQs) are good for Science and Maths based exams, but not so good for Humanities subjects
Students are finding it difficult to write examinations by hand, when they do all their assignments on the computer:

It can be painful to write for up to 3 hrs when you aren’t used to it
Difficult to structure the assignment when you are used to being able to move text around

Tutors find it easier to mark assignments that are typed than hand-written:

They are more legible
They may be better structured

University of Edinburgh offer students the choice to complete their exams on a computer or by hand and students can change their mind on the day. Students install software on their own laptops to use on the day and connect to a wireless network to submit their work. The software also enables the work (as an encrypted file) to be collected on a USB stick if there are problems with the Internet connection.

University of Edinburgh found a miniscule bias grading hand written assignments versus the identical typed assignments. The average person typed more than they can hand write in a fixed time, so they believe this small bias evened out.

University of Edinburgh found in their experience that the older students (who are used to typing at work) are choosing to type their exams, more so than school leavers.

Some questions posed by the group:

Is it reasonable to expect students to write by hand when all their assignments need to be typed?
Could it disadvantage some students who are slow typers?
Do we need to test the students and recommend a minimum typing speed before they take an e-exam?
How do you stop students from using a spell-checker and looking things up online?
Should we necessarily stop students from using a spell-checker and looking things up online? - What about open-book exams?

Developing practical guidelines to manage and support summative e-assessment activities

Richard_Walker — Wed, 18 Nov 2009 09:30:38 CST

These notes are a conflation of the discussion from the two parallel sessions on this theme of ‘Developing practical guidelines to manage and support summative e-assessment activities’. Acknowledgements go out to all those that participated so fully in the discussions.

The sessions were primed around four questions:.

Who do we need to produce guidance for? (ie what groups of staff - (and students?))
What guidance does each group need?
What form should this take?
How should this be integrated with the delivery of summative assessment? (ie responsibilities)

Each group dealt with these questions difefrently. In addition there was some recognition that guidelines developed for summative e-assessment using multiple-choice type examinations would be different from summative e-assessment using e-portfolios and other technologies. The discussions raised a series of questions which it was considered needed to be covered in any practical guidelines. These make up the bulk of the notes below, where illustrative examples were provided by participants they have been inserted at the relevant point in the notes (indented in italics).

Who are the key stakeholders? And what guidance do they need?

STUDENTS

Students are accustomed to completing paper-based exams at school level onwards. They are not experienced in online assessment and are not familiar with how to navigate around electronic examination environments. Orientation is crucial, students need to be provided with opportunities for mock/practice tests.
- Nottingham Medical School addresses this problem by ensuring that all summative assessment activities mirror those that students have performed for formative assessment, so that they are aware of the assessment model and the procedure that they will be following (no surprises).
Students can benefit from a thorough induction – pre-registration checks and performance of a dummy test. Expectation setting on assessment mode and tools to be used is crucial.
Accessibility requirements need to be addressed in advance of the test.Clear instructions on the nature of the test, including how it will be accessed, how they will log-on and the environment it will take place in should be clearly published.
Students should be made aware of alternative arrangements that might be available to them.

COURSE LEADERS

Attention should be drawn to the risk of mismatch between the assessment framework and the activity design. Problems often arise when staff engage in adapting existing paper-based assessments to online delivery. Assessments should be tested pedagogically (through external examiners) as well as technically.
Staff need training on how to assess coursework online – i.e. activities which cannot be assessed through an assessment engine (e.g. MCQs). Discussion board contributions are one example.
What sort of rubrics are available for assessing free text responses? Digital story telling might be another approach, which presents assessment challenges.
- Information on example rubrics with regard to digital storytelling can be located at http://digitalstorytellingsynthesis.pbworks.com/Assessment-criteria-for-digital-storytelling
Beyond question design (relevant to MCQ), training is also needed for activity design for use of Web 2 tools such as blogs.
For MCQ style activities, staff need guidance and support in validating and reviewing the questions that they will be using. How to identify good and bad questions? Staff should be made aware of the statistical information that is available with many systems to help evaluate the validity and reliability of the questions. Guidance is needed on pre-testing / post-testing questions. How to proof an exam script and run through pre-testing of the examination tools. What sort of allowances should be built into the exam for students with special needs?
For the workflow in delivery of the assessment, course leaders need to think about contingency measures at each stage of the workflow. Paper-based back-up if system fails, or reschedule of assessment or design of activity so that students incrementally submit responses to test (20 questions at a time). Where will student responses be stored – is network connection secure? Are back-up PCs available if other machines freeze? Is back-up LT support on hand and available, if problems arise. How scalable are these arrangements for large cohorts? Flexible timing if students are interrupted – system glitches.
- Bristol found that students tend to prefer paper based exams, as they are more familiar with them and can doodle on the exam script. Images however are clearer online than on paper. As a compromise students are provided with a paper based version along side the online. This means they can flick through the questions and doodle on the paper. Answers must be submitted using the online version.

INVIGILATORS

Invigilators require a code of practice to guide them on how to approach electronic assessments – basic tasks such as checking module registrations should be performed in advance of the assessment. They need to be aware of the IT back-up support on call on the day – service level arrangements.
Need to be clear on how to respond to specific problems that might arise through loss of power, failure to save results etc

MARKERS

A concern for the assessment designer is how to support blind marking of summative pieces of work – especially if the software is not geared up to supporting this requirement.

DEPARTMENT / UNIVERSITY

Record retention is a key concern. How long should assessment records be retained for? What sort of digital evidence should be retained – i.e. the digital footprint for the assessment. Where should be retained? Online assessment can offer a greater range of evidence than traditional assessment activities in this respect. But what should we retain? Space / filestore issues and records management issues are related to this process and need to be addressed.
For evidence contained in wikis / blogs / discussion boards which are hosted within commercial VLE platforms for instance – attention needs to be drawn to the security and availability of that data, if the licence for the software expires (e.g. if an institution migrates away from a specific software solution). How will evidence be migrated and secured for the duration of the retention cycle?

EXAMINATIONS UNIT

Guidelines need to be in line with Exams Office regulations and ideally integrated
Grievance and appeals procedures to be clear
Room allocation in relation to numbers of students; are holding rooms required

EXTERNAL EXAMINERS

Examiners need to validate questions which will be used in summative examinations. For online exams, they need to be able to log in to the exam environment.
- At Nottingham Medical School, development work around the VLE platform supports the facility for external examiners to log in and comment on each question in the assessment via text-box review option.
There needs to be an audit trail within the environment on the development of the questions.
Examiners will require evidence / statement of purpose for the test - the objectives for the test - the statement should address validity issues.

ICT SUPPORT

Clearly identify support that is required at different levels. Support within examination room for getting the test started and dealing with local issues; back office server support for monitoring sytsem.

Workflow for guidance

Suggested workflow from one group:
The Academic has the idea to use eAss
eLearning expert? Confirm techniques and skills
1. Approve technique
Produce draft of exam
QA exam (internal & external
Inform Exams Office
Arrange room & support staff
1. Communicate guidelines to students
Mock/formative test
Define contingency plans & check they are in place
Final check of systems & exam
Exam
Back up submissions
Marking
Statistical analysis of Q’s
Confirm marks and release
Feedback

Suggested workflow from second group:

Example guidelines

University of Gloucestershire (developed 2001 when still Cheltenham & Gloucester College of Higher Education) attached (see link below).

Martin Jenkins
mjenkins@glos.ac.uk

Supporting summative e-assessment: the role of support staff

Richard_Walker — Wed, 11 Nov 2009 11:10:21 CST

As institutions increasingly look to online delivery for summative assessment activities, we need to consider how e-assessment is supported within an institution. Typically support comes from staff such as learning technologists, examinations officers, invigilators and IT support staff, but how is this support organised and co-ordinated? Can we utilise existing support structures or do we need to set up new support units for e-assessment? What skills are required by staff involved in the assessment process and how do we develop these skills? What levels of support are required for different types of assessment? How will the support team interface with academic and administrative staff within the academic departments?

This session will seek to draw on participants' experiences of how e-assessment is supported within their institutions and will aim to develop illustrative support models for e-assessment.

-----------------------------
Feedback from the session

The session dealt with the role of support staff in supporting summative e-assessments. Participants had the opportunity to work in pairs and in small groups.

In the pairs activity participants discussed their roles within their institutions and shared experiences in either supporting e-assessment or the steps taken towards this direction.

In the small groups, participants discussed and developed illustrative checklists of the responsibilities of specific roles on a given scenario.
A series of checklists were developed, based on the discussions of the two groups (the scenario and list are shown below).
Some of the observations of the two sessions, discussions and questions raised are listed below:
- it is difficult to think of the responsibilities in isolation
- the priorities in supporting e-assessment will differ for the different roles involved

Management role	Technical role	Administration role	Academic role

Role of the VLE and associated learning systems in supporting summative e-assessment

FionaStrawbridge — Tue, 10 Nov 2009 08:00:07 CST

HE institutions are familiar with the use of virtual learning environments to support learning and teaching, and staff are likely to be comfortable using a VLE for formative assessment activities. However VLEs were not designed for highly secure assessment purposes and lack some of the management, audit and security features of dedicated assessment software. The question for many institutions is whether the benefits of using a VLE platform (i.e. familiarity to academics, no additional licence costs, all student assessment information in one place, one system to support) outweigh those of dedicated secure systems (improved management, audit and security features). This session will seek to draw on participants' experiences of both approaches to:

Identify what might be done to improve the robustness of VLE-based summative e-assessment;
Discuss the support implications at institutional, technical and pedagogical level.

Some background

A straw poll conducted through the Heads of E-learning Forum recently asked a series of questions about use of the VLE versus use of dedicated secure assessment software. The questions were:

Does your institution allow the use of quizzes in the VLE for summative assessment?.
Does your institution insist on the use of Questionmark Perception (QMP) or other dedicated 'secure' e-assessment tools instead of the VLE?
Or do you have a mixed economy?
Are you planning a change in policy?

The 18 responses showed a roughly even split between

7 institutions which allowed the use of the VLE;
6 which insisted on QMP (but were thinking of allowing the use of the VLE);
4 had a mix;
1 was not doing any summative e-assessment.

Some quotes from the straw poll:

“It is easier, safer and more secure to manage high-stakes on a completely different server.” QMP institution

“While it is fine for formative assessments and quizzes I don't want my neck on the line because of technical problems with the load balancer, or a java setting, or any of the other things that we find tend to occur with WebCT.” QMP institution

“…we considered/evaluated QMP, but academic staff were not convinced of that the overheads were worth their time investment.” VLE institution

“… summative assessment is supposed to be restricted to QMP but in practice use of Moodle Quiz seems to be creeping through the back door.” QMP institution

“We would not have been given permission for our VLE.” QMP institution

“we use VLE only - IT guys have never been able to get QMP working here.” VLE institution

Aside from security issues, a few people commented on the better range of question types, and improved reporting in QMP compared with the VLE.

Report from ‘Role of the VLE’ workshop

by Fiona Strawbridge, UCL

THE WORKSHOP

At the start of each session there was a show of hands to find out how many participants’ institutions were currently allowing the use of the VLE for summative assessment. Most reported that the VLE was in use for summative assessment in their institution, although it was pointed out that few institutions had formal e-assessment policies in place. It was recognised that the participants for the sessions were probably self-selecting – the session was less likely to be of interest to those whose institutions had clear policies prohibiting the use of the VLE.

In the workshop sessions participants worked together to identify and discuss barriers to the use of VLEs in comparison with dedicated tools for secure assessment such as Questionmark or Respondus. In many cases suggestions were made of ways to address the barriers.

The barriers identified in the workshop may be divided into categories as follows:

Functionality
Systems/infrastructure
Process
Security
Staff development

FUNCTIONALITY

Many of the barriers related to general functionality of the VLE for assessment. Specific examples included:

Narrower range of question types
Question banks not easily sharable or scalable
Lack of reporting and audit tools
Poor integration with student information systems for transfer of marks
Lack of blind, double marking
Poor support for accessibility
Mathematical questions are a particular issue for VLEs
Difficulty in finding quizzes if ‘buried’ within existing VLE courses

It was pointed out that poor VLE functionality might be addressed by in-house or community development for open source VLEs, or by lobbying the vendors in the case of proprietary systems. Examples include development of a plugin for anonymous and blind-double marking for Moodle by a group of institutions led by ULCC, and the development of a Mathematical quiz tool for Moodle by the University of Birmingham. It was recognised that such developments are easier for users of open source VLEs.

It was noted that integration of testing systems with student information systems was equally challenging for VLEs and dedicated secure assessment products and that this wasn’t necessarily a VLE-specific barrier.

On balance there are no ‘show-stopper’ barriers in this category.

SYSTEMS/INFRASTRUCTURE

One very real challenge can be large loads on the VLE server/network at one time. Examples given included (i) students using the VLE to revise for one exam which impacted on a live exam which took place at the same time; (ii) synchronous chat for an online seminar which slowed down an exam; (iii) simultaneous logins/submissions placing severe load on the database.

Such problems may be alleviated by load-balancing and providing more/better servers (virtual machines?). Knowing what is taking place in other classrooms would be helpful but realistically is not possible for an institutional system used across all courses such as a VLE; in reality learning technology support staff are only really likely to know about other live exams. Load testing using real people in a real classroom environment rather than using one of the simulated testing systems is an important activity.

A dedicated ‘assessment VLE’ using a separate server/system was another possible solution; this would involve staff setting up tests in a separate location from the rest of the course, and would require an additional integration step for management of marks.

A third possible solution was the use of optical mark reading tools for paper-based multiple choice assessment. Whilst this would completely remove network/server issues it had a significant administrative overhead and was felt by some likely to put academics off using ‘e-assessment’.

It was pointed out that the use of thin-client systems, and network dependence of many VLEs increase the impact of network problems as there will be no local alternative; in contrast the Questionmark To Go tool can be used without a network connection.

Finally it was pointed out that all upgrades entail risk of unexpected changes, and that use of a dedicated tool, separate from the VLE, would minimise the risk of any upgrades or updates to the main VLE affecting tests.

PROCESS

Many institutions provide guidance and checklists to staff who are planning summative e-assessment. Often these require the member of staff to provide plenty of notice of any forthcoming assessments so that learning technology support staff have time to examine test settings and make sure that all is in order. However it was clear from the workshop that in many cases this requirement is ignored and that tests are being run without the involvement – or even the knowledge- of support staff. This might be characterised as ‘under the radar’ e-assessment. Examples included (i) tests being released to students too early; (ii) tests being released to the wrong groups of students; (iii) a three hour test which had been set up without any means nor requirement for candidates to save their work – the system timed out after 2 hours and students work was lost.

Another risk is of questions ‘leaking out’ because they may be visible to more than just the immediate teachers. In some cases departmental policies allow a large number of staff to see all VLE content within their department and there is scope for security breaches – for instance if a member of staff forgets to log out.

The absence of a ‘gatekeeper’ is a key factor here. In a dedicated secure system access to test creation, editing and released is closely controlled by the learning technology support team, whereas in a VLE anyone whose role is tutor/teacher/administrator in the VLE course may access and modify tests. A separate ‘assessment VLE’ with limited access may help here, or a ‘walled garden’ part of the main VLE.

Promotion of the existence of – and reasons for – e-assessment policies, guidelines and checklists, and the availability of sanctions for non-compliance would be another strategy.

SECURITY

A key risk in using a standard VLE without any special precautions is that students will be able to access other websites or other local resources in their files area. Some VLEs offer ‘secure windows’ but these can be easily broken out of; another option would be to use the VLE in conjunction with a dedicated secure browser. It was pointed out that the risk of students making use of illicit resources can be reduced by ensuring that there are the right number of questions for the allotted time to make taking time to searching for answers counter-productive. Careful and active invigilation is another essential precaution.

There were concerns that canny students would be able to find security holes that staff would be unaware of, such as students returning to cached pages and printing out questions; and whether the secure systems would stop this?

Access to tests from outside the invigilated test room can be prevented by the use of a test password which is only issued to those present. Monitoring of IP addresses can also detect illicit attempts.

Another challenge – not limited to the VLE but also for secure systems – is the ability to identify the students and to ensure that they are logged into their own accounts and have not got someone else to do their test for them. Simple but effective measures include: requiring students to sign a form and record their login ID; recording what machine they are using, their IP address (in order to see what machines are side-by-side); and the invigilator verifying this information and checking their id card.

STAFF DEVELOPMENT

It was recognised that the design of tests and preparation of challenging questions for use in multiple choice assessments involves considerable care and skill, and that there is a requirement for dedicated staff development effort. Some institutions have dedicated e-assessment staff to address this need, but many rely on VLE support staff.

KEY STEPS TO IMPROVE THE ROBUSTNESS OF THE VLE FOR SUMMATIVE ASSESSMENT

1. Lobby developers/vendors to improve functionality/reporting/security features.
2. Load test the live environment.
3. Ensure that academic staff are aware of the proper procedures and that they inform learning technology support staff early on when planning assessments.
4. Restrict access to summative tests by location/IP/password.
5. Gatekeeper to control access to the test delivery environment and check test settings before release – maybe through use of separate part of VLE, or a separate instance of the VLE.
6. Careful identity checking.
7. Active invigilation.

Keynote 1

roger_hewitt — Mon, 09 Nov 2009 11:37:14 CST

Keynote address 1: Developing an institutional policy

David Walker
Dr David Walker is Senior Learning Technologist and head of the Assessment Development & Enhancement Team in the Library and Learning Centre, University of Dundee. He has considerable experience in the fields of e-learning and online assessment and has written on the importance of staff education and understanding the student experience to the achievement of success in these respective areas.

In this keynote presentation David will provide a context for the event in commenting on high stakes assessment and emerging good practice over the last few years in the HE sector. He will comment on Dundee's approach to managing e-assessment and how the institutional policy in this area has been developed.

David's presentation is available here for viewing: David Walker's keynote presentation [PDF] and also here: PPT version of presentation file [PPT].
You can also listen to the recording: David Walker Keynote (mp3)

A revised e-assessment policy has just passed through Dundee's Learning and Teaching Committee and will now go to Senate for final approval. David has provided a copy of Dundee's online assessment policy (revised 2009) (also attached to the end of this page) for delegates to review.
David would like to invite delegates to try and establish if their own institution has developed a similar policy (and where this originated from) and to consider whether in fact there is a need for a dedicated policy covering online (e-) assessment, or whether this should simply be addressed by an overarching assessment policy. Please feel free to share copies of your institutional policy via the discussion space below..

Classroom set-ups for large scale summative e-assessment (thin client technology lab)

roger_hewitt — Mon, 09 Nov 2009 11:33:16 CST

In this session delegates will have the opportunity to visit the University of Bradford's new e-assessment thin client cluster, as a follow-up to the keynote address by John Dermo. During the tour, delegates will see how the new thin client cluster is being used for e-assessment and general teaching use. After a demonstration of the cluster, there will be a discussion on how best to manage learning spaces for e-assessment. The session will led by e-assessment adviser John Dermo as well as Alastair Wills from IT services and Rick Graves from Management Information Services, who were involved in the IT4SEA project which set up this facility. They will be happy to take questions on the implementation and running of the thin client cluster.

The discussion will offer a chance for delegates to share experiences and raise issues in an open forum on a range of themes related to learning spaces for e-assessment: for example, optimum room size, regulatory standards, workstation positioning and layout, accessibility, furniture design, security, invigilation, room booking, and not least accommodating other uses and needs for the space.

Please note that this session will take place in room F42 of the Richmond building, which is a short walk from the Norcroft Centre. Guides will be on hand to point you in the right direction, but please also bring these directions along so that you are heading the right way: Directions to thin client demo

You can listen to recordings of the two sessions:

John Dermo parallel #1 (mp3)
John Dermo parallel #2 (mp3)

Keynote 2

roger_hewitt — Mon, 09 Nov 2009 11:29:46 CST

Keynote address 2: Implementing thin client technology for e-assessment - practical considerations and lessons learned
John Dermo, e-Assessment Adviser, University of Bradford

PPT presentation is available here: http://tinyurl.com/yjms8nv (PDF; 2.55MB) You can also listen to the presentation John Dermo keynote (mp3)

In this second keynote presentation, John Dermo will be reflecting on the practical challenges in implementing a thin client technology solution for e-assessment, addressing support and management issues. John will also comment on the ways in which Bradford's e-learning team are evaluating the effectiveness of this solution for e-assessment.
For background reading about the thin client solution, please view the JISC project webpage and Bradford's own project webpage:
http://www.jisc.ac.uk/whatwedo/programmes/institutionalinnovation/its4sea.aspx
http://www.brad.ac.uk/lss/tqeg/projects/its4sea/

Please also view the project final report, a document on "Designing a new facility for Online Assessments", and the Functional Design Specification for a model for integrated e-assessment systems "Exam Admin", which are available as follows:
http://www.jisc.ac.uk/media/documents/programmes/institutionalinnovation/workforcedev/it4seafinalreport.docx
http://www.brad.ac.uk/lss/jisc/layout.pdf
http://www.brad.ac.uk/lss/jisc/examadmin.pdf

Sign up for parallel sessions

Richard_Walker — Tue, 20 Oct 2009 04:15:02 CDT

Please click on the link below to access the Doodle page, where you can sign up for the parallel sessions.

http://www.doodle.com/ps5qbczpd7u679yf

Parallel Sessions

Richard_Walker — Fri, 16 Oct 2009 10:59:26 CDT

Welcome to the Parallel Sessions. You will find session descriptions and supporting resources on the left. Please do post questions and comments for the speakers.