Resource Database: Geography, Earth and Environmental Sciences

Outline:

First year students in geography and geology have been partly examined using computer-based assessments since 1985. Our current practice is to use multiple-choice tests which are undertaken at a computer terminal. Marking of test answers is objective, absolutely consistent and extremely rapid, and output includes an analysis of facility and of the distractors chosen.

Context:

Cheltenham & Gloucester College operates a modular degree scheme based on a semester system, with 15 weeks per semester. First year modules have relatively large student numbers (up to 120 in the Geography and Geology Department) because they are taken by most minor, as well as major and joint, subject students. Computer-based tests are used in geology (up to 60 students), physical geography and geography modules, and have been used in modules in earth's resources. They help to ease the assessment bottleneck that occurs twice a year at the end of each semester.

Currently, automated testing is used in five of the Department's eleven first-year modules, an increase over previous years; between 1985-1994 only 10-25% of level I modules were assessed in this way. In two modules they are organised as part of the formal examination programme at the end of the semester, but in the other three they occur within the modules as part of the continuous assessment (see table below).

Description

All the computer-based tests are undertaken at a computer-terminal. The software used is called CMAPROG and was developed in house. It is loaded up in advance by the invigilator, who also distributes any other ancillary material. Student fill in a cover-sheet, noting their student number and terminal number, and begin. The software is self-timing, starting from the point where the question file is loaded.

The software is text-based - it presents the question on-screen and an array of possible answers. Examinees respond by typing in a key (usually a number in the range 1 to 6) for the selected response.

About half the tests in current use refer to ancillary material - mostly specimens in geology exams and printed graphs, diagrams and tables in geography. The method of selecting (or editing) an answer means that the time limit is not a problem. Candidates are set between of 60-80 questions per hour and usually finish with 10-20 minutes to spare. Large modules may require two sittings, which means that members of the first test-group have to remain supervised until the second group is seated.

Computer-marked tests in geography and geology modules at CGCHE: assessment timing (within a 15-233k semester), weighting and duration

It takes less than 5 seconds for a separate computer program to mark 120 answer files, although printing out results, including a basic question analysis, takes a few minutes. The question analysis lists

1. the percentage of candidates who were successful on each question, and
2. a table showing how often each distractor was chosen.

Resource Implications

Physical resources: Sufficient terminals and specimen sets are required for each member of the module, or for half the group if two sittings are planned. Running an exam in two sittings is fairly straightforward, but more than two becomes a logistical problem. Generally there will be no resource problem as most institutions will have computer rooms for class teaching, though sometimes the spacing of the machines means that only alternate places can be used in an exam. The specimens required are no more than would be used for a conventional exam in subjects such as geology. If investment in teaching computers has already occurred then the only finance that might be required is to acquire a suitable software package (usually less than the cost of one computer).

Staff time: This can be broken down into three areas:

• Preparation of the test
  Preparation time is longer than for a conventional written paper on the first occasion that a module uses multiple choice testing, although converting existing paper-based tests to computer-format will typically take only 4-6 hours. A new 80-question test will take from 12-24 hours of work depending on staff experience and question complexity. It is advisable that all questions are proof-read to ensure that they are of appropriate difficulty and unambiguous, adding 4-6 hours to the process. Once a test bank has been established, about 4-6 hours of staff time will be sufficient to select an appropriate mix of questions. It is better to use a majority of questions of known facility if possible, so new questions are only required to keep pace with changes in the module curriculum.

• Invigilation
  Invigilation staffing is comparable to that for conventional exams. Invigilators have to type in the passwords that restrict access to the test software in lieu of dishing out blocks of paper.

• Marking
  Marking is rapid and painless. Overall, computer-based testing requires a greater investment of staff time than conventional examinations, but only for the first year or two. By the end of three years the staff demands should have broken even; beyond that is net profit.

Student performance

Correlations in student performance between computer-based tests and other forms of assessment are quite high - there was a correlation coefficient of between 0.55 and 0.70 in three recent comparisons between computer-based and written answers in geology. The question analysis highlights areas of difficulty both in the course and also general processes such as the manipulation of units and the sequences of reasoning in making inferences.

Evaluation

The computer-based tests used at Cheltenham offer similar advantages and disadvantages to paper-based multiple-choice tests including:

Advantages:

• Consistent marking
• Error-free totalling
• Easy question analysis
• No discrimination against candidates who write slowly
• Time-saving once a question bank has been established and tested
• They compel candidates to revise the whole of a course
• They can provide feedback to students when they are used within a module

Weaknesses:

• They do not test precise recall (e.g. spelling) of technical terms
• They do not assess synthesis
• English expression - both grammar and clarity - are not tested
• They do not assess the construction of an argument
• They do not assess systematic breakdown and treatment of broad questions

Overall, the advantages make these techniques worth employing, but the disadvantages limit their application. Examiners have most to gain with large first-year modules, and their use in this department is restricted to level I. They do assess more than recall, and can be used to test numerical manipulation, practical observation and the application of sequences of reasoning, argument and synthesis. In this department we have experimented with open-ended computer-marked tests (Angseesing, 1989) but have found that this approach requires much more marking time, even after several runs of a question have built up a substantial answer bank.

Key advice

• Have a contingency plan for computer breakdowns (these are not common, but we have had both individual monitors and keyboards die during exams). The best arrangement is to use a software package that continuously (and automatically) saves work and allows it to be retrieved once the candidate's computer has been restarted.
• Allow plenty of time to set up the first question set and ensure that it is adequately proof read.
• Maybe have a specimen question set available to student in advance, so they can familiarise themselves with the question style and how the software runs.
• Ensure that the questions are drawn more or less equally from the whole course e.g. with 80 questions and twelve teaching weeks there should be 6-7 questions on each week's work.
• Each question set should include some very easy straightforward questions, some tough ones and a range of in-between ones; collectively the set will then have good discriminating power.
• When planning a second exam using pre-existing question bank, check that new questions are written to take new material into account.
• Read old questions before setting a new exam. Some questions are not so clear to their composer a year later!
• Ensure that the questions relate to the learning outcomes for the module. Where appropriate, check that the questions assess a range of skills (such as manipulating units, interpreting graphs etc.) as well as testing the facts and ideas that form the knowledge base of the module.

Reference

Keywords

This is one of the case studies from the Science Education Enhancement and Development (SEED) project's handbook Computer Based Assessment (Volume 2): Case studies in Science and Computing

Keywords can be used to search for specific topics
Abstracts are also listed by Originator

Resource Database Home Page