In this paper several testing systems are compared. Such comparing of existing and successfully used systems is very important in creation of new syst... Testing - Computers - Software - Organizations - Documentation - Training - Book reviews - automatic testing - computer aided instruction - computer-based testing systems - failings - automatized ECDL certificate - European Computer Driving Licence - computer skills certification - Lithuania - Computer-based testing - adaptive testing - computer literacy
Analysis of Computer-Based Testing Systems Renata Danielienơ, and Eugenijus Telešius Kaunas, Lithuania
[email protected], [email protected] Abstract — In this paper several testing systems are compared. Such comparing of existing and successfully used systems is very important in creation of new systems. It is important to use experience of existing systems by considering failings and by intercepting advantages. Having analysis of several systems the typical model of testing system can be suggested. Keywords — Computer-based testing, adaptive testing, computer literacy.
HE exploitation of possibilities provided by information technologies becomes increasingly relevant in developing knowledge society. To further develop e-services successfully, the e-content must be expanded and access to this content must be provided to citizens. Therefore measures are taken to ensure that enough of such e-services would be provided and that these services would be of good quality and simple to use. ECDL (European Computer Driving Licence) certificate was created especially for a common assessment of computer knowledge and skills. ECDL - is a computer skills certification, which is accepted by many organizations in the whole world. This certificate shows that its owner has enough skills of information technology and is able to work with personal computer and can accomplish operations with most widely used software. ECDL certificates are issued, when particular number of tests from special themes is passed. Testing can be performed manually or by using automated systems. In this paper we will analyze several ECDL testing systems and indicate advantages and disadvantages of these testing systems. After analysis of ECDL testing systems we will propose means for improving ECDL testing system in Lithuania. It is necessary to discover new innovative decisions, which will help to test knowledge of students better. Also in this paper we will provide suggestions to introduce computer adaptive testing (CAT) in ECDL testing system. By using CAT students get questions depending on their knowledge level. II. COMPUTER-BASED TESTING Computer-based training is a type of education in which the student learns by executing special training programs on a computer. Such process of education is especially
effective for training people to use computer applications, and students can get new information or improve their knowledge . Computer-based training usually includes computer-based testing as an integral part. By using both manual and computer-based testing it is possible to assess knowledge of students. During the test students get questions with several answers. However good test must satisfy many conditions, so when a test is being designed, we need to evaluate quite a few characteristics which describe test quality. III. IMPORTANCE OF AUTOMATED ECDL TESTING SYSTEMS ANALYSIS
A. About ECDL Currently online testing systems are widely used for checking knowledge of students. Some companies design their own testing systems, other use systems designed by software companies. However both types of testing systems – own created and designed by software companies have advantages and disadvantages. In this paper we will analyze Lithuanian ECDL testing system and will compare it with ECDL testing systems used in other countries. ECDL Foundation (ECDL-F) is the main organization, which establishes ECDL representative offices in countries and coordinates their work. The ECDL-F is non profit organization, which use funds for further development of activity. ECDL-F is the global governing body and licensing authority of the ECDL, the world’s leading end-user computer skills certification programme . ECDL Foundation has prepared documentation for Testing System quality management, where technological and facility requirements, activity processes are described, as well as information of how testing must be performed (both for manual and automatized cases). By using this quality management documentation members of ECDL-F can create their own home-made testing system. However, testing system may not necessarily correspond to documentation by 100%, because for each country belonging to the ECDL-F their representative office could be treated as separate organization (licensee or department) inside the main organization and may interpret some parts of documentation in a different way. ECDL-F representative office in every country executes the ECDL program. Every licensee in it’s own country authorizes it’s testing centres and supervises them to fulfil requirements. Testing centres perform the ECDL testing.
The collaboration of country licensees and testing centres is illustrated in figure 1.
9. ECLD representative office issues ECDL certificates, delivers them to testing centres, and testing centres presents them to test-taker.
Country „Z„ Representative „z“
A TC N.1
Country „A“ Representative “a”
TC N.d ECDL-F headquarters
Country “B” Representative “b”
Testing center 1 N
Tester TC 3.s
Country „C“ Representative “c”
Note. TC – Testing Centre
A – Participants of the system B – Registration C – Designing of Tests D - Delivering of Questions E – Issue of the ECDL Certificates
Fig. 1. The general structure of the ECDL Foundation B. The general structure of ECDL testing infrastructure Main participants of the infrastructure and main functions that they perform are illustrated in figure 2. The main participants are ECDL-F headquarters, country representative office, testing centre, tester and student. The participants of infrastructure have relationship “one to many”, it is illustrated in figure as 1:n. The main actions which are performed by participants are shown in the top of figure. These are Registration, Designing of Tests, Delivering of Questions, and Issue of Certificates. The sequence of actions is shown by arrows: Registration 1. ECDL-F establishes its representative offices and supervise them 2. ECDL representative office authorizes its testing centres and supervises their activities 3. Testing centres employ testers 4. Testers test students Designing of Tests 5. ECDL representative office creates test questions and delivers them to ECDL-F for evaluation 6. ECDL-F revises test questions according to the Quality Management System Delivering of Questions 7. Representative office manages the testing infrastructure and delivers tests to the testing centres. 8. Testing of students’ computer skills is performed in testing centres. Issue of Certificates
Fig. 2. The general structure of the ECDL testing infrastructure . C. Inference For the system to operate without failures, it is necessary to begin managing and observing IT infrastructure and collecting data . Firstly the present system must be analyzed, the processes supporting service offerings must be simplified and standardised, business and IT processes must be integrated and their metrics harmonized, the use of resources must be observed, business processes must be provided with IT instruments, and automation must be expanded. IV. REVIEW OF TYPICAL COMPUTERIZED TESTING SYSTEMS
To create a definitive testing system, we must to analyze some typical computerized testing systems, i.e. analyze their structure, features and functions, and discover advantages and disadvantages. In this paper we will review ECDL testing systems like: x SkillCheck Professional and Professional Plus testing systems; x TestaStation testing system; x Lithuanian testing system; x FastTEST Professional Testing System. Testing systems SkillCheck FastTEST Professional,
created by Americans, and TestStation, created by Swedes, are widely used and well known. Using these systems as example, many other similar ECDL testing systems were created. SkillCheck and TestStation testing systems can be considered typical testing systems; therefore we choose these systems for our analysis. Lithuanian testing system is an original automated testing system developed by Information Technologies Institute and is used to perform ECDL tests in Lithuania, Belarus, Ukraine and Russia. Communications between ECDL-F, countries’ ECDL representative offices and testing centres are performed by these parts of the testing system: x Tests’ implementation part x Administrating part x Database of tests and questions (AQTB – Automated Tests and Question Base) x User interface part It is another common feature between SkillCheck, TestStation, FastTEST Professional and Lithuanian ECLD testing systems. In all of these systems performance based testing is used. This means that not only Questions and Answers tests are used in these testing systems, but simulations of some software can also be imitated, where students must perform given tasks. Working with simulations is similar to working with real software program.
passwords, Testers can create their own tests in questions’ banks, which can be created hierarchically, Tests are formed from already created items, which are saved in questions’ bank, The question bank includes a full-featured word processor, Testers can create tests with random subsets of items within bank categories or across categories by manually or randomly selecting questions, Testers can print generated tests and give them to the students on the paper, Testers can monitor the statistical and psychometric characteristics of tests if question banks include item statistics - conventional, item response theory, or your own item statistics, Testers can protect questions from changing, Testers can present each student with a unique randomly scrambled test, Testers can create common or IRT-based computerized adaptive tests (CAT) for dichotomously scored questions .
A. Realisation of AQTB Main realization features of AQTB are shown in table 1. TABLE 1. REALISATION OF TESTING SYSTEMS
Lithuanian ECDL TS SkillCheck TS
Questions are provided randomly. Testers can create their own tests, Tests are provided from test list, Questions are provided randomly, All newly created tests are placed in list of SkillCheck tests, There is possibility to group desirable tests, give them desirable names, and to designate which group of tests will be delivered to students . In this testing system hierarchical tree is used as visual tool to improve process of information delivery, Questions are provided randomly . Testers can create their own workspaces, protected by
FastTEST Professional TS
Currently in the Lithuanian ECDL testing system questions are generated in two different ways. When questions are provided randomly these questions are grouped in sets by Syllabus items (thematic). During the test questions are selected from these sets. This is illustrated in figure 3. Thematic
The Questions by thematic
Fig. 3. Random order of questions
SkillCheck TS Lithuanian testing system also uses another improved way to generate test questions. Questions are also created according the existing ECDL Syllabus version. During the test questions are chosen randomly from each item of the ECDL Syllabus. In this way question sets for each student will be different. This is illustrated in figure 4. Program
Sets of questions
In SkillCheck testing system tests are grouped by skill levels. There are tree skill levels – beginners, intermediate and advanced. During the test the student is provided with rules how to perform the test. There is possibility to review skipped questions (this feature can be turned on or off). If this feature is turned on, the student will be informed if he answered correctly. If student answered incorrectly he will have possibility to answer repeatedly. 2
Fig. 4. Random order of questions (II way) B. The testing process The main features of testing process are shown in table 2. TABLE 2. THE PROCESS OF TESTING
Lithuanian ECDL TS
In Lithuanian ECDL testing system questions with image hotspots are used. In addition answer to question can be selected from list of answers where there are some correct and some incorrect answers. Simulations of some software (for example, Microsoft Word, Microsoft Excel and similarly) can also be imitated. When the test is finished the state of passing the test is shown. In Lithuanian testing system questions must be answers sequentially, skipped question can not be revised later. There is no skill level assigned to users depending on earlier tests results.
When the test is finished a generated report of assessment will be displayed. In the TestStation testing system it is possible to sort results by difficulty of questions. Tests are not grouped by skill levels. In the TestStation testing system questions with image hotspots are used Simulations of some software (for example, Microsoft Word, Microsoft Excel and similarly) can also be imitated. There is possibility to review skipped questions (this feature can be turned on or off). If this feature is turned on, the student will be informed if he answered correctly. If student answered incorrectly he will have possibility to answer repeatedly. 2
FastTEST Professional TS
When the test is finished a generated report of assessment will be displayed. Examinee responses can be accepted using highlighted fields, checkboxes or radio buttons, or using free-response text boxes. The possibility of placing examinee response options anywhere on the screen allows for complete flexibility. Created item with the preferred appearance and layout can be used as a template for other items. Tests can be of different kinds –
conventional, random, branched, or adaptive. Test questions can use audio, video, graphics, and popups. Test questions can be presented by using multi-windowed screens. Questions have options for scoring and branching. By using adaptive testing students can’t skip questions and return to them later. It is because next question is presented according to correct or incorrect student answer. 2 – In some tests it is possible to view if the answer is correct or incorrect. It depends on type of test. In some questions it is possible to answer repeatedly. There are graphical tools used (indicators and colours) to show if the answer is correct or incorrect. V. SUGGESTIONS A. About computer-based testing When there were no computer-based testing systems tests were designed manually, when test-giver decided which questions must be in the test. In the computer-based testing this method is not popular in comparison with tests automatically generated from sets of questions. The test generation can be accomplished in various ways . However there is another kind of testing, called adaptive testing, where each test-taker gets questions individually. Adaptive testing is like oral exam. When a teacher wants to check knowledge of a student more effectively, he can give easy, medium or hard questions. The teacher can consider when there are enough questions to determine student’s knowledge level. In Computeradaptive test (CAT) theory, this process has been automatized. Computer-adaptive test (CAT) is a form of computerassisted assessment where the level of difficulty of the questions administered to individual student is dynamically tailored to their proficiency levels . CAT usually starts with a question of medium difficulty. If the student answers correctly, the program gives an item that is a little more difficult. Conversely, if answer is not correct, the program gives a less difficult item. This process should be repeated until the condition of completion is reached . CAT theory ensures that the obtained student knowledge estimations do not vary in terms of items used in the estimation process.
B. Suggestions for ECDL testing system Students’ examination must be performed by evaluating more characteristics. This can’t be accomplished more exactly by using classical students’ testing. After analysis of computer-based testing and typical testing systems we recommend to introduce adaptive testing theory in ECDL testing system. This will help to estimate knowledge of students more precisely and increase testing efficiency. VI. CONCLUSION 1. In this paper several typical testing systems SkillCheck, TestStation, FastTEST Professional TS and Lithuanian testing system are compared. The main features of these systems are described. 2. Such comparing of existing and successfully used systems is very important in creation of new systems. It is important to use experience of existing systems by considering failings and by intercepting advantages. 3. SkillCheck, TestStation and Lithuanian testing system have similar main features. In FastTEST Professional testing system you can create adaptive tests which are based on the item response theory (IRT). Adaptive tests allow to reduce test time, increase security and efficiency. 4. The typical ECDL testing system can be created using computer adaptive testing theory. In this case we can estimate students more exactly, because the estimation is recalculated after every answered question, and the next question is given to the student according to how he had answered the previous question. This methodology allows to give questions to the students according to their knowledge. REFERENCES D. Bartram, R. K. Hambleton, Computer-Based Testing and the Internet Issues and Advances. John Wiley & Sons Ltd, 2006.  An official European Computer Driving Licence (ECDL). Available: http://www.ecdl.com. [Last viewed 2008 01 10]. of ECDL in Lithuania. Available:  Representative http://www.ecdl.lt. [Last viewed 2007 05 19].  C. Roever, “Web-Based Language Testing,” Language Learning & Technology, vol. 5, No. 2, pp. 84-94, May 2001.  An official SkillCheck, Inc. Available: http://www.ncs.com. [Last viewed 2008 01 10].  An official TestStation. Available: http://www.teststation.com. [Last viewed 2008 01 10].  FastTest Pro 2 Manual. Available: http://assess.com/. [Last viewed 2008 01 10].  A. Olson, “Improving schools one student at a time,” Educational Leadership, vol. 62 No. 5, pp. 37–40, 2005.  M. Lilley, T. Barker, “Students’ Perceived Usefulness of Formative Feedback for a Computer-adaptive Test,” The Electronic Journal of e-Learning, vol. 5 Issue 1, pp. 31 - 38, 2007.  M. Al-A’ali, “Implementaion of an Improved Adaptive Testing Theory,” Education Technology & Society, vol. 10 No. 4, p.80-90, 2007.