KIKO Evaluation Plan
prepared October 18, 2004
This plan also available as a Microsoft Word Document.
|
KIKO Evaluation Plan
This plan also available as a Microsoft Word Document. |
| Introduction | Decisions | Instrumentation | Budget |
| Background | Questions | Limitations | |
| Purposes | Methods | Logistics | |
| Audiences | Sample | Timeline |
This evaluation plan describes the procedures that will be used to evaluate KIKO (Kiswahili kwa Komputa or Kiswahili using the Computer), a series of interactive computer-based Swahili language tutorials developed by a team of instructors from the University of Georgia, Indiana University, and the University of Pennsylvania. KIKO is presently used as a supplemental instructional tool in Swahili classes at these and other institutions. This document contains a narrative section describing KIKO's background and development as well as detailed discussions of the purposes of the evaluation, the decisions and related questions shaping its design, the planned methods and instrumentation, the sample populations to be examined, and the evaluation logistics, timeline and budget.
This plan was prepared by Mary Lynn Miller, a graduate student in the Department of Instructional Technology at the University of Georgia under the supervision of Dr. Thomas C. Reeves of the Department of Instructional Technology for Dr. Lioba Moshi, Director of the African Studies Institute at the University of Georgia.
KIKO is a US Department of Education-funded project (Grant #P17A000053). Its primary purposes are threefold: to increase student motivation and interest, to reduce teacher workload, and to provide Kiswahili students with opportunities for drill and practice of language skills. KIKO was initially developed for CD-ROM delivery but is now accessed via the World Wide Web. The URL for KIKO is http://www.africa.uga.edu/Kiswahili/doe/index.html
The core of KIKO is a series of lessons with accompanying exercises and tests. These will be discussed in some detail below. As seen in this screen capture of the KIKO homepage, the central course learning tools in KIKO are supported by a description of the Kiswahili language and grammar and pronunciation guides.
The "About Kiswahili" section describes the language's history, significance, and current usage. The grammar guide, seen in part in the following screen capture, provides a useful introduction to Kiswahili's fifteen noun classes. Learners may click links to hear words pronounced.
The pronunciation guide, seen in the next screen capture, presents the vowels, consonants, nasal consonants and nasal consonant clusters used in Kiswahili. As in the grammar guide, links provide learners with the opportunity to hear the sounds pronounced.
The course section of KIKO contains the central instructional units. The instruction is divided in to Year 1 (Beginning), Year 2 (Intermediate) and Year 3 (Advanced). Each year's instruction is divided into six thematic units. Each unit comprises six lessons and six quizzes, a glossary, a diagnostic test, plus additional informational and interactive sections.
For example, Year 1, Unit 1, teaches the vocabulary of greetings and farewells. The second lesson of this section, Kuamkia shikamoo, shows how a younger person would greet their elder with respect. As seen in the screen capture below, the learner plays a video featuring authentic, native speakers, while reading a transcription of the dialogue from the video Links in the transcription take the learner to definitions of the highlighted words. Each lesson is accompanied by relevant grammar and cultural notes, accessed via links beneath the video window.
Each lesson has a related quiz. Learners receive immediate feedback on the quizzes and may repeat them as often as they wish. A quiz is shown in the screen capture below.
Following the completion of all the lessons in the unit, each learner completes a diagnostic test. Diagnostic tests do not provide immediate feedback and are not repeated. Question formats vary somewhat from unit to unit. Below is a portion of the diagnostic test from Unit 1.
As mentioned above, each unit contains a glossary. A portion of the glossary for Year 1, Unit 1 is seen in the screen capture below.
Supplemental materials for Year 1, Unit 1 include a song (text and accompanying audio), a matching game, a listening exercise and a list of useful expressions.
The purpose of this evaluation is to provide timely and accurate data for use by the project director (Dr. Moshi) to support decisions related to:
• continuing use of KIKO the module at development institutions,
• modifications and enhancements to the module as it currently exists, and
• potential licensing of the module to other institutions.
The specific sub-purposes of the evaluation are:
• To collect information regarding the KIKO's usability, appeal, and instructional effectiveness from students enrolled in beginning and intermediate Kiswahili courses.
• To collect information regarding the impact on teacher workload and student performance from instructors who are using KIKO in their beginning and intermediate Kiswahili classes.
• To collect information regarding the quality of the instructional design of the KIKO module through expert review and usability testing.
This evaluation has both summative and formative purposes. As a summative evaluation, the effectiveness of KIKO within beginning and intermediate Kiswahili courses will be evaluated. As a formative evaluation, ideas for enhancing KIKO will be sought.
The primary audience is the client for this evaluation: is Dr. Lioba Moshi, the KIKO project director.
Important secondary audiences include: other KIKO project developers, the U.S. Department of Education, University of Georgia , University of Pennsylvania and Indiana University students enrolled in Kiswahili classes and their instructors. This evaluation is conducted within the context of a graduate course, and thus, Dr. Thomas C. Reeves, Feng Wang and other participants in EDIT 8350, Instructional Product Evaluation, Fall 2004 at the University of Georgia will also constitute a component of this evaluation's secondary audience.
This evaluation will be used to inform decisions to be made by the client and project director, Dr. Lioba Moshi, regarding the KIKO module. Three primary decisions are assumed:
• Should KIKO continue to be used by the development institutions?
• Should the interface and/or other aspects of KIKO be modified?
• Should KIKO be licensed to other institutions?
The following questions will be answered in order to supply the data needed to support these decisions named above.
What evidence can be found that KIKO is currently meeting its objectives (increase student motivation, ease instructor workload, and enhance student learning)?
A number of methods will be used to gather data to answer the questions above. These methods include:
The following matrix illustrates the relationship between the questions this evaluation seeks to answer and the various methods of data collection the evaluation will employ. The larger the X in a given field, the greater the correlation between the question and the method.
| Is KIKO meeting objectives for current users? | Is KIKO marketable to new users? | |||||
| Students | Teachers | |||||
| Like it? | Learning? | Like it? | Work eased? | Competetive products? | Interface quality? | |
Interviews with Developers |
X | X | X | X | ||
| Classroom Observation | X |
X | X |
X |
||
| Interviews with Student Sample | X |
X | ||||
| Student Questionnaire | X |
X | ||||
| Student Focus Group | X |
X | ||||
| Interviews with Instructors | X |
X |
X |
X |
X |
|
| Product Review | X |
|||||
| Expert Review | X |
|||||
| Usability Testing | X |
X |
||||
Although KIKO is currently in use at all three development institutions (the University of Georgia , Indiana University , and the University of Pennsylvania ), data will only be collected at the University of Georgia . Two students from the beginning Kiswahili class will be invited to participate in the initial interviews. All students in both the beginning and the intermediate Kiswahili classes at UGA will be asked to complete the questionnaire. Two students each from the beginning and intermediate classes will be sought for the focus group. The instructors of both the beginning and the intermediate classes will be interviewed.
The expert interface design reviewers will be graduate students in the Department of Instructional Technology at the University. Two expert reviewers will be sought. Usability test participants will be undergraduates enrolled at the University of Georgia. Two to three usability test participants will be sought.
The instruments and protocols used to gather data for the evaluation are provided in the appendix.
The interview questions for both students and instructors are general guidelines only. It is anticipated that additional questions not listed in the documents appended to this plan will be asked in response to data gathered while the interviews are in progress. Student interviewees will be sought following completion of the classroom observation. A small incentive will be offered.
One of the purposes of the initial interviews is to gain insight into the preferences, problems, and concerns of student users of KIKO, and data collected in these interviews will likely lead to modification of the user questionnaire. All students enrolled in the beginning and intermediate Kiswahili classes will be asked to complete the questionnaire. Ideally, the questionnaires will be administered at the conclusion of a class period, but it may prove necessary to distribute them to the students to complete on their own time.
Data gained from both interviews and questionnaires will shape the questions used to guide discussion in the focus groups. Focus group questions will explore concerns raised in student interviews and seek greater insight into problems or dissatisfaction with various aspects of KIKO as reported in questionnaires. Focus group participants will be sought in each class, and a small incentive will be offered to participants.
A team of two design experts will be filmed as they interact with the KIKO module and discuss aloud their decisions and observations. Design experts will be recruited from University of Georgia Masters ' of Instructional Technology graduate students who have taken classes in instructional design and instructional product development.
Usability test participants will be recruited from a University of Georgia undergraduate class, "Introduction to Africa ." Each usability test participant will be filmed as he or she uses the KIKO module individually to complete a series of assigned tasks while following a think-aloud protocol. A small incentive will be offered to participants.
The following documents are appended to this plan (All available as word documents).
This evaluation is being designed and implemented by Mary Miller, a graduate student and novice evaluator. Because of the evaluator's lack of experience, some important nuances may be overlooked. The evaluation is to be completed within the course of a semester, a short time frame which prohibits a longitudinal study or true impact evaluation. Data collection depends on the willingness of volunteers to participate in the process. KIKO users are undergraduate students who face many competing time pressures and have other priorities than product evaluation. Because the population of UGA KIKO users is small, findings will not have statistical validity. Moreover, because KIKO is designed so that instructors may take different approaches to incorporating it into their classes, findings relating to the use of KIKO at the University of Georgia may not generalize to KIKO use at other institutions.
This evaluation is being designed and implemented by Mary Miller, with the guidance of Dr. Thomas C. Reeves, Professor in the Department of Instructional Technology at the University of Georgia. Mary Miller is responsible for all data collection and analysis, and for preparation of the final evaluation report.
All data collection will occur in October-November 2004 and the final report will be delivered in December 2004. The instructor interviews and classroom observations will be conducted by November 5. Questionnaires will be administered between November 10 and 17. The focus group will be held during the week of November 29. Expert review and usability testing will be concluded by November 23. Data analysis will be ongoing, with a final data synthesis conducted during the last week of November. A draft evaluation report will be delivered December 6 and the final report will be presented on December 13.
The following budget information is strictly hypothetical. There will be no charge for this evaluation. It is anticipated that the evaluator will absorb out-of-pocket costs for tape stock, questionnaire printing, and participant incentives (at a lower rate than that indicated below).
| Evaluator - Mary Lynn Miller | 89 hours @ $100/hour | $8,900 |
|
||
| Consultant - Dr. Thomas C. Reeves | 5 hours @ $400.00/hour | $2,000 |
| Videographer | 6 hours @ $150.00/hour | $ 900 |
| Expert evaluator incentives | $150.00 x 2 reviewers | $ 300 |
| Usability test incentives | $75 x 3 testers | $ 225 |
| Student participants incentives | $25 x 6 students | $ 150 |
| Tape stock, printing, clerical costs | $ 150 | |
| Total evaluation cost |
||
| $12,625 |
 |
Last update: November 10, 2004 Comments to: Mary Miller mlmiller@uga.edu Created for Dr. Thomas Reeves' UGA class EDIT 8350, Fall 2004 URL=http://www.arches.uga.edu/~mlmiller/8350web/index.html |