View
213
Download
0
Category
Preview:
Citation preview
1 Assessment and Evaluation
Basic Iraqi Survival (BIS) Language Guide
California State University Monterey Bay
Selwa Alkadhi
Magdi Ahmed
Assessment and Evaluation IST622
Bude Su, Ph.D
May 2009
2 Assessment and Evaluation
Contents
Introduction .................................................................................... Error! Bookmark not defined.
Methodology ................................................................................. Error! Bookmark not defined.
Rationale and Target Audience .................................................. Error! Bookmark not defined.
Prototype Design ........................................................................ Error! Bookmark not defined.
Learners ...................................................................................... Error! Bookmark not defined.
Study Conditions ........................................................................ Error! Bookmark not defined.
Process ....................................................................................... Error! Bookmark not defined.
Entry Conditions ........................................................................ Error! Bookmark not defined.
Results ......................................................................................................................................... 8
Instruction .................................................................................. Error! Bookmark not defined.
Outcomes ................................................................................... Error! Bookmark not defined.
Recommendations ...................................................................... Error! Bookmark not defined.
Summary ........................................................................................ Error! Bookmark not defined.
Appendices ..................................................................................... Error! Bookmark not defined.
Appendix A --Informed Consent ............................................... Error! Bookmark not defined.
Appendix B – Module Assessment Instrument ........................ Error! Bookmark not defined.
Appendix C – Pre-Test Startification Survey ............................ Error! Bookmark not defined.
Appendix D -- Raw Data Table#1 ............................................. Error! Bookmark not defined.
Appendix E -- Raw Data Table#2 ............................................. Error! Bookmark not defined.
3 Assessment and Evaluation
I. Introduction
The manner in which technology is integrated in instructional design of language
development often impacts the efficiency and effectiveness of relaying the new
information to students. Prior to this MIST program, the designer had relied on
conventional means to deliver language tools to students through face-to-face
instruction. The purpose of this evaluative paper is to assess the extent at which
technological-based teaching achieves instructional goals in comparison to teaching
approaches that couple both technology and conventional student and instructor
interaction. This is especially relevant to the teaching/learning atmosphere at the CE
(Continuing Education) program. The students depend highly on face expressions,
gestures, and body language in acquiring enough survival language skills to succeed
in their military missions. The goal of the study is to compare the extent at which the
prototype “Basic Iraqi Survival (BIS) Language Guide” achieves its instructional
objectives with or without the inclusion of student instructor interaction.
II. Methodology
Rationale and Target Audience
The target audience of the program is military personnel who are being deployed
to Iraq within a month of completing the given course. The typical class size varies
from 15-30 students. There exists a diversity in students characteristics in terms of
age (ranging from 21-55), rank, educational backgrounds, and social backgrounds.
4 Assessment and Evaluation
The prototype is part of a program that focuses on survival language and culture
familiarization in a condensed and fast-paced presentation. The students are military
personnel who have limited training or no training in either the Arabic language or
the Arabic culture. They may or may not have had exposure or direct contact with the
Middle East; its language or its culture. The LFAST (Language Familiarization and
Area Studies Training) is often the primary means of exposing them to this type of
information in a manner that prepares them for their specified objective when
deployed. Despite the dire need for such exposure and familiarization, the curriculum
is terse and limited to between eight and thirty hours of instruction. Hence, it is not a
surprise that the students are overwhelmed with the large amount of vocabulary,
phrases, and cultural elements that they must retain in a given time. Students often
memorize provided phrases and vocabulary without a notion of when and how to
manipulate these language tools in real situations. The new information does not
cohere with any pre-existing knowledge. Rather than establishing these vocabulary
lists as the base of their developing knowledge of the survival language, this module
provides the students with a more practical base on which the previously provided
vocabulary lists and phrase lists can be supplemented. These students receive
booklets that they use to find vocabulary needed to communicate. The military’s duty
is mainly to train and/or guide the Iraqis in different areas of need. Hence, simply
memorizing words and/or phrases alone will not be sufficient for specific military
objectives. When it comes to grammar, students have a difficult time making sense of
new knowledge attained. Sentence structure is most efficiently learned when it is
5 Assessment and Evaluation
delivered in context of situations during which students can apply their new learned
information.
Students need to be given the opportunity to apply basic new language skills to their
everyday lives before they relay these skills to the complex situations of their military
profession. A basic foundation for everyday conversation and language usage is vital
to build upon.
The manner in which new knowledge is presented plays a pertinent role in the extent
at which the students will grasp and appreciate their newly acquired language skills.
Demonstrating the manner in which their new language abilities can be applied is an
essential means of motivating them to consistently develop their skills. With regard to
language learning, the intertwining of culture and language is a valuable motivating
factor to the students. Exposing the students to a variety of representations of the
situations they learn about in the new language fosters recognition learning and will
achieve significant results in the affective domain.
The Prototype
The “Basic Iraqi Survival (BIS) Language Guide” prototype is designed to be
launched as a link to a webpage. Minimal computer skills required to launch the
prototype
Tryouts and Sample Populations
A pool of fifty randomized students is selected for the experiment in accordance
to the characteristics sought in the target population. As an attempt to stratify our
population, these fifty students are provided a pre-test. The purpose of the pre-test is
6 Assessment and Evaluation
to eliminate any confound that may alter our results or what we are actually testing.
For example, characteristics such as prior knowledge of the language, prior
deployment experience in the Middle East region, prior experience with learning
languages, etc. can influence the extent at which the prototype has on instructional
efficacy. Such characteristics are equally stratified among the two blocks of samples
which are (sample 1) students who utilize the prototype only and (sample 2) students
who utilize the prototype coupled with instructor interaction. Once the two samples of
25 students are structured, the experiment proceeds to test the effect of student
instructor interaction on the results of the prototype in achieving its objectives. The
variable condition in the two samples is the presence or lack of an instructor during
the 30 minute prototype. While close observations are made to assess student-student
interaction, student-teacher interaction and student-module interaction especially
during the instructor's presence in class, the most valued tool in evaluating our
subjects is through a comparison of the sample’s performance on their post-tests in
comparison to that in their pre-tests. These samples ultimately play the role of the
critics/analysts of the prototype and whether or not it must rely on instructor
intervention to most effectively reach its instructional goals.
Domains Evaluated
The pre- and post tests for the evaluation will address the user’s knowledge in both
the cognitive/knowledge and cognitive/application. The questions featured in the pre-
tests and post-tests address the domains according to the following:
7 Assessment and Evaluation
Table #1: Pre-tests and Post-tests Domains
Topic/task
Objective
Domain and level
Confirm whether or not
the student has had prior
Arabic language
instruction.
to accurately assess the prototype’s
effect on student learning as well as
for stratification purposes. (survey
only)
Cognitive/knowledge
Confirm whether or not
the student has had prior
Middle East deployments.
to accurately assess the prototype’s
effect on student learning as well as
for stratification purposes. (survey
only)
Cognitive/knowledge
Confirm whether or not
the student is bilingual,
trilingual, etc.
to stratify those students who are more
experienced in language learning.
(survey only)
Cognitive/knowledge
Identify feminine and
masculine forms of Arabic
objects.
users will be able to identify
appropriate feminine and masculine
designations for objects.
Cognitive/knowledge
Identify correct use and
placement of pronouns in
varying sentence forms.
users will be able to identify the
correct designation of pronouns in
sentences.
Cognitive/knowledge
Identify correct sentence
structure in imperative
form sentences.
users will be able to identify correct
imperative sentence structure.
Cognitive/knowledge
Identify definitions of
vocabulary.
users will have a broadened pool of
new basic and practical vocabulary.
Cognitive/knowledge
Identify vocabulary within
common relationships. Ex.
Antonyms, synonyms, etc.
a learner will be able to identify
vocabulary in terms of relationships.
Cognitive/knowledge
Distinguish between the
sounds of letters that are
found in both the Arabic
and English language.
a student will be able to identify and
pronounce different sounds of the
Arabic alphabet.
Cognitive-application
8 Assessment and Evaluation
Construct basic survival
sentences.
a student will be able to construct
basic survival sentences using newly
learned vocabulary.
Cognitive-application
Ability to change forms of
pronouns to feminine,
masculine, plural,
singular, etc.
a student will be able to direct
commands to different pronouns by
changing the forms of pronouns.
Cognitive-application
III. Results
Upon the start of this trial, our hypothesis was that students who are in the sample that is exposed
to both the prototype and the face-to-face instruction will demonstrate more improvements in
their post-test results in comparison to their pre-test results. This conclusion would be measured
by a quantitative comparison of percentages attained on the pre-test and post-test of the two
sample groups. In such a trial, we recognized that it is particularly vital to stratify our population
to be tested in order to get rid of confounds that may affect a change in test results besides the
variable we are actually testing- the instructors influence on learning when coupled with the
prototype. Essentially, this trial not only tested the efficacy of the prototype itself, but also its
dependence on face-to-face instructor interaction for optimal learning results.
A pre-test was given to all 50 students and results were utilized to block and stratify the pool of
fifty students into two samples of 25 students each. In reference to Table #1.
Entry Conditions
The average scores attained by all 50 students in the pretest is 18.4%. The score represents
students’ performance prior to exposure to the prototype.
Most of the students in the population had no history of knowledge of the Arabic language.
Some students may have demonstrated a score higher than the average due to circumstances
9 Assessment and Evaluation
unrelated to our trial. Hence, the population was stratified so that students who have been
deployed and have had exposure to the language are evenly distributed among the two samples.
In assessing the effectiveness of the prototype itself.
Observing the results, we noticed that although some students received prior instruction they still
scored low to very low scores because the kind of
Arabic instruction they had received was
Modern Standard Arabic (MSA) which is
considerably different from the Iraqi dialect
especially in pronunciation.
Some students with no measurable background,
scored higher than zero on the pretest. Upon
further discussion with them, we found that they had learned some Arabic words or phrases from
either their casual Arabic friends, from following up with the media on the Iraqi war, or from
their fellow soldiers who have been deployed. However, as the results show, the scores were not
significant.
Outcomes
The test scores of all 50 students with or without an instructor demonstrate that the prototype
improved their awareness of the language and culture. Looking at the correlation results we see
that students did considerably well in general. Scores showed that the prototype was effective
even when used alone. This is exhibited in the graphs #1 and #2 shown below.
10 Assessment and Evaluation
The average score of the posttest for students without the instructor and the average for the
posttest score with an instructor are 73.4% and 82.16% consecutively. Hence, it is evident that
there is an improvement in teaching and learning.
Graph #1 Graph #2
The main objective of this trial was to evaluate the extent of the effectiveness of this
prototype with the inclusion of face to face instructor interaction. Although the prototype
achieves its objectives alone, our study shows that the students who utilize the prototype in
conjunction with instructor exposure did notably better.
Mean 73.04 Standard Error 2.3017964 Median 73 Mode 75 Standard Deviation 11.508982 Sample Variance 132.4566667 Kurtosis -0.101805279 Skewness 0.078180312 Range 45 Minimum 50 Maximum 95 Sum 1826
Count 25
Mean 82.16 Standard Error 2.387243878 Median 82 Mode 79 Standard Deviation 11.93621939 Sample Variance 142.4733333 Kurtosis 1.719178611 Skewness -1.037186053 Range 49 Minimum 51 Maximum 100 Sum 2054 Count 25
11 Assessment and Evaluation
Graph #3 Comparison between Pre and Post Test: Prototype Without Instructor
Graph #4 Comparison between Pre and Post Test: Prototype With Instructor
Recommendations
Conducting this tryout has proved to be useful in providing tips that may enhance the learning
process for the deploying troops. It is vital for the military personal to interact with an instructor
especially if only to understand the culture of the language they are learning.
12 Assessment and Evaluation
This tryout has helped us realize that the prototype is only a catalyst to the learning process and
that instructors will always have a pertinent roll in language teaching.
t-Test: Two-Sample Assuming Equal Variances
Post-Test Score % Without
Instructor Post-Test Score % with
Instructor
Mean 73.04 82.16 Variance 132.4566667 142.4733333 Observations 25 25 Pooled Variance 137.465
Hypothesized Mean Difference 0
df 48 t Stat -2.750133502 P(T<=t) one-tail 0.004187424 t Critical one-tail 1.677224197 P(T<=t) two-tail 0.008374848 t Critical two-tail 2.010634722
Observing the p-value of the T-Test for scores of Post-Test with and without instructor we can
conclude that the null hypothesis is discredited and our original hypothesis is supported.
Comparison between Post Test Scores for taking Prototype with and without Instructor
13 Assessment and Evaluation
Appendix A
CONSENT TO PARTICIPATE IN RESEARCH
Title of Program: Basic Iraqi Survival Language Guide
You are invited to participate in a research study conducted by Selwa Alkadhi, student in IST 695, Educational
Assessment and Evaluation at California State University, Monterey Bay and instructor at the CE-LFAST
Program at the DLIFLC in Monterey, CA.
The purpose of this research is to evaluate the effectiveness of an instructional prototype on Language
familiarization and cultural training to evaluate the effect of instructor inclusion in a module presentation. You
were selected as a prospective participant in this study because you are currently a sample of a large population of
Iraqi Survival language learners. No particular level of computer experience is assumed or required for this study.
Taking part in this project is entirely up to you. You can choose whether or not to be in the study. If
you volunteer to be in this study, you may withdraw at any time without consequences of any kind. You
may also refuse to answer any questions you do not want to answer and still remain in the study. The
investigator may withdraw you from this research if circumstances arise that warrant doing so.
If you decide to participate in this research, you will be asked to do the following: Answer survey questions (three
general demographic questions, in addition to language based questions. The process will take between 30
minutes to 45 minutes to complete.
You are not being asked to sign any portion of the evaluation or to identify yourself in any way. Any information
that is obtained in connection with this study and that could be identified with you will remain completely
confidential. After the results are compiled, the paper surveys will be promptly destroyed.
If you want to know more about this research project or have questions or concerns, please contact Selwa at
Selwa.alkadhi@us.army.mil
You will get a copy of this consent form. Thank you for considering participation.
Sincerely,
Selwa Alkadhi
Consent Statement
I understand the procedure described. My questions have been answered to my satisfaction and I freely agree to
participate in this study. I know what I will have to do and that I can stop at any time.
I have been given a copy of this Consent Form.
Signature Date
14 Assessment and Evaluation
Appendix B
Module Evaluation
The purpose of this evaluation is to assess the effectiveness of the presented material throughout
this course and to make necessary revisions both in the content and structure of the course. Your
feedback is valuable to us and will be utilized to make necessary revisions to improve this
course. Please do not hesitate to make any comments or suggestions
1. Strongly Agree
2. Agree
3. Neutral
4. Disagree
5. Strongly Disagree
1 2 3 4 5
1.The training meets the course objectives.
2.The course materials were well prepared and
organized.
3. The concepts were delivered in a clear manner.
4. The length of the program was adequate in order to
meet goals.
5. The tasks and/or activities were well organized and
relevant to each subject.
6. Training units were detailed thoroughly.
7. The organization and layout of the design were easy
to follow.
8. Technology was well integrated into the course.
9. I will be able to apply what I learned here to my job.
15 Assessment and Evaluation
Appendix C
Pre-Test Stratification Survey
Please circle a suitable answer to the following questions:
1- Have you previously been deployed to the ME region during your military career?
2- Have you had any previous instruction in the Arabic language?
3- Have you had any other language instruction?
4- Have you been to the ME region for reasons other than the military?
Appendix D
Table#1 Raw data
Appendix E
yes no
yes no
yes no
yes no
Student Prior
Deployment Prior
Instruction Other
Languages Pre-Test Score %
Post-Test Score % Without
Instructor
1 no no no 0 75 2 yes no no 5 72 3 no no no 8 50 4 no no no 0 63 5 no no no 0 62 6 no no no 0 70 7 no yes no 25 68 8 no yes yes 55 87 9 no no yes 38 80
10 no no no 0 71 11 yes no no 10 68 12 yes no no 23 74 13 no no no 5 63 14 yes no no 0 75 15 no no no 0 63 16 yes no no 30 78 17 no yes yes 68 85 18 yes yes no 58 93 19 no no no 0 78 20 yes yes yes 89 95 21 no no no 0 52 22 yes no no 5 75 23 yes no no 8 73 24 no no no 0 65 25 yes yes yes 75 91
16 Assessment and Evaluation
Table #2 Raw data
Student Prior
Deployment Prior
Instruction Other
Languages Pre-Test Score %
Post-Test Score % with
Instructor
1 no no no 0 53 2 no no no 0 51 3 no no no 0 82 4 yes no no 6 79 5 no no no 8 77 6 no no no 0 76 7 yes yes yes 73 96 8 yes yes no 26 92 9 no no yes 15 98
10 no no yes 10 84 11 yes no no 15 77 12 no no no 0 81 13 no no no 0 79 14 yes no no 35 88 15 yes yes yes 78 100 16 no yes no 36 93 17 yes no no 12 86 18 yes yes yes 68 94 19 yes no yes 16 94 20 yes no no 5 75 21 no no no 0 80 22 no no no 0 79 23 yes no no 10 83 24 yes no no 5 85 25 no no no 0 72
Recommended