Upload
lina-shehu
View
218
Download
0
Embed Size (px)
Citation preview
7/30/2019 LA Test Project
1/33
Shehu & Tidwell 1
Lina Shehu
Jenna Tidwell
EDUC 8540
Dr. Jean Turner
14 May 2013
Original Test Project
China International Publishing Group
Original Test Specifications
With the purpose of using this assessment as a part of the curriculum of a cooperative exchange
program here at the Monterey Institute of International Studies, our test will be created with the
students participating in the China International Publishing Group as the primary target audience.
These students are adult Chinese nationals who currently work for the China International Publishing
Group, hereafter referred to as CIPG, and will be participating in an exchange program which will
allow them to live in Monterey and attend courses at MIIS during the fall semester of 2013. This
program is a cooperative venture between CIPG and MIIS, which aims to enhance the students' editing,
writing, translating, listening, reading, and speaking skills in the English language. Yet, to be chosen to
participate in this program, these students must already have a high command of the English level, and
most of the participants enter the program with a high to advanced level of proficiency in the target
language.
This particular assessment will be given at the beginning of the program and will act as a pretest
that allows instructors to gather information concerning what tasks the students can perform without
difficulty and to gauge the level of English that the learners possess at the beginning of the program in
order to better tailor the lessons and coursework to the needs of these individuals. This test will
measure two constructs: the students' reading and writing skills in the English language. There will also
be a self-assessment portion of the test that will further enhance the instructors' abilities to tailor the
7/30/2019 LA Test Project
2/33
Shehu & Tidwell 2
coursework to these particular students. Therefore, this test will separate consist of three sections: the
reading section, the writing section, and the self-assessment section. Each of these sections will be
scored and evaluated independently due to the fact that they will be used as a part of a needs
assessment.
Due to the fact that these students are currently employed by a publishing company, this test
aims to assess skills that could be considered quite important in the world of publishing, both reading
comprehension and writing in a foreign language. After reviewing previous student evaluations from
the years 2010 and 2012, it was concluded that CIPG students are generally quite interested in traveling
and American culture. Therefore, the authentic materials utilized in this assessment will include travel
based content.
Reading Sections
The reading sections of this test will aim to evaluate how well the students participating in the
CIPG program are able to read and comprehend a short passage in English. This task seems appropriate
for these particular learners due to the fact that they are expected to have an advanced level of English
and that they will need to be able to comprehend passages in order to correctly translate them into their
native language of Chinese. Continuing with the theme of traveling, the passages that the students are
asked to read will be authentic and will come directly from Sunset Magazine, which features articles
concerning Wine Country in California. An example of one of the passages that may appear on the test
would be as follows:
The sky is dark, the kind of inky black that only exists without a single streetlight. We bouncedown the washboard dirt road and, when it forks, turn toward our destination: the areas best new
restaurant. Or is it the other way? The road keeps twisting and branching off. My husband, Todd, grips
the official map of Baja Californias main wine region, Valle de Guadalupe. All day we used this map,with its cartoonish purple icons of grape bunches indicating wineries, to sip our way among the
valleys 50 wineries. Visiting them often means venturing off the areas three paved roads onto rutted
dirt ones, but Todd laughs now at our assumption that the straight brown dotted line we took to be ashortcut would translate to a straight road. He tosses the map to the floor, I pull a U-turn, and we
carefully retrace our path back to the highwaya paved two-laner that links the port city of
7/30/2019 LA Test Project
3/33
Shehu & Tidwell 3
Ensenada, 20 miles to the west, to the border town of Tecate.
This is a booming wine country, with new tasting rooms, a couple of fine restaurants, and
hotels with high-thread-count sheets, but it still has a wild side. And for a moment, the utter darknessgives me a familiar feeling, like the one I get hiking in the wilderness or paddling my surfboard into a
wave. The natural world is powerful.
After reading a passage such as this one, students will be asked to answer eight multiple-
choice questions, which have four optional answers each: A, B, C, and D. Students will then repeat this
process and will read a second passage and answer eight more multiple-choice questions. Therefore, in
total, the students will answer a total of sixteen multiple-choice questions that will all be scored
objectively. An example of a multiple-choice question that may appear on the final version of the test
would be as follows:
The main theme of this article is to:
A. talk about how easy it is to get lost in California.
B. introduce one of Californias wineries.C. express how difficult it is to travel with a spouse.
D. none of the above
An answer key will be created that instructors will be able to use to score this portion of the test.
Each multiple-choice question will be worth one point. A student will earn one point if he or she
answers the question correctly but will not be penalized for answering a question incorrectly. If a
student answers 0-4 questions correctly, he or she will be considered to have a low level of reading
comprehension in English. Answering 5-8 questions correctly will be considered as an intermediate
level, answering 9-12 questions will be considered as a high level, and answering 13-16 questions
correctly will be considered as an advanced level.
Writing Sections
For the writing portion of this assessment, the theme of traveling will continue, but students will
then be asked to produce an original text. This portion of the test will also be split into two sections,
and the students will be asked to create two original short stories. The genre of short story was chosen
7/30/2019 LA Test Project
4/33
Shehu & Tidwell 4
due to the fact that the students are working in a publishing house and also in hopes that this task would
be fun and engaging for the students. Writing prompts are often open-ended, and this test will follow
that example. The students will be provided with two travel related photographs and asked to compose
a short story of no more than 150 words. An example of a photograph that might appear on the final
version of the test could look as follows:
2012 COND NAST. ALL RIGHTS RESERVED.
The stories that the students compose will be subjectively scored based on a rubric used by
ACTFL, which scores students based on criteria in six different sections: task completion,
comprehensibility, level of discourse, vocabulary, language control, and mechanics. Students will be
awarded a score between 0 and 6 based on the quality of their composition, with a perfect score being
36.
Self-assessment Section
For the self-assessment portion of this test, students will be asked to rate their own abilities
based on how well they feel they can accomplish a certain task. For example, students will be presented
with a statement such as, I can understand native speakers of English at the grocery store, and will
then be asked to rate themselves on a likert scale with one being I strongly agree, two being I agree,
three being I disagree, and four being I strongly disagree. The statements that will be presented to the
7/30/2019 LA Test Project
5/33
Shehu & Tidwell 5
students in the self-assessment portion of the test will be based on the curricular goals of the CIPG
exchange program. For example, students will be asked to rate their own speaking, listening, reading,
and writing skills in English due to the fact that this program aims to improve these skills, and this
information could greatly influence and aid instructors and program directors in the subsequent design
of the coursework and lesson plans. The scores that students receive on this portion of the test will be
interpreted only in the sense that they will affect the needs assessment process of the instructors of the
program.
Development of the Test
We decided to assess students reading and writing skills in English, because CIPG students will
have to read and comprehend various texts in their classes, and will additionally be required to keep a
blog about their classes, tasks, and extracurricular activities while at MIIS. The decision to create a
diagnostic test was made because we wanted to design a curriculum that is appropriate to their level of
English.
The CIPG program coordinator here at MIIS, Ms. Heather Kokesch, informed us that the
majority of students from CIPG are highly proficient in English. Therefore we chose texts that are
slightly challenging, Flesch-Kincaid Reading Ease of 60 and 59.4. but appropriate to an advanced
speaker of English.
The objectively scored portion of the text features two texts, the first with 7 multiple choice
items and the second with 8 multiple choice items. The subjectively scored portion calls for a minimum
of one written paragraph. Originally, we only had one text in the objectively scored portion, but
decided to include another text for several reasons. First, it was almost impossible to have 15 multiple
choice items on a text that has a readability score of 60. The questions would have simply become
redundant. Second, we wanted to include two different genres on travel. Our multiple choice item
questions were revised several times because the goal was to create a challenging task for our students
7/30/2019 LA Test Project
6/33
Shehu & Tidwell 6
but not overwhelming. Furthermore, the wrong answers were revised so that they are not immediately
obvious to be the wrong answer.
Original Test
China International Publishing Group
Diagnostic Test
Directions:Please read the passage below and answer the following multiple choice questions by
circling the letter of the correct answer.
The Road to Anatolia
Yugoslav-Greek frontier
When you leave Yugoslavia for Greece, the blue-the color of the Balkans- follows you, but its naturechanges; you move from a slightly muted night-blue to an intensely gay sea-blue, which affects the
nervous system like caffeine. That's just as well, as the rhythm of conversations and exchanges speeds
up considerably. You've got into the habit of explaining things slowly-usually twice rather than once-and of pausing over words for comprehension to catch up...
Constantinople
The very morning we arrived, we ferried the car across to the Asian shore and were prowling through
the little streets of the Moda neighborhood in search of a lodging that would beckon to us, when a frailbut imperious voice calling out in French made us turn round. it belonged to a large woman with snow-
white hair, wearing a heavy amethyst brooch and elegant mourning. From the top of her steps she
gazed thoughtfully at our bags, as if they reminded her of something, and asked what we were looking
for. We explained.'My season ended last week, but I've kept on my servants and I rather like travelers. You can stay here.'
With her cigarette-holder, she indicated a little gold plate above the door: Moda-Palas.
In silence our bags were taken across a sombre Victorian dining room. On the sideboard, a mustard catwas asleep between ornate Christofle teapots. The room gave on to a faded garden, and had a light but
distinct odor of polish and mildew. Except for a chambermaid, and madame Wanda, the proprietress,
the hotel was deserted and, with its shutters closed, more intimidating than a tomb.
1. The sentence an intensely gay-sea blue, which affects the nervous system like caffeine suggests that
the author is:
A. negatively stimulatedB. positively stimulated
C. not stimulated
2. What does and of pausing over words for comprehension to catch up mean?
A. The author believes one must speak slowly in order to be understood.
B. The author is making fun of the people with whom he is speaking.C. The author conceives of comprehension as a person that is physically catching up.
7/30/2019 LA Test Project
7/33
Shehu & Tidwell 7
3. The sentence we ferried the car across to the Asian shore means:
A. The author drove really fast across a bridge to the Asian shore.B. The author drove a stolen car across to the Asian shore.
C. The author drove his car onto a large boat that carried it across to the Asian shore.
4. What does in search of a lodging that would beckon to us mean?A. The author is searching for lodging that appeals to him.
B. The author means to say that he is unable to find lodging.
C. The author is commenting on the neighborhoods shabby lodgings.
5. When the French woman says my season ended last week, she means:
A. the time for renting out her lodgings is pastB. She is complaining about the change of seasons.
C. She is implying that she will charge them double for lodging.
6. Why does the author compare the hotel to a tomb?A. because it smells like polish and mildew
B. because the hotel is quiet and dark
C. the hotel is built on an old cemetery
7. What is the genre of this passage?
A. It is an excerpt from a major newspaper.B. It is an excerpt from a travelogue.
C. It is an excerpt from a complaint letter.
Directions:Please read the passage below and answer the following multiple choice questions by
circling the letter of the correct answer.
Explore Valle de Guadalupe Wine Country
The sky is dark, the kind of inky black that only exists without a single streetlight.We bounce down the
washboard dirt road and, when it forks, turn toward our destination: the areas best new restaurant. Oris it the other way? The road keeps twisting and branching off.
My husband, Todd, grips the official map of Baja Californias main wine region, Valle de Guadalupe.All day we used this map, with its cartoonish purple icons of grape bunches indicating wineries, to sip
our way among the valleys 50 wineries. Visiting them often means venturing off the areas three paved
roads onto rutted dirt ones, but Todd laughs now at our assumption that the straight brown dotted linewe took to be a shortcut would translate to a straight road. He tosses the map to the floor, I pull a U-
turn, and we carefully retrace our path back to the highwaya paved two-laner that links the port
city of Ensenada, 20 miles to the west, to the border town of Tecate.
This is a booming wine country, with new tasting rooms, a couple of fine restaurants, and hotels with
7/30/2019 LA Test Project
8/33
Shehu & Tidwell 8
high-thread-count sheets, but it still has a wild side. And for a moment, the utter darkness gives me a
familiar feeling, like the one I get hiking in the wilderness or paddling my surfboard into a wave. Thenatural world is powerful.
Only a few minutes later, we walk through the door of Corazn de Tierra. The sleek one-roomrestaurant is a Venice-modern oasis, with a large garden off the back deck. Diners can either sit facingrows of beets, basil, and arugula or watching chef Diego Hernandez and his team grilling, searing, and
assembling food in the open kitchen. There are no menus. Its nature that decides, says Hernandez,
29, who opened Corazn de Tierra two years ago and cooks and talks about food with equal gusto.Here in Baja, we have a palette of flavors, like a painter.
Each dish that comes is a surprise: a pesto of beet greens served with a thick hunk of rosemary breadthats been both baked and smoked. A salad of broccoli, radishes, and Ramonetti (a soft, local cheese
akin to a blend of brie and manchego) with an orange calendula flower and parsnip pure artfully
arranged on a slab of slate. Then its a piece of perfectly seared fresh tuna bathed in a tangy sour-milk
and mint-oil broth, served in a warm stone bowl. After that, roasted partridge with carrots, roastedpumpkin pure, and beef sauce, paired with a spicy Vena Cava Tempranillo from the winery at Villa del
Valle, the B&B where Corazn de Tierra is located. Finally, its Hernandezs dessert oeuvre, each
tasting portion made from fresh oranges: ice cream, panna cotta, marmalade cake, and a cookie.
Hernandez is building on the regions farm-to-table legacy established by the valleys first and most
renowned farm-to-table restaurant, Laja, where during a delicious, epic meal the night before, I learnedthat the restaurants refrigerators and freezers often sit emptythe fresh ingredients never make it
there.
We grew up eating sashimi and ceviches, says Hernandez of his childhood in Ensenada. Now he
delivers his own sophisticated takea mix of Mexican, Mediterranean, and Asianhoned at top
restaurants in Mexico City and Tijuana. Hernandez says that visitors who come expecting only
traditional Mexican cuisine are surprised. The truth is, this is the other Mexico.
1. Why does the author use the metaphor inky black to describe the sky?
A. to convey the darkness of the sky to the readerB. to explain the need for streetlights
C. to lament getting lost
D. to show off her writing skills
2. The sentence the road keeps twisting and branching off is used to convey:
A. the road looks like a tree
B. the dark and labyrinthine settingC. the authors hatred of rural areas
D. the need for a map
3. The second paragraph is about:
A. the difficulty of traveling with a spouse
B. some of Californias roadsC. some of Californias wineries in the Baja region
D. the reasons why the author and her husband got lost
7/30/2019 LA Test Project
9/33
Shehu & Tidwell 9
4. The word booming can be understood as:
A. extraordinarily loudB. quickly prospering
C. slowly developing
D. exceptionally beautiful
5. Why does the author say the natural world is powerful?
A. because she is frightened of it
B. because despite the darkness in the wine country, she can still see its beautyC. because she loves hiking and surfboarding
D. because despite the fine restaurants and hotels, the area has a wild side
6. What does Chef Hernandez mean when he says its nature that decides?
A. He is referring to the power of the natural world.
B. He is referring to his gardens abundance.
C. He is referring to the vegetables that are in season.D. He is referring to the wild atmosphere in his restaurant.
7. Is the authors view of the restaurant that of:A. appreciation
B. dismay
C. shockD. disappointment
8. In this context what does farm-to-table mean?A. The restaurant, Laja, is the only one who does it.
B. The ingredients used at the restaurant are very fresh.
C. The majority of Mexican restaurants are expensive.
D. The food at Corazon de Tierra is quite unusual.
7/30/2019 LA Test Project
10/33
Shehu & Tidwell 10
Directions:Look at the picture below. Imagine that the two men in the picture are traveling in
California and write a brief (one paragraph) short story about their adventure.
7/30/2019 LA Test Project
11/33
Shehu & Tidwell 11
Directions: Considering your own abilities, answer the following questions by circlingTrueorFalse.
1. I can read and understand a magazine in English.
True False
2. I can read and understand academic texts in English.
True False
3. Writing fiction short stories in English is difficult for me.
True False
4. When I write in English, I wish that my vocabulary was larger.
True False
5. The questions about the reading passage were easy.
True False
6. I had to reread the text many times to answer the questions about the reading passage.
True False
Scoring Key for the Objectively Scored Section
First Text
1. B
2. A3. C
4. A
5. A6. B
7. B
Second Text1. A
2. B
3. C4. B
5. D
6. C7. A
8. B
7/30/2019 LA Test Project
12/33
Shehu & Tidwell 12
Information on the scoring protocol for the subjectively scored section can be found in the
revised test specifications section in the form of a rubric used as a part of the TOEFL PBT Writing and
Structure Section.
Guide to Test Administration
In order to administer this test, instructors involved in the CIPG program will not need to
undergo extensive training. Test administrators will be in charge of transporting paper versions of the
test to the testing site, along with extra writing utensils in case of test taker emergency. Test
administrators will control the classroom and ensure that students are seated and remain silent
throughout the duration of the test. Students will be allotted the time necessary to complete the test, and
they will not be allowed to use any outside sources of aide such as a dictionary or the internet. Test
administrators will pass out the tests to the students, remain in the classroom as the test is being given,
collect the tests as students complete them, and transport the tests away from the test site once each
student has finished. Test administrators will also not be allowed to answer any questions concerning
the content of the test before or during the actual test administration to ensure that students complete
the test without assistance.
Actual Test Administration
Due to time constraints and a lack of eager and willing participants, the actual trial
administration of this test was not performed in a controlled setting as it idealistically would be when
functioning as a part of the CIPG program. The test designers recruited a total of eight nonnative
English speakers to complete the test, many of whom were students at the Monterey Institute of
International Studies. The test takers were recruited on the basis that they were, first and foremost,
nonnative English speakers, and secondly, that they were able to communicate with ease with native
speakers of English. These requirements were set in hopes of emulating the levels of English of the
China International Publishing Group students.
7/30/2019 LA Test Project
13/33
Shehu & Tidwell 13
The recruited test takers were allowed to take their tests home with them and return them to the
test designers at a later time. Yet, the test takers were instructed to refrain from using any type of
dictionary or outside aide while they completed the test, although there was no set time limit for
completing the test. While there is no certainty that the test takers did not consult a dictionary or
another source of aide while completing the test, the test designers found that allowing participants to
take their tests home was the only option to successfully recruit enough participants. All of the tests
were given to the test takers in a paper format; however, some test takers chose to scan their tests after
completion and returned them electronically.
Revised Test Specifications
With the purpose of using this assessment as a part of the needs assessment process for a
cooperative exchange program here at the Monterey Institute of International Studies, our test will be
created with the students participating in the China International Publishing Group as the primary target
audience. These students are adult Chinese nationals who currently work for the China International
Publishing Group, hereafter referred to as CIPG, and will be participating in an exchange program
which will allow them to live in Monterey and attend courses at MIIS during the fall semester of 2013.
This program is a cooperative venture between CIPG and MIIS, which aims to enhance the students'
editing, writing, translating, listening, reading, and speaking skills in the English language. Yet, to be
chosen to participate in this program, these students must already have a high command of the English
level, and most of the participants enter the program with a high to advanced level of proficiency in the
target language.
This particular assessment, which will be diagnostic in nature, will be given at the beginning of
the program and will act as a pretest that allows instructors to gather information concerning what tasks
the students can perform without difficulty and to gauge the level of English that the learners possess at
the beginning of the program in order to better tailor the lessons and coursework to the needs of these
7/30/2019 LA Test Project
14/33
Shehu & Tidwell 14
individuals. This test will measure two constructs: the students' reading and writing skills in the English
language. There will also be a self-assessment portion of the test that will further enhance the
instructors' abilities to tailor the coursework to these particular students. Therefore, this test will
separate consist of three sections: the reading section, the writing section, and the self-assessment
section. Each of these sections will be scored and evaluated independently due to the fact that they will
be used as a part of a needs assessment, and the designers of this test would like to know where each
student stands in the areas that are being assessed. By having the students complete a reading
comprehension section, generate a written text, and assess their own ability levels, the instructors of the
course will be able to better understand the levels of each student in order to better tailor the course to
their specific needs.
Due to the fact that these students are currently employed by a publishing company, this test
aims to assess skills that could be considered quite important in the world of publishing, both reading
comprehension and writing in a foreign language. After reviewing previous student evaluations from
the years 2010 and 2012, it was concluded that CIPG students are generally quite interested in traveling
and American culture. Therefore, the authentic materials utilized in this assessment will include travel
based content.
Reading Sections
The reading sections of this test will aim to evaluate how well the students participating in the
CIPG program are able to read and comprehend a short passage in English. This task seems appropriate
for these particular learners due to the fact that they are expected to have an advanced level of English
and that they will need to be able to comprehend passages in order to correctly translate them into their
native language of Chinese. Continuing with the theme of traveling, the passages that the students are
asked to read will be authentic and will come directly from Sunset Magazine, which features articles
concerning Wine Country in California. In order to make the course engaging and interesting for the
7/30/2019 LA Test Project
15/33
Shehu & Tidwell 15
CIPG students, the designers of the curriculum have decided to create a content-based curriculum
focused on traveling. Therefore, the designers of this test made the decision to also use travel-based
content in this diagnostic test. An example of one of the passages that may appear on the test would be
as follows:
The Road to Anatolia
Yugoslav-Greek frontier
When you leave Yugoslavia for Greece, the blue-the color of the Balkans- follows you, but its naturechanges; you move from a slightly muted night-blue to an intensely gay sea-blue, which affects the
nervous system like caffeine. That's just as well, as the rhythm of conversations and exchanges speeds
up considerably. You've got into the habit of explaining things slowly-usually twice rather than once-
and of pausing over words for comprehension to catch up...
Constantinople
The very morning we arrived, we ferried the car across to the Asian shore and were prowling through
the little streets of the Moda neighborhood in search of a lodging that would beckon to us, when a frail
but imperious voice calling out in French made us turn round. it belonged to a large woman withsnow-white hair, wearing a heavy amethyst brooch and elegant mourning. From the top of her steps
she gazed thoughtfully at our bags, as if they reminded her of something, and asked what we were
looking for. We explained.
'My season ended last week, but I've kept on my servants and I rather like travelers. You can stay here.'With her cigarette-holder, she indicated a little gold plate above the door: Moda-Palas.
In silence our bags were taken across a sombre Victorian dining room. On the sideboard, a mustard cat
was asleep between ornate Christofle teapots. The room gave on to a faded garden, and had a light butdistinct odor of polish and mildew. Except for a chambermaid, and madame Wanda, the proprietress,
the hotel was deserted and, with its shutters closed, more intimidating than a tomb.
The following data were retrieved from Readability-Score.com. The first text, The Road to
Anatolia has a Flesch-Kincaid Reading Ease of 60. The second text, Explore Valle de Guadalupe Wine
Country, has a Flesch-Kincaid Reading Ease of 59.4. Since both texts have a very similar score, the
data below apply to both texts. The Flesch-Kincaid Scores range from 0-100. Higher scores indicate
easier to read material. Scores over 22 should generally be taken to mean graduate level text. Scores
ranging from 60-70 should be understood by 13 to 15 year old students. The Gunning fog index for our
texts was 11.7. Texts that are accessible for a wide audience generally need a fox index less than 12.
The Average Grade Level for our texts was 10.4. That means the texts are comprehensible for high
7/30/2019 LA Test Project
16/33
Shehu & Tidwell 16
school sophomores.
The multiple choice items on our objectively scored portion of the test, were designed to
measure students comprehension of a passage from two different genres: a travelogue, and a magazine
article. The CIPG students for whom this test was designed will be reading similar texts in their English
classes at MIIS. We wanted to assess their reading comprehension so that we may develop a curriculum
applicable to their skill and level of English. Furthermore the multiple choice items measure students
knowledge of English metaphors and idiomatic expressions. We believe both are important since they
add to a students communicative competence. In addition we purposefully chose texts about travel,
because our curriculum will include travel as complementary to students studies and we wanted to
familiarize them with different genres on travel.
After reading a passage such as this one, students will be asked to answer eight multiple-
choice questions, which have four optional answers each: A, B, C, and D. Students will then repeat this
process and will read a second passage and answer eight more multiple-choice questions. Therefore, in
total, the students will answer a total of sixteen multiple-choice questions that will all be scored
objectively. An example of a multiple-choice question that may appear on the final version of the test
would be as follows:
2. What does and of pausing over words for comprehension to catch up mean?
A. The author believes one must speak slowly in order to be understood.
B. The author is making fun of the people with whom he is speaking.C. The author conceives of comprehension as a person that is physically catching up.
An answer key will be created that instructors will be able to use to score this portion of the test.
Each multiple-choice question will be worth one point. A student will earn one point if he or she
answers the question correctly but will not be penalized for answering a question incorrectly. The
designers of this test have created their own scoring guide and will interpret the levels of the students
based on how many correct responses they provide, which therefore implies how much of the text a
7/30/2019 LA Test Project
17/33
Shehu & Tidwell 17
student understood. Students will be evaluated on their comprehension of the text on four levels. If a
student answers 0-4 questions correctly, he or she will be considered to have understood 25% or less of
the text, therefore having a low level of reading comprehension in English. Answering 5-8 questions
correctly will imply that a student understood 50% or less of the text, and he or she will be considered
to have an intermediate level. Answering 9-12 questions will be considered as a high level due to the
student being able to understand 51-75% of the text, and answering 13-16 questions correctly will be
considered as an advanced level, as a student will have been able to comprehend 76-100% of the given
text.
Writing Sections
For the writing portion of this assessment, the theme of traveling will continue, but students will
then be asked to produce an original text. This portion of the test will also be split into two sections,
and the students will be asked to create two original short stories. The genre of short story was chosen
due to the fact that the students are working in a publishing house and also in hopes that this task would
be fun and engaging for the students. Writing prompts are often open-ended, and this test will follow
that example. The students will be provided with two travel related photographs and asked to compose
a short story of no more than 150 words. An example of a photograph that might appear on the final
version of the test could look as follows:
7/30/2019 LA Test Project
18/33
Shehu & Tidwell 18
The stories that the students compose will be subjectively scored based on a rubric used to score
the TOEFL PBT, which can be seen below.
Writing Scoring GuideThe following scoring guidelines relate to the TOEFL PBT Test Writing & Structure section.
Score of Six
An essay at this level:shows effective writing skills
is well organized and well developed
uses details clearly and properly to support a thesis or illustrate ideas
displays consistent ability in the use of language
demonstrates variety in sentence structure and proper word choice
Score of Five
An essay at this level:
may address some parts of the task more effectively than othersis generally well organized and developed
uses details to support a thesis or illustrate an idea
displays ability in the use of the language
shows some variety in sentence structure and range of vocabulary
Score of Four
An essay at this level:
7/30/2019 LA Test Project
19/33
Shehu & Tidwell 19
addresses the writing topic adequately but does not meet all of the goals of the task
is adequately organized and developed
uses some details to support a thesis or illustrate an idea
shows adequate but possibly inconsistent ability with sentence structuremay contain some usage errors that make the meaning unclear
Score of Three
An essay at this level may reveal one or more of the following weaknesses:
inadequate organization or development
poor choice of details or does not provide enough details to support or illustrate generalizations
a noticeably improper choice of words or word forms
numerous errors in sentence structure and/or usage
Score of Two
An essay at this level is seriously flawed by one or more of the following weaknesses:
serious disorganization or underdevelopment
little or no detail, or irrelevant specifics
serious and frequent errors in sentence structure or usage
serious problems with focus
Score of One
An essay at this level:
may be incoherentmay be undeveloped
may contain severe and persistent writing errors
Score of 0
An essay will be rated 0 if it:
contains no response
merely copies the topic
is off-topic, written in a foreign language or consists only of keystroke characters
Self-assessment Section
For the self-assessment portion of this test, students will be asked to rate their own abilities
based on how well they feel they can accomplish a certain task. For example, students will be presented
with a statement such as, Writing fiction short stories in English is difficult for me, and will then be
asked to rate themselves by answering either true or false in response to the given statement. The
7/30/2019 LA Test Project
20/33
Shehu & Tidwell 20
statements that will be presented to the students in the self-assessment portion of the test will be based
on the curricular goals of the CIPG exchange program. For example, students will be asked to rate their
own reading and writing skills in English due to the fact that this program aims to improve these skills,
and this information could greatly influence and aid instructors and program directors in the subsequent
design of the coursework and lesson plans. The scores that students receive on this portion of the test
will be interpreted only in the sense that they will affect the needs assessment process of the instructors
of the program. Therefore, the activities and lessons planned for the course may be subject to change
after the instructors obtain this information and are aware of what areas in which students believe they
need help.
Descriptive Statistics
For the subjectively scored portion of the test, the test takers were scored using a rubric with a
scale of 1 to 6. We entered the test takers scores on the computer program known as R in order to
calculate the mean and the standard deviation (sd). Two different scorers scored the subjectively scored
portion of the assessment. The first set of scores was: 5, 4, 4, 4, 4, 6, 6, 3. The second set of scores was:
4, 4, 4, 4, 5, 6, 6, 2. The mean for the first set was: 4.5, and the mean for the second set was: 4.375,
which shows that the first rater gave slightly higher scores than the second. The mode for the scores of
the subjectively scored portion of the exam was 4, which appeared four times in both sets of scores.
The median of these scores was also 4. The sd for the first set was: 1.069045, and the sd for the second
set was: 1.30247.
As for the objectively scored section, there were 15 multiple-choice questions that were worth
one point each, making the section worth a total of 15 points. The scores for this section were: 4, 9, 11,
11, 11, 11, 12, and 13. The mode for this section of the test was 11, which was the median score as
well. The mean of these scores was 10.25. The standard deviation for the scores in this section was
2.56.
7/30/2019 LA Test Project
21/33
Shehu & Tidwell 21
Internal Consistency
In order to calculate the internal consistency of the objectively scored items, the test designers
entered the scores into the reliability calculator and found that the estimated internal consistency was
0.91 using the Spearman-Brown Prophecy formula. This number was much larger than anticipated and
shows that this test has a high rate of internal consistency, which speaks favorably of the design of this
assessment.
Item Difficulty
According to the chart below, questions 3, 7, and 14 were too easy for test takers because every
single tester answered it correctly, whereas question 10 was too difficult because no one answered
correctly. Questions 4, 5, 8, 13 and 15 were also a little easy since the majority of test takers answered
correctly, whereas question 2 was quite difficult since relatively few people responded with the correct
answer.
7/30/2019 LA Test Project
22/33
Shehu & Tidwell 22
Question Number Item Facility
1 .5
2 .25
3 1
4 .875
5 .875
6 .75
7 1
8 .875
9 .375
10 0
11 .75
12 .375
13 .875
7/30/2019 LA Test Project
23/33
Shehu & Tidwell 23
Item Discrimination
Question Number High Low Item Discrimination
1 1 0 1
2 0.5 0.5 0
3 1 1 0
4 1 0.5 0.5
5 1 0.5 0.5
6 1 0.5 0.5
7 1 1 0
8 0.5 0.5 0
9 0.5 0 0.5
10 0 0 0
11 1 0 1
12 1 0 1
13 1 0.5 0.5
14 1 1 0
15 1 0.5 0.5
The test designers found that item 10 did not discriminate well due to the fact that none of the
test takers answered this item correctly. Similarly, since all of the examinees answered items 3, 7, and
14 correctly, it was determined that these questions did not discriminate well either. Therefore, these
questions could possibly be deemed as too easy for the test takers, just as item 10 could be viewed as
too difficult.
Distractor Content Analysis
We took great care in creating multiple choice items answers, which were challenging but not
overwhelming. Obviously, some were simply too challenging and others were too easy, nonetheless we
feel that we inserted distractor content in our multiple choice item answers that made our test
interesting and creative. For example: on question number 1, The sentence an intensely gay sea-blue,
7/30/2019 LA Test Project
24/33
Shehu & Tidwell 24
which affects the nervous system like caffeine suggests that the author is a. positively stimulated, b.
negatively stimulated and c. not stimulated, we thought that since many people are negatively affected
by caffeine they may be tempted to just answer b, even though the sentence in its context suggests that
the author was positively stimulated. For question number 2 we wanted to test knowledge of idiomatic
expression thus: what does pausing over words for comprehension to catch-up mean? one of the
distractors we provided was c. the author conceives of comprehension as a person that is physically
catching up. This idiomatic expression, however, means to be understood. On question 6 we asked:
why does the author compare the hotel to a tomb? and one of the distractors was answer c. because it
was built on an old cemetery.
Subjectively Scored Writing Section
Interrater Reliability
For the subjectively scored writing section of the test, both of the test designers used the TOEFL
PBT Writing Rubric, as mentioned in the revised test specifications, to score the narratives produced by
the test takers. After both test designers had independently scored all of the tests, the interrater
reliability was calculated using Spearman rho with the aide of the computer program known as R.
Through the use of this program, the interrater reliability was found to be .83, which was quite high for
this type of diagnostic assessment. This number also came as a shock to the test designers since neither
had participated in any formal training on how to properly use the writing rubric for scoring the tests,
and the expectation was that the interrater reliability would be quite for two novice test designers and
scorers. However, inferences could be made that the rubric used was very clear and self-explanatory
due to the fact that the scorers were able to achieve such a high rate of interrater reliability without any
formal training.
Content Analysis
The content of the narratives that were written by the test takers for the subjectively scored
7/30/2019 LA Test Project
25/33
Shehu & Tidwell 25
section of the assessment was also quite surprising for the test designers. Overall, the narratives that
were written were well-organized, creative, quite detailed, and relatively free of major grammatical
errors. All of the narratives were one paragraph in length, as specified in the directions, and some even
contained extensive dialogue. This section of the test was deemed successful as little to no changes or
revisions were needed due to the fact that the test takers successfully followed all of the directions, a
substantial amount of ratable text was produced, and future instructors of the CIPG program would be
able to easily decipher the levels of incoming students' written English based on their completion of
this task.
Self-Assessment Section
After analyzing the answers provided in the self-assessment portion of the exam, it was
concluded that many of the test takers found this assessment to be difficult. Many of the students
circled the word false in response to the statement, The questions about the reading passage were
easy. Also, many of the students claimed that they were forced to reread the passage several times in
order to answer the multiple-choice questions with accuracy. In addition, many of the test takers
claimed that they wished that their vocabularies in English were larger and that they found it difficult to
write short fiction narratives in English. Therefore, these answers could help the instructors working in
the CIPG program choose what to focus their lessons on as they begin preparing for the upcoming
course.
Yet, there were portions of the self-assessment section that could and should be improved upon
before the test is given again. When the test was first given, the self-assessment section asked the test
takers to circle either true or false in response to a given statement. However, after analyzing the
data, the test designers concluded that the responses to these statement could have been more useful if
the test takers were required to rate themselves on a Likert scale. With this revision, test takers would
respond to the same statements, but instead of circling true or false, they would be asked to circle a
7/30/2019 LA Test Project
26/33
Shehu & Tidwell 26
number between one and four, with one being strongly disagree, two being disagree, three being agree,
and four being strongly agree. This type of scale was mentioned in the original test specifications, and
the test designers agree that these types of answers could have provided them with more specific
information about how the students rate their own abilities in both reading comprehension and writing
in English.
Revisions to the Objectively Scored Section
After analyzing item facility we have decided to revise the following questions: 2 and 10,
because very few people answered 2 correctly, and no one answered 10 correctly. Below are the
original questions followed by the revisions.
2. What does and of pausing over words for comprehension to catch up mean?A. The author believes one must speak slowly in order to be understood.
B. The author is making fun of people with whom he is speaking.
C. The author conceives of comprehension as a person that is physically catching up.
Many test takers answered c, so we would revise c into: the author is irritated that they dont speak his
native language.
10. The second paragraph is about:
A. the difficulty of traveling with a spouseB. some of Californias roads
C. some of Californias Wineries in the Baja region
D. the reason why the author and her husband got lost
None of the test takers answered question 10 correctly and we believe that is because the second
paragraph mentions roads and getting lost, in addition to Californias Wineries. We would revise the
second question into: Reread the second paragraph. What do you think is the authors main purpose for
this trip?
10. What is the main idea of the second paragraph?A. The difficulty of traveling with a spouse.
B. They are going on an adventure with no destination in mind.
C. They want to visit Californias Wineries in the Baja region.D. It is a complaint about Californias highways.
7/30/2019 LA Test Project
27/33
Shehu & Tidwell 27
Revised Test
China International Publishing Group
Diagnostic Test
Directions:Please read the passage below and answer the following multiple choice questions bycircling the letter of the correct answer.
The Road to Anatolia
Yugoslav-Greek frontier
When you leave Yugoslavia for Greece, the blue-the color of the Balkans- follows you, but its naturechanges; you move from a slightly muted night-blue to an intensely gay sea-blue, which affects the
nervous system like caffeine. That's just as well, as the rhythm of conversations and exchanges speeds
up considerably. You've got into the habit of explaining things slowly-usually twice rather than once-
and of pausing over words for comprehension to catch up...
Constantinople
The very morning we arrived, we ferried the car across to the Asian shore and were prowling through
the little streets of the Moda neighborhood in search of a lodging that would beckon to us, when a frail
but imperious voice calling out in French made us turn round. it belonged to a large woman with snow-white hair, wearing a heavy amethyst brooch and elegant mourning. From the top of her steps she
gazed thoughtfully at our bags, as if they reminded her of something, and asked what we were looking
for. We explained.'My season ended last week, but I've kept on my servants and I rather like travelers. You can stay here.'
With her cigarette-holder, she indicated a little gold plate above the door: Moda-Palas.
In silence our bags were taken across a sombre Victorian dining room. On the sideboard, a mustard cat
was asleep between ornate Christofle teapots. The room gave on to a faded garden, and had a light butdistinct odor of polish and mildew. Except for a chambermaid, and madame Wanda, the proprietress,
the hotel was deserted and, with its shutters closed, more intimidating than a tomb.
1. The sentence an intensely gay-sea blue, which affects the nervous system like caffeine suggests that
the author is:
A. negatively stimulatedB. positively stimulated
C. not stimulated
2. What does and of pausing over words for comprehension to catch up mean?A. The author believes one must speak slowly in order to be understood.
B. The author is making fun of people with whom he is speaking.
C. The author conceives of comprehension as a person that is physically catching up.
3. The sentence we ferried the car across to the Asian shore means:
A. The author drove really fast across a bridge to the Asian shore.B. The author drove a stolen car across to the Asian shore.
C. The author drove his car onto a large boat that carried it across to the Asian shore.
7/30/2019 LA Test Project
28/33
Shehu & Tidwell 28
4. What does in search of a lodging that would beckon to us mean?
A. The author is searching for lodging that appeals to him.B. The author means to say that he is unable to find lodging.
C. The author is commenting on the neighborhoods shabby lodgings.
5. When the French woman says my season ended last week, she means:A. the time for renting out her lodgings is past
B. She is complaining about the change of seasons.
C. She is implying that she will charge them double for lodging.
6. Why does the author compare the hotel to a tomb?
A. because it smells like polish and mildewB. because the hotel is quiet and dark
C. the hotel is built on an old cemetery
7. What is the genre of this passage?A. It is an excerpt from a major newspaper.
B. It is an excerpt from a travelogue.
C. It is an excerpt from a complaint letter.
Directions:Please read the passage below and answer the following multiple choice questions by
circling the letter of the correct answer.
Explore Valle de Guadalupe Wine Country
The sky is dark, the kind of inky black that only exists without a single streetlight.We bounce down the
washboard dirt road and, when it forks, turn toward our destination: the areas best new restaurant. Or
is it the other way? The road keeps twisting and branching off.
My husband, Todd, grips the official map of Baja Californias main wine region, Valle de Guadalupe.
All day we used this map, with its cartoonish purple icons of grape bunches indicating wineries, to sip
our way among the valleys 50 wineries. Visiting them often means venturing off the areas three pavedroads onto rutted dirt ones, but Todd laughs now at our assumption that the straight brown dotted line
we took to be a shortcut would translate to a straight road. He tosses the map to the floor, I pull a U-
turn, and we carefully retrace our path back to the highwaya paved two-laner that links the portcity of Ensenada, 20 miles to the west, to the border town of Tecate.
This is a booming wine country, with new tasting rooms, a couple of fine restaurants, and hotels withhigh-thread-count sheets, but it still has a wild side. And for a moment, the utter darkness gives me a
familiar feeling, like the one I get hiking in the wilderness or paddling my surfboard into a wave. The
natural world is powerful.
Only a few minutes later, we walk through the door of Corazn de Tierra. The sleek one-room
7/30/2019 LA Test Project
29/33
Shehu & Tidwell 29
restaurant is a Venice-modern oasis, with a large garden off the back deck. Diners can either sit facing
rows of beets, basil, and arugula or watching chef Diego Hernandez and his team grilling, searing, andassembling food in the open kitchen. There are no menus. Its nature that decides, says Hernandez,
29, who opened Corazn de Tierra two years ago and cooks and talks about food with equal gusto.
Here in Baja, we have a palette of flavors, like a painter.
Each dish that comes is a surprise: a pesto of beet greens served with a thick hunk of rosemary bread
thats been both baked and smoked. A salad of broccoli, radishes, and Ramonetti (a soft, local cheese
akin to a blend of brie and manchego) with an orange calendula flower and parsnip pure artfullyarranged on a slab of slate. Then its a piece of perfectly seared fresh tuna bathed in a tangy sour-milk
and mint-oil broth, served in a warm stone bowl. After that, roasted partridge with carrots, roasted
pumpkin pure, and beef sauce, paired with a spicy Vena Cava Tempranillo from the winery at Villa delValle, the B&B where Corazn de Tierra is located. Finally, its Hernandezs dessert oeuvre, each
tasting portion made from fresh oranges: ice cream, panna cotta, marmalade cake, and a cookie.
Hernandez is building on the regions farm-to-table legacy established by the valleys first and mostrenowned farm-to-table restaurant, Laja, where during a delicious, epic meal the night before, I learned
that the restaurants refrigerators and freezers often sit emptythe fresh ingredients never make it
there.
We grew up eating sashimi and ceviches, says Hernandez of his childhood in Ensenada. Now he
delivers his own sophisticated takea mix of Mexican, Mediterranean, and Asianhoned at toprestaurants in Mexico City and Tijuana. Hernandez says that visitors who come expecting only
traditional Mexican cuisine are surprised. The truth is, this is the other Mexico.
1. Why does the author use the metaphor inky black to describe the sky?
A. to convey the darkness of the sky to the reader
B. to explain the need for streetlights
C. to lament getting lostD. to show off her writing skills
2. The sentence the road keeps twisting and branching off is used to convey:A. the road looks like a tree
B. the dark and labyrinthine setting
C. the authors hatred of rural areasD. the need for a map
3. What is the main idea of the second paragraph?A. The difficulty of traveling with a spouse.
B. They are going on an adventure with no destination in mind.
C. They want to visit Californias Wineries in the Baja region.D. It is a complaint about Californias highways.
7/30/2019 LA Test Project
30/33
Shehu & Tidwell 30
4. The word booming can be understood as:
A. extraordinarily loudB. quickly prospering
C. slowly developing
D. exceptionally beautiful
5. Why does the author say the natural world is powerful?
A. because she is frightened of it
B. because despite the darkness in the wine country, she can still see its beautyC. because she loves hiking and surfboarding
D. because despite the fine restaurants and hotels, the area has a wild side
6. What does Chef Hernandez mean when he says its nature that decides?
A. He is referring to the power of the natural world.
B. He is referring to his gardens abundance.
C. He is referring to the vegetables that are in season.D. He is referring to the wild atmosphere in his restaurant.
7. Is the authors view of the restaurant that of:A. appreciation
B. dismay
C. shockD. disappointment
8. In this context what does farm-to-table mean?A. The restaurant, Laja, is the only one who does it.
B. The ingredients used at the restaurant are very fresh.
C. The majority of Mexican restaurants are expensive.
D. The food at Corazon de Tierra is quite unusual.
7/30/2019 LA Test Project
31/33
Shehu & Tidwell 31
Directions:Look at the picture below. Imagine that the two men in the picture are traveling in
California and write a brief (one paragraph) short story about their adventure.
7/30/2019 LA Test Project
32/33
Shehu & Tidwell 32
Directions: Considering your own abilities, answer the following questions by circling1, 2, 3,or4,
with one (1) being strongly disagree, two (2) being disagree, three (3) being agree, and four (4) being
strongly agree.
1. I can read and understand a magazine in English. 1 2 3 4
2. I can read and understand academic texts in English.
1 2 3 4
3. Writing fiction short stories in English is difficult for me.
1 2 3 4
4. When I write in English, I wish that my vocabulary was larger.
1 2 3 4
5. The questions about the reading passage were easy.
1 2 3 4
6. I had to reread the text many times to answer the questions about the reading passage.
1 2 3 4
7/30/2019 LA Test Project
33/33
Shehu & Tidwell 33
References
Bouvier, Nicolas. (1985). The Road to Anatolia. The Way of the World. New York Review of Books:
New York, NY.
Educational Testing Service (2012). TOEFL Writing Score Guide. Retrieved from
http://www.ets.org/toefl/pbt/scores/writing_score_guide/
Martin, C. (2003). Explore Valle de Guadalupe wine country. Sunset Magazine
http://www.sunset.com/travel/hawaii-mexico/valle-de-guadalupe-00418000082058/