Embed Size (px)
Citation preview
DOCUMENT RESUME
BD 097 640 CS 001 $81
AUTHOR Siegel, Arthur I.; And OthersTITLE Increasing and Evaluating the Readability of Air
Force Written Materials.INSTITUTION Air Force Human Resources Lab., Lowry AFR, Colo.
Technical Training Div.REPORT NO AFHRL-TR-74-28PUB DATE Aug 74NOTE 87p.; See related document CS 001 383
EDRS PRICE MF-$0.75 HC-$4.20 PLUS POSTAGEDESCRIPTORS Armed. Forces; *Readability; Reading; *Reading
Comprehension; Reading Instruction; Reading Level;*Reading Materials; Reading Research
IDENTIFIERS *Air Force
ABSTRACTThis report describes how to apply techniques that
have been used in measuring the readability/comprehensibility andreading level of textural materials. A review of availablereadability/comprehensibility measurement techniques and of previousexperimental studies into methods for developing such techniques ispresented. As a result of this review, techniques for reading leveldetermination and methods for increasing thereadability/comprehensibility of textual materials are discussed.Instructions are provided in a step-by-step fashion for determiningthe reading level of written material and for presenting subjectmatter material through methods other than prose. In addition,procedures for simplifying written material and for determining theeffectiveness of written material are described. (NR)
AIR FORCE
cdP1riza
U.B. DEPARTMENT OF Hiram.snuck:tom a welsaltiNATIONAL INSTITUTE OF
EDUCATIONTHIS DOCUMENT HAS BEEN REPRODICED EXACTLY As RECEIvEO FROMTHE PERSON OR ORGANIZATION ORIGINATING iT POINTS OF VIEW OR OPINIONSSTATED 00 NO/ NEcEssARIL v REPRESENT OF F tCiAt. NATIONAt NCtIllITE OsEDUCATION Post ?ION OR POt ICY '
AFHRL-TR-74.28
BEST COPY MUNIINCREASING AND EVALUATING THE READABILITY
OF AIR FORCE WRITTEN MATERIALS
By
Arthur I. SiegelPhilip J. Federman
Applied Psychological Services, Inc.Wayne, Pennsylvania 19087
James R. Burkett
TECHNICAL TRAINING DIVISIONLowry Air Force Base, Colorado 80230
August 1974
Approved for public release; distribution unlimited.
LABORATORY
AIR FORCE SYSTEMS COMMANDBROOKS AIR FORCE BASEJEXAS 78235
NOTICE
When US Government drawings, specifications, or other data are usedfor any purpose other than a definitely related Governmentprocurement operation, the Government thereby incurs noresponsibility nor any obligation whatsoever, and the fact that theGovernment may have formulated, furnished, or in any way suppliedthe said drawings, specifications, or other data is not to be regarded byimplication or otherwise, as in any manner licensing the holder or anyother person or corporation, or conveying any rights or permission tomanufacture, use, or sell any patented invention that may in any waybe related thereto.
This final report was submitted by Applied Psychological Services, Inc.,Wayne, Pennsylvania 19087, under contract F41609-72-C0011, project1121, with the Technical Training Division, Air Force HumanResources Laboratory (AFSC), Lowry Air Force Base. Colorado 80230.Dr. James R. Burkett, Instructional Technology Branch, was thecontract monitor.
This report has been reviewed and cleared for open publication and/orpublic release by the appropriate Office of Information (01) inaccordance with AFR 190.17 and DoDD 5230.9. There is no objectionto unlimited distribution of this report to the public at large, or byDDC to the National Technical Information Service (NTIS).
This report has been reviewed and is approved.
MARTY R. R OCKWAY, Technical DirectorTechnical Training Division
Approved for publication.
HAROLD E. FISCHER, Colonel, USAFCommander
UNCLAS$VIEDSECURITY CLASSIFICATION OF THIS PAGE (When Data Entered)
REPORT DOCUMENTATION PAGE READ INSTRUCTIONSBEFORE COMPLETING FORM
1. REPORT NUMBER
AFHRL-TR-74-282. GOVT ACCESSION NO. 3. RECIPIENT'S CATALOG NUMBER
t. TITLE (and Subtitle)
INCREASING AND EVALUATING THE READ-ABILITY OF AIR FORCE WRITTEN MATERIALS
5. TYPE OF REPORT & PERIOD COVERED
Final.
6. PERFORMING ORG. REPORT NUMBER
7. AUTHOR(*)Arthur I. SiegelPhilip J. FedermanJames R. Burkett
8. CONTRACT OR GRANT NUMBER(*)
F41609-72-C-00119. PERFORMING ORGANIZATION NAME AND ADDRESS
Applied Psychological Services, Inc.Wayne, Fennsylvania 19087
10. PROGRAM ELEMENT.PROJECT. TASKAREA 6 WORK UNIT NUMBERS62703F11210403
It. CONTROLLING OFFICE NAME AND ADDRESS
Air Force Human Resources Laboratory (AFSC)Brooks Air Force Base, Texas 78235
12. REPORT DATEAugust 1974
13. NUMBER OF PAGES
9014. MONITORING AGENCY NAME 8 ADDRESS(it different from Controlling Office) ~
Technical Training DivisionAir Force Human Resources LaboratoryLowry Air Force Base, Colorado 80230
15. SECURITY CLASS. (of this report)
Unclassified15w. DECLASSIUE FICATION. DOWNGRADING
SCHED
16. DISTRIBUTION STATEMENT (of this Report)
Approved for public release; distribution unlimited.
17. DISTRIBUTION STATEMENT (ol the abstract entered in Block 20, if different from Report)
18. SUPPLEMENTARY NOTES
19. KEY WORDS (Continue on reverse side if necessary and identify by block number)readabilitytechnical writingreading levelsreading comprehensionteytiml material
3f; ils TrRew f ko rain d my, eg g geolg tte(seagisgy iciniffnbreigesnwvb 1tt have been used in measuringletthe readability/comprehensibility and reading level of textual materials. In-structions are provided, in a step-by-step fashion, for determining the readinglevel of written material and for presenting subject matter material through meth-ods other than prose. In addition, procedures for simplifying written materialare presented. Experimental procedures, to be used in determining the effectiveJness of written material, are described. Finally, multisensory presentation ofAir Force traininl7 material is examined.
DD IFJAON"73 1473 e:OITION OF 1 NOV 65 IS OBSOLETE
SECURITY CLASSIFICATION OF THIS PAGE (When Data Enterer,
SUMMARY
Problem
This manual presents techniques and considerations for reduc-ing the reading level of Air Force training materials so as to improvethe capability of these materials to transfer information to the student.
Approach
A review of available readability/comprehensibility measure-ment techniques and previous experimental studies into methods fordeveloping such techniques is presented. As the result of this review,techniques for reading level determination and methods for increas-ing the readability/comprehensibility of textual materials are discuss-ed. The application of the techniques is demonstrated, and practicalconsiderations, including cost/effectiveness determination, for the AirForce are considered. Methods for presenting subject matter mate-rial, other than through written prose, are described. In order todetermine the effectiveness of a modified technique, performance fol-lowups are suggested. Various experimental techniques and testingprocedures for this purpose are discussed.
Results
Several techniques for measuring readability are describedin detail, and methods for their use and application are fully deline-ated. The readability techniques included are: cloze, Flesch, FOR-CAST, SMOG, and the Automated Readability Index (ARI). A quantita-tive approach for determining the cost/effectiveness of a subject mat-ter presentation technique is developed and described. Instructionsare provided for presenting material through auditory cassettes andfor preparing illustrations of various types. Techniques for simplify-ing material are explained.
Conclusions
Application of the various methods and techniques describedwill reduce training time and costs and will serve to increase the lev-el of comprehension by students who are exposed to various trainingmaterials.
TABLE OF CONTENTS
Page
I. INTRODUCTION 7
Techniques for Measuring Readability 7Early (1920s) Studies into Analytic Methods 8
Studies During the 1930s 10Studies of the 1940s and Early 1950s 12Recent Studies 14The Flesch Method 17The Cloze Procedure 18The Dale and Chall Method 20Reading Level Determination 20
II. APPLICATIONS FOR INCREASING RELIABILITY 23
Gains to be Anticipated 23Time for Comprehension 24Logical and Precise Writing 24Word and Sentence Simplification 28Repetition-Reduadancv 31
Textual Supplementation 33.
Auditory Supplementation 31Illustration Supplementation 33Flowcharts and Tables 35Graphs 39Schematic Diagrams 43
III. "NONA.JALYTIC"METHODS FOR EVALUATING TEXTUALMETHODS 4c1
Design of Materials EvaluationsSingle Group Methods
,introl Group StudiesPa.ctorial DesignConduct of Nonanalytic StudiesCharacteristics of Useful Measure of Performance
3
495050515354
ageExamples of Dependent Variables 55
Written Tests 56
Criterion and Norm Referenced Testing 56Performance Tests 57The Evaluative Milieu 58Additional Reading List 59
60REFEREN CES
APPENDIX A
A PPEN DIX B
APPENDIX C
APPENDIX D
APPENDIX F
APPENDIX F
A
How to Perform a Cloze Analysis 63
Examples of Before and After Material andAutomated Reading Index (ARI) and GradeLevel (GL) Analyses 68
How to Perform a Flesch Analysis 71
How to Perform the FORCAST Analysis 77
How to Perform the SMOG Analysis 78
Glossary of Readability Measures 81
4
LIST OF ILLUSTRATIONS
Figure pra.
1 Sample of instructions in prose 35
2 Example of flowchart to represent instructions ofExhibit I 36
3 Example of tabular presentation of instructions 38
4 Weight as a function of height for men and women betweenthe ages of 18 and 64 41
5 Annual mean income of males (25 years and older) in1956 and 1965 as a function of years of education 42
6 Percentage of United States! population completingvarious years of schooling, 1966 44
7 Example of schematic incorporating comprehensibilityprinciples 46
I. INTRODUCTION
The mismatch between the reading grade level of military per-sonnel and the readability level of the material they read has beenfound to be quite vast in some instances (i. e., four to eight gradelevels). This manual presents techniques which can be used to im-prove and measure the readability/intelligibility of Air Force read-ing materials and, accordingly, reduce the gap between the readinggrade levels of the readers and the material they read.
This manual is intended to serve as a source document for useby Air Force training and development personnel, as well as otherswho prepare reading material for the Air Force. The techniquespresented are described historically, evaluated, and demonstrated.Methods for evaluating the comprehensibility of reading material arediscussed. Finally, several practical supplements are presented.These contain information regarding auditory, pictorial, and multi-sensory presentation of information. Such considerations are impor-tant to the preparation of materials used in training some types of per-sonnel having limited reading skills.
Techniques for Measuring Readability
The readability of written material is linked to the ease of un-derstanding the material and to the speed at which the printed mattercan be read. A large number of methods are available for assessingreadability. In this manual, we group them under: (1) analytic, and(2) nonanalytic methods. Analytic methods include any method whichinvolves isolation of the various characteristics of the textual materi-al itself. Nonanalytic methods are based on some form of measure-ment of what has been learned by the reader from the text.
Most of the analytic methods are based on come formula whichconsiders the structural attributes (e.g., sentence length, word length,sentence complexity, use of familiar words) of the written material.Such formulas provide a quantitative index of the readability of the ma-terial. Under analytic methods, we also include the ncloze" technique(Appendix A).
7
Although the readability formulas are practical in many situ-ations, they also suffer from certain drawbacks that can be avoidedthrough the use of the nonanalric approaches. Specifically, whilethe analytic methods .provide an index of various attributes of writtenmaterial (such as difficulty, reading grade level, interest), they tellus little about how much is learned from reading the text. The non-analytic methods provide such information.
Early (1920s) Studies into Analytic Methods
Initial analytic efforts to measure readability objectively weresparked by the difficulty experienced by teachc 's when they attemptedto select the most appropriate texts for their classes. Science teach-ers, in particular, found that the textbooks intended for classroom usecontained a vocabulary level which was beyond that of the students. Con-sequently, objective methods were developed for determining the vocabu-lary difficulty of textbooks, so that those books with exceptionally diffi-cult vocabulary could be avoided.
One early technique was developed in 1923 by Lively and Pressey.They developed an index number to reflect vocabulary difficulty. In orderto obtain the index, a sample of 1, 000 words, evenly distributed through-out a text, was taken. Ten reference was made to a list of the 10, 000most commonly occurring words in the English language. This list wascompiled by Thorndike in 1921. Values were assigned to each word inthe sample, such that if the word appeared in the most common thousandit received an index value of 10; those words in the second thousand re-ceived a value of 9, and so on down through the last thousand-word blockin the list. Words not appearing on the list were assigned an index valueof zero. The difficulty of the vocabulary of the text under considerationwas computed by obtaining the median of the index values assigned toeach word. The resulting vocabulary difficulty indices agreed substanti-ally with the judged difficulty of a wide variety of reading selections (fromsecond grade reading to college level). However, such a readability de-termination technique possesses a number of drawbacks. These includeconsiderations of: (1) the appropriateness of the 10, 000 word referencelist, (2) the small size of the sample (samples of more than 1, 000 wordsfrom an entire book may be more appropriate), (3) prior experience withthe subject matter area (readability may be easy, despite difficult vocabu-lary, because of familiarity with the subject matter, or readability may bedifficult, despite easy vocabulary, because of a lack of familiarity with the
8
subject mattek), (4) difficulty of the concepts, (5) the reader's gen-eral intelligence, and (6) his interest in the subject matter.
Washburne and Vogel, working in the Illinois school system,generated norms for use with the Lively and Pressey method. Thesenorms were based on calculated Lively and Pressey index values for700 books that were reported as having been read and liked curing thepreceding year by at least 25 tudents. They also determined the dif-ficulty level for 150 of the 700 books. To do this, books read by lowergrade level students and by Higher grade level students were separated.Several factors were found that related to the criterion of grade level(these factors could distinguish those books read by, lower grade pupilsfrom those read by higher grade pupils). These factors correlated sig-nificantly with the difficulty of the textual material. Four of these fac-tors were set into a formula for predicting readability (Y). The fourfactors were: number of different words appearing (x1), number of oc-currences of prepositions (x2), nvinb(:r_ of words not on Thorndike'slist of 10,000 (x3), and a number of simple sentences in a 75 sentencesample (x4). A regression equation was derived for determining thedifficulty of those books reported as being read and liked. The equa-tion for predicting grade levels as measured by the Standard Para-graph Meaning Test (Y) is:
Y = .085x1 + . 101x2 + .604x3 .411x4 + 17.43
A drawback inherent in this study is that the reading material sampleincluded only those books which were read and liked. Accordingly,the sample is confounded by the evaluation of the books by the students.However, the Washburne and Vogel technique is significant historicallybecause it: (1) analyzed the effects of structural factors in the text, (2)employed an objective criterion (textual difficulty) instead of qualitativejudgment, and (3) described readability in terms of -Inhool grade levelsrather than in terms of a relative measure.
The two readability techniques, discussed previously, that of Livelyand Pressey and that of Washburn?. and Vogel, are typical of the earlymeasures which emphasized readability measuremeat as a function ofvocabulary uniqueness with reference to Thorndike's word list, andwhich used relatively crude criteria of the difficulty of the text.
9
Studies During the 1930s
The readability studies undertaken during the 1930s were largelycharacterized by: (1) emphasis on evaluation of a greater number ofreadability factors, (2) reduced emphasis on word frequency lists (e.g.,Thorndikets 10, 000 most commonly occurring words in the English Ian-uage), and (3) a trend to nonanalytic methods, i. c, , concern over moreobjective readability criteria such as comprehension test scores.
For example, Ojemann studied the use of comprehension testscores as the criterion of readability. He administered 16 passages,each of approximately 500 words, to adults and then tested their com-prehension of each passage. The passages were arranged in order ofdifficulty and their contents were analyzed for eight sentence factors,six vocabulary factors, and three qualitativ e factors. Instead of a formu-la for determining readability, Ojemann presented his 16 listed selections,along with the values he determined for the quantitative aspects (i.e., thevocabulary and sentence factors), and their respective difficulty levels.This type of presentation was intended to allow other analysts to evaluatethe qualitative factors in new passages, in addition to comparing the newpassages with his tested passages on the quantitative factors. Whilesuch a technique is cumbersome and time consuming, Ojemann's studyis noteworthy because it was the first to: employ adult subjects, use com-prehension test s .ores as a criterion of readability, and demonstrate thevalue of qualitative, rionstatistical factors in determining readability.
However, this period was not devoid of some continued emphasison the devolopment of formulas. For example, Dale and Tyler askedadults of low reading ability (third to fifth grade level) t.) take a compre-hension test after reading health education articles that z..p,leared in news-papers, magazines, and books. Adequate test scores work not obtained.Dale and Tyler rewrote the passages, making them much simpler to read.They found that using very basic vocabulary, an informal style character-ized Iv a conversational manner and anecdotes, and never digressingfrom the main topic increased the readability of the articles. Their anal-ysis, based on the readministration of the rewritten passages, resultedin several factors correlating significantly comprehension. Threeof the factors were included in a regression equation for predicting com-prehension: the number of technical words (x1), the number of nontech-nical words not known to 90 per cent of sixth grade pupils, as determinedin a study performed by Dale (x2), i11111 the number of indeterminateclauses (x3). The equation used to predict comprehension (Y) is:
LO
Y = -9.4x1
- 0'4x2
+ 2' 2x3
+ 114.4
One study which fully exemplifies the trend in formula exten-sion during this period is that of Gray and Leary. They listed over80 factors related to readability. Through various processes of elim-ination, Gray and Leary were able to develop regression equationsbased on only a few factors, The most frequently used of their formu-las was one consisting of five factors. This formula is
where:
Y = .01029x2 + .009012x5- .02094x
6- .03313x
7- .01485x8 + 3.774
Y = predicted average comprehension score, along ascale of + 4 to -4
x2= average number of difficult words (words not ap-pearing on Dale's list of 769 easy words)
x5= average number first, second, and third personpronouns
x6= average sentence length in words
x7= average percentage of different words
x8= average number of prepositional phrases
Since this formula is most applicable to low ability readers, it lacks broadutility.
li
Studies of the 1940s and Early 1950s
The trend in readability research during the 1940-1950 decadetended toward more efficient and more easily applied formulas. Lorgemodified the formulas developed by Gray and Leary and obtained highercorrelations with his criterion Qf reading passages than did Gray andLeary. Lorge attributed this increased correlation to his use of a great-er and more adequately standardized set of criterion materials. Lorge'scriterion materials were the Standard Test Lessons in Reading of McCalland Crabbs. The Standard Test Lessons in Reading consist of 376 read-ing passages containing norms related to grade levels. The formulaLorge used included two of Gray and Leary's factors, x6 (average sen-tence length in words) acid x8 (number of prepositional phrases per hun-dred words), in addition to the factor x9 (ratio of hard words to total num-ber of words, hard words being those that do not appear on Dale's list of769 easy words). The final readability formula was:
where:
C50= . 10x2 + . 06x6 + 10x
8+ 1.99
C50 = reading grade level of individuals answering50 per cent of the McCall and Crabbs questionscorrectly
Flesch was dissatisfied with the then current readability formulasbecause of the poor relationship found between the rankings of magazinearticles (as determined by the formulas) and the rankings of the educa-tional levels of the magazines' readers. He thought that the misrankingwas due to an emphasis on vocabulary and insufficient stress on otherfactors. He suggested three factors that should highly influence readabil-ity for adults: (1) sentence length, (2) abstracts, as determined throughthe number of affixes (i.e., prefixes and suffixes), and (3) number of per-sonal reference words (i. e., pronouns., names, and words relating topeople such as aunt, baby, etc.) Flesch's initial formula underwent arevision several years after it was first developed. This revised formu-lation which yields a reading ease and a human interest index has becomewidely used for readability analytic purposes. This revised formulationfor measuring readability will be discussed in another section of this chap-ter.
12
Dale and Chall employed Flesch's original formula to evaluatethe readability (for the average adult) of educational material publishedby the National Tuberculosis Association. They sought to develop theirown measure of readability becuuse they were dissatisfied with two as-pects of the Plesch formula. To them, counting of affixes (prefixesand suffixes) was not completely manageable. Dale and Chall found thatdifferent raters, counting affixes in the same passage, came up with dif-ferent numbers. However, they did agree with Flesch that a measureof abstractness was of value and had a place in a readability formula.Dale and Chall's second objection to the Flesch formula related to theuse of personal words. They felt that Flesch's interest formula wasoverly simplified and that the definition of personal words (for purposesof readability) should be more restrictive. Dale and Chall's formulafor measuring readability will also be discussed more fully in a subse-quent section of this chapter, since it is also a popular formula formeasuring readability today.
Another readability formula produced in this era was that ofPowers, Sumner, and Kearl. Using the revised passages of McCalland Crabb's Standard Test Lessons in Read (which was updated sothat 60 passages dealt with then modern topics), Powers, Sumner, andKearl recomputed Flesch's formulas, as well as others that had beendeveloped earlier. They intended to produce formulas which would ac-count for changes in reading abilities over the period between the twotest editions of the Standard Test Lessons in Reading (some 24 years)and to facilitate comparisons of the formulas they recomputed. The re-searchers claimed that the revised formulas provided estimates moreconsistent with one another than did the original Flesch and Dale-Challformulas.
McElroy's entry into the field of readability resulted in a meas-ure referred to as the "fog count. " The fog count is also related to theFlesch formulas in that it too counts syllables. Weights are assigned towords such that one and two syllable words receive a value of 1; a valueof 3 is given to each remaining word. The sum of these weights equalsthe fog count. The reading grade level was determined from the fogcount in the following manner: divide by 2 for sums of 20 and over; sub-tract 2 from sums under 20, and then divide by 2.
13
Recent Studies
The trends continued in the 1940-early 1950 period. However,there has recently been some emphasis on linguistic theoretic ap-proaches to comprehensibility. Spache, in 1953, developed a formu-la to predict p Lmary grade level (grades 1-3) of textual material onthe basis of sentence length and percentage of words not appearing onthe Dale list of 769 common words. The formula is:
where:
Grade level = 141x1
+ 086x2
+ 839
x1
= average sentence length in words
x2 = percentage of words not appearing onDale's list of 769 common Words
Wheeler and Smith, in 1954, developed a formula which equatedgrade level to 10 times the product of the mean length of units (sentences)in words and the percentage of multisyllable words. The value deter-mined by the multiplication of these factors was located in a table andgrade levels 1 through 4 could then be read from the table.
Tribe, in 1956, used the revised form of the McCall and CrabbsStandard Test Lessons in Reading as the criterion of readability. Thegrade level score of pupils, in grades 2 through 8, who could answer 50per cent of the reading test questions correctly is:
where:
. 0719x1
+ 1043x52.9347
x1
= average sentence length in words
x5 = percentage (times 100) of words notappearing on a standard word list
114
In 1954 and 1958, Flesch introduced three alternate readabilityindices These have not become highly popular, possibly because allof the indices are based on measures of word counts. The countedvalues are converted to arbitrary scales by referring to conversiontables. Flesch's "r" score is an index of realism, based on the num-ber of references to specific human beings, their attributes or posses-sions, locations, objects numbered or named, dates, times, and col-ors. His "e" score is an index of energy, based on indications of voicecommunications, such as inflection. Finally, the "formality-popularity"scale is based on the total number of capitalized, underlined or itali-cized words, numbers (not spelled out), punctuation marks, symbols
$), beginnings of paragraphs, and endings of paragraphs.
Coleman, in 1965, developed four formulas using from one tofour variables to predict readability. The readability criterion wasthe mean "cloze" score on the passage achieved by a sample of collegestudents. The "cloze" score is the percentage of deleted words in apassage filled in correctly by a reader. The " close" technique will bediscussed more fully below. The simplest of Coleman's formulas,which correlates very highly with the involved formulas, is:
Predicted close score = 1.29 (percentage of onesyllable words) - 38.45
McLaughlin, ln 1969, found that the number of syllables in 100words is equal to three times the number of words of over two syllables,plus 112. A very close approximation to his formula, referred to as theSMOG grade, is:
where:
SMOG Grade = 3 + square root of the polysyllablecount in 30 sentences
polysyllable = number of words with threc.or moresyllables
15
A formula for measuring readability was developed at HUMRROin 1972. The measure is referred to as FORCAST and was developedfOr measuring the readability of Army technical literature. The devel-opers of this formula considered existing formulas inappropriate forArmy technical material because extant formulas were most often basedon school students and on general text material. Cloze scores were usedas the criterion of readability. It was noted that cloze consistently yield-ed very high correlations with multiple choice tests and other more sub-jective measures of comprehension and difficulty. Prior research indi-cated that if a cloze score of 35 per cent was achieved by a readpr for agiven passage, then he might be expected to achieve a score of 70 percent on a multiple-choice test of the same reading passage. Hence,cloze scores were demonstrated to be good indicators of comprehensionand reading achievement level. The FORCAST formula for predictingreading grade level (y) is:
Y = 20 - (number of one syllable words oc-curring in a 150 word sample/10)
Smith and Senter's readability analytic technique was unique be-cause it was based on data which could be collected from mechanicalcounters that are installed on an IBM Selectric typewriter. The tech-nique allows measurement of readability at rough draft typing speed.The mechanical counters record the number of key strokes, spaces,and sentences. The mean number of words per sentence (number ofspaces divided by the number of sentences, w/s) and the mean lengthof words (number of strokes divided by the number of spaces, s/w) maybe computed. The regression equation which predicts grade level (GL)from the above ratios and which was based on graded school texts is:
GL = 0.50 (w/s) + 4.71 (s/w) - 21.43
This formula may be simplified to:
GL w/s + 9(s/w)
As such, it is referred to as the Automated Readability Index (ARI). Inaddition to the speed of data collection that it affords, the advantages ofthe All technique are: (1) its objectivity, (2) its reliability, and (3) theease with which it may be incorporated into modern computerized type-setting machinery.
16
The Flesch Method
The most widely employed Flesch formulation rests on two in-dices--reading ease and human interest. The reading ease (RE) for-mula is:
or:
where:
where:
RE = 206. 835 - . 846 WL - 1. 015 SL
RE = 206. 835 - 1. 015(SL) + . 846(WL)]
WL = word length measured as syllables per100 words
SL = average sentence length in words for agiven sample
The human interest (HI) index is determined from the formula: .
HI = 3.635 PW + .314 PS
PW = average percentage of personal words
PS = percentage of sentences addressed to thereader (including exclamations and gram-matically incomplete sentences whose mean-ing is interpreted from the context of thepassages)
This index tests the conversational style of the passage. The conjecturebehind the human interest factor is that prose written in a conversationalstyle will be more readable than that which is nottconversationally oriented.
These Flesch formulas provide index v:.ilues which range from 0to 100, and have become the most widely applied of any which have beendeveloped. The wide use of the formulas is due, in part, to their easeof application.
17
A simplification of the Flesch reading ease formula was intro-duced by Farr, Jenkins, and Patterson in 1951. They suggested thatFlesch's formula use the number of one syllable words, thereby allowinga more rapid test, instead of counting all syllables per 100 words. Suchan approach was found to correlate .95 with the scores produced by theFlesch formula. Consequently, Farr, Jenkins, and Patterson suggestedthat their reading ease formula be substituted for Flesch's. Their formu-la is:
RE = 1.599 (number of one syllable words)- 1.015 (mean sentence length in words) - 31.517
The Cloze Procedure
The doze procedure, introduced by Taylor in 1953, was demon-strated to rank standard reading passages in the same order as did theFlesch and the bale -Chall formulas. In the doze procedure, readersare presented with samples of text, from Ouch some words are deletedand replaced by blank spaces. The readers are requested to fill in theblank spaces wi'ir the words they think were deleted. To the extent thatthe author uses the words that the reader expects and understands, thereader will fill in the correct words. The technique assumes that read-ability is a direct function of the number of omitted words which thereader is able to fill in.
Taylor indicated that the ordering of doze scores is maintainedregardless of the system employed in word deletion. He used four dif-ferent deletion systems on three standard passages, each of a differentdifficulty level, and found that the rankings were the same despite thedeletion system employed. The four deletion systems involved deletingevery fifth word, every seventh word, every tenth word, and 10 per centof the words at random. Others have reported that deleting 20 per cent ofa passage will yield sensitive measures.
The doze method is free from many of the disadvantages of thetraditional readability measures. It can be applied more appropriately tohighly technical and unusual materials. Very technical material might berated as difficult by the lelesch and Dale-Chall formulas, but not by the
18
cloze technique, if the subjects reading the passage were trained in thesubject matter area. Conversely, the readability of the writings ofauthors such as Gertrude Stein, who write in short sentences with rela-tively simple vocabulary, but whose style is such that the material isdifficult might not be accurately reflected by the Flesch and Dale-Challtechniques. The cloze test might be expected to reflect accurately thedifficulty of such reading passages.
Taylor reported correlations of .70 and .80 between cloze scoresand comprehension scores received by Air Force trainees on Air Forcetechnical material. A study performed by Bormuth indicated that a 30per cent cloze score was associated with a 75 per cent comprehension testscore; 50 per cent cloze score was associated with 90 per cent compre-hension score.
In a study performed at Applied Psychological Services; a seriesof norms was developed for use in evaluating the readability of submarinesonar manuals through the cloze technique. In order to obtain normativedata, the researchers administered a representative sample of submarinesonar manuals, in accordance with the cloze procedure, to a representa-tive sample of sonar technicians. The cloze scores achieved by the sonartechnicians constitute the normative standard against which scores achiev-ed by similar groups on other manuals may be compared. Thus, thenorms allow an answer to the question, "Relative to the group of sonartechnicians in the normative sample, what percentage of technicians canbe anticipated to achieve a given comprehensibility score on this new man-ual'? "
The inherent advantages of the cloze procedure are: (1) scoringease, (2) scoring reliability, (3) ease of application to nonstandard materi-al, and (4) accounting for the reader's interest in and prior knowledge of thecontent. The disadvantages of the procedure are: (1) cloze is a measureof readability, not a predictor of readability, (2) a sizeable sample ofsubjects is required, and (3) it may not reflect all types of comprehen-sion.
19
The Dale and Chall Method
The readability formula developed by Dale and Chall was basedon the average sentence length, and the "Dale score. " The "Dale score"is the relative number of words in the passage that do not appear on aparticular list of words, a list of 3,000 words known to 80 per cent ofa sample of fourth graders. The Dale and Chall readability formula is:
where:
Xc50 = 1579 (Dale score) + .0496 (sentence length)+ 3.5365
Xc50 = the reading grade level of an individual ableto answer 50 per cent of the comprehensiontest questions correctly
Heading Level Determination
Several of the readability formulas discussed above predict read-ability in terms of reading grade level (RGL). RGL may be determinedthrough several different approaches. The formulas above predict RGLas measured by standard tests of reading ability, for example, McCalland Crabbs' Standard Test Lessons in Reading. These tests may be usedfor groups or individuals. An annotated list of these tests is available inThe Sixth Mental Measurements Yearbook (Buros, 1965).
Although the standardized tests of reading comprehension havedemonstrated validity and reliability, there are more efficient techniquesavailable for use in the military situation. Madden and Tupes provided amethod for estimating RGL from Air Force aptitude test scores. Theyfound that the general aptitude index (AI) of the Airman Qualifying Exam(AQE) correlated .70 with RGL. This high correlation was largely attri-buted to the reading vocabulary subtest in the AQE. In predicting theRGL of an individual Airman, one of the following three formulas may beemployed, depending upon the Airman's career field:
20
Administrative: RGL = .0437 (Gen AI) + .0501 (Ad AI) + 5.0730
Mechanical: RGL = .0991 (Gen AI) + .0085 (Mech Al) + 5.0459
Electronic: RGL = .0743 (Gen AI) + .0222 (El AI) + 4.6088
Another regression equation for predicting an individual's RGL,in the military, was developed by Caylor, Sticht, Pox, and Ford (1972).Their formula uses the individual's Armed Forces Qualification Test(AFQT) score. The formula is:
RGL = .75 (AFQT score) 1- 5.52
21
Additional Reading_ List
Chall, J. S. Readability: An appraisal of research and application.Columbus: Bureau of Educational Research, Ohio State Uni-versity, 1958.
Mare, G. R. The measurement of readability. Ames, IowaState University Press, 1963.
Osgood, C., Suci, G., & Tannenbaum, P. The measurement of mean-ing.: Urbana: University of Illinois Press, 1957.
Pool, I., De So la Trends in content analysis. Urbana: University ofIllinois Press, 1959.
Szalay, T. G. Validation of the Coleman readability formulas. Psycho-logical Reports, 1965, 17, 965-366.
Williams, A. R. , Jr., Siegel, A. I. ,&Burkett, J. R. Readability of textu-al material- -a survey of the literature. Lowry Air Force Base,Colorado, AFHRL-TR-74-29, in preparation.
Yoakum, G. A. Basal reading instruction. New York: McGraw-Hill,1955.
22
II.. APPLICATIONS FOR INCREASING READABILITY
Communication through writing, in a learning situation, is dif-ficult because of the many complexities involved. Often the languagethat must be used is technical, students' reading skills are low, andthe subject matter material is difficult to present in an interestingfashion. Despite the difficulties inherent in transferring informationthrough the written word, textual presentation is a communication tech-nique that trainers must use. Written material is considered maximal-ly effective when a student can gain a clear understanding of the contentafter a single reading. Accordingly, it is imperative for the writer touse language that will be meaningful to the studentlanguage that iseasily understood.
This section suggests methods for increasing the readability ofmaterial prepared for military training purposes. Because multimodalpresentation of the material to be learned has been recently suggestedin a number of contexts, this concept will also be discussed. Althoughblanket statements about the benefits that can be gained from a multi-modal presentation of material cannot be made for all tykes of materialor for all types of learners, consideration should be given to multimodalpresentations when: (1) the material lends itself to presentation in morethan one mode, (2) the increased benefits outweigh the increased costs(section IV), and (3) readirig grade levels of personnel to be trained arelow.
Gains to be Anticipated
The gains that can accrue from an effective presentation of train-ing material are:
less time tc achieve the required skills andknowledges
less effort required to learn the material
These factors translate directly to increased effectivenessthrough lower costs both in the training center (through the savings intime to train the student) and in the field (through the diminished amountof time required for on-the-job training).
23
Time for Comprehension
There are certain types of material that are more easily graspedand retained when they are presented in writing, Some types of =feri-a! require time to be absorbed, The time needed to absorb the materialcan be taken by the student if he reads the material, but he may not havethe time he needs if he is unable to control the rate of presentation. li'orexample, the student must pace himself according to the speed set by theinstructor in the lecture, whereas he can move at his own pace when read-ing written material.
Logical and Precise Writing
Logical and precise writing is the key to improving readability.Quite obviously, readability will be improved if logic and precision areconsidered in the context of the structural factors suggested by the vari-ous readability formulas,
When preparing readable material, it is important to know thereading ability of those for whom the material is being prepared. Cer-tainly, a writer of textual material would not write at an eigth gradereading level if he were preparing course material for college seniors.On the other hand, it would be folly to write a discourse at the sixteenthgrade reading level when it is to be read by enlisted Airmen taking a tech-nical training course. Therefore, any "rules" for preparing written ma-terial must be tempered by the background and reading grade levels of thereaders.
Flesch suggests:
1, preparing the material in a personalized mannerin-crease the number of personal words (e.g., I, you, he,him, she, her; words of masculine and femioive gendersuch as John, Miss, mother, brother; and group wordssuch as people) and increase the number of personal sen-tences (e.g., spoken sentences in which quotation marksare used; sentences addressed to the reader as in "dothis" and "don't do that;" exclamations; and incompletesentences)
24
2. discussing all points presented
3. shortening the length of words and sentences (iflong technical words have to be used, make surethe meaning will be clear to the reader even if adefinition must be included)
4. using punctuation as an aid to the reader
5. discussing points in chronological order or in orderof increasing difficulty
6. avoiding excessive wordiness
7. providing anecdotes or illustrative examples whereverpossible
8. saving the most important points for last; many learn-ers remember best what is presented last
Learning from textual material will also increase when the follow-ing are included:
1, an introduction outlining the points that follow
2. a summary at the end
3. major and minor headings to guide the reader
4. transitional paragraphs which lead into subsequenttopics
5. statements telling the reader what the important pointsaretelling him what to remember
25
In addition to those aspects outlined above, which all serve toimprove clarity, the writer should present one point at a time and stickto that point. Deviations from the main theme must be avoided and im-portant points within the theme must be emphasized. Emphasis can beaccomplished in several different ways; e.g., underlining, italics, repe-tition, a statement such as "this is an important point, " and devotingmore space to the points which are to receive greater emphasis.
A conversational writing style also enhances readability becauseone tends to present his ideas simply when speaking, but may becomea little more complex when committing his thoughts on paper. As an ex-ample, an instructor might say to his class "Tomorrow you will have a10 minute multiple choice test on Chapter 2. " The message is clear andconcise. If a literary written instruction was given, it might be worded,"In order to determine your level of achievement on the subject of bio-logical warfare, it is recommended that you be examined at the comple-tion of the subject. Therefore, a paper and pencil test will be admin-istered. " The consequences of such an instruction could be sizeableto the instructor (if the students require clarification from him person-ally) or to the students (if they appeared at the next session of the classunprepared for the test). Although written instructions may be betterthan oral 'instructions because they are available for additional refer-ence, they should be direct, concise, and follow the style of the spokenversion in order to be effective.
Other suggestions which will assist the writer in prep4ring moreeffective written materials are:
1. Do not begin a sentence with the indefinite pronoun"this. " "This" refers to something in the precedingsentence or paragi aph, but the specific antecedentmay not be clear to tie reader. Since the goal is toincrease comprehensibility, repeat a word or conceptrather than chance being unclear to the reader.
2. Avoid using colloquial expression which may be mean-ingful to certain groups of people but not to others; e. g. ,"groovy, " instead of "very acceptable. "
26
3. Use parentheses to set off explanations. Do notenclose one set of parentheses by another; usebrackets to enclose parentheses; e. g. , 10[(a+b)(c+d)].
4. Use abbreviations only after a full explanationof the abbreviation has been provided, except wherethe abbreviation is very widely known; e.g. ,12 a. m. An abbreviation is often defined in paren-theses after the first use. The abbreviation may beused alone thereafter.
5. Use only those symbols that are very common; other-wise spell out the symbol. Examples of some com-mon symbols are: $, #, =, etc. If less popularsymbols are needed, define them for the reader.
6. Do not omit an article for the sake of brevity; e.g.,"The students alid the instructors were invited to theparty, " not "The students and instructors were in-vited to the party. "
7. Do not omit the word that; e. g. , "Lt. 'Smith felt thathe should speak to Pvt. Jones, " not "Lt. Smith felthe should speak to Pvt. Jones. "
8. Do not omit a verb; e. g. , "He finished the course onJune 6 and was tested on June 8, " not "He finished thecourse on June 6 and tested on June 8. "
9. Place modifiers as close to the word they modify aspossible; e. g. , "He studied diligently for three hours, "not "He studied for three hours diligently. "
10. Avoid splitting infinitives unless the sentence would beawkward; e.g., "She wanted to return promptly afterChristmas, " not "She wanted to promptly return afterChristmas. "
11. Avoid using the word and to connect two unrelatedideas; either rephrase the sentence using other con-nectives or write two sentences. For example, "Sincethe instructor taught the material, the students wereable to answer the question," or, "The instructor taughtthe material. Thus, the students were able to answerthe question;" but not "The instructor taught the materi-al and the students were able to answer the question. "
12. Use the active voice rather than the passive voice. Forexample, "Use a screwdriver to help you raise the lid;"not "A screwdriver should be used to help raise thelid. "
Word and Sentence Simplification
The selection of words is crucial to preparing readable copy. A
general principle to follow is to select words that have a high frequencyof occurrence and are thereby familiar to the readers. In addition to se-lecting familiar words, the words used should be:
a. shortb. nontechnical (where possible)c. American sounding rather than foreign soundingd. concrete rather than abstract
The next logical consideration is sentence structure. Here, thebasic factor is the length of the sentences. The characteristics of thereader will be the largest single factor in determining how long or shortsentences should be. Generally, shorter sentences are more readable.However, other sentence structure characteristics should be consideredin making written material more readable. The characteristics to beavoided are:
1. prepositional phrases (e. g. , The two roads ofwhich I speak are...)
2. compound sentences (e.g. , He campaigned fornomination, ran for election, and won the vote. )
3. complex sentences (e.g., I would have passedthe test if I studied more. )
28
Since the more complicated sentences are usually longer thansimple sentences, it can be demonstrated that sentence length is direct-ly related to sentence structure. Considering that sentence length iseasier to analyze than sentence structure, length is the more frequent-ly used factor in readability measurement.
The following examples represent excerpts from Air Forcemanuals, as they appeared in their original form and then in a modifiedversion.. The modified versions demonstrate the effects word and sen-tence simplification.
Original: The Air Force uses two basic methodsprepost and post-postfor issuing supplies. A prepost issue is processedthrough the computer before the property is actually se-lected from storage. A post-post issue made before theissue request is processed through or into the computer.A simple way to determine which method is being usedis that prepost documents are machine printed and post-post documents are normally handscribed.
Modified: The Air Force uses two methods for issuing supplies.These are prepost and post-post. A prepost issue isprocessed by the computer before the supplies are takenfrom storage. A post-post issue is processed by thecomputer after the supplies are taken from storage. Youcan easily tell which method is being used because pre-post order documents are printed by machine. Post-post order documents are usually written by hand.
Original: Post-post issues from the base service store are ac-complished in the same manner as normal warehousepost-post issues. The AF Form 2005 and the DD Form1348-1 are handscribed. Unlike the singlt. item pre-post issue, the property is not removed from stock un-til the document arrives at the BSS. If the entire quan-tity requested is not available, the quantity actually is-sued is entered in the quantity block of the DD Form1348-1. The property and documents are forwarded tothe pickup and delivery section for delivery and docu-ment distribution.
29
Modified: Post-post issues from the base service store aremade the same way as regular warehouse post-postissues. The AF Form 2005 and the DD Form 1348-1are filled in. The procedure is different than thatused for a single item prepost issue. Supplies arenot removed from stock until the issue form arrivesat the BSS. If you don't have enough of an item instock to fill an order completely, write the amountthat you can supply in the quantity block of DD Form1348-1. The items and forms are sent to the pickupand delivery section. These people will deliver theitems and distribute the forms.
Original: Copy 1 of the AF Form 1991 for a location additionis forwarded to the warehouse remote operator forinput, or to the punch card accounting machine (PCAM)unit for keypunching and input via the card reader.Copy 2 is held in a suspense file, pending receipt ofthe output notice or its appearance on the daily docu-ment register. If the input is by a remote device, anoutput notice is produced on the remote. If the inputis made by the card reader, no printed notice is pro-duced.
Modified: If the new warehouse location input to the computerwill be through remote, send copy 1 of the AF Form1991 to the warehouse remote operator. An outputnotice will be produced on the remote. If the inputwill be through a card reader, send the form to thepunch card accounting machine (PCAM) operator. Hewill keypunch and enter the information. No outputnotice will be produced in this case. Keep copy 2 ofthe AF Form 1991 in a suspense file. When you re-ceive the output notice, remove it from the file. Youalso remove copy 2 from the suspense file if that lo-cation input appears on the daily document register.
3U
Repetition-Redundancy
Repetition is an important concept in readability-comprehensi-bility. Repetition reduces uncertainty in the reader by helping him toremember important points.
To increase redundancy, it is not necessary to repeat specificpoints verbatim or even to repeat the same point using different words.In some cases this could lead to monotony; those who wish to reread aspecific section could do so. Increasing redundancy can be accom-plished by writing about a specific issue from another point of view. Re-dundancy can also be increased by using words in a sequence; e.g., heis loud, dumb, and vulgar.
Textual SupplementationReading is a complex process involving the reader's education,
reading techniques, intellect, and emotional status, as well as the char-acteristics of the textual material itself. Other factors affecting the abil-ity of the reader to read quickly and understand textual material are thereader's visual acuity, his physical condition, his educational background,cultural factors, distractions present, the reader's ability to concePtual-ize language, to think symbolically, and to organize the material mean-ingfully. Diagnosis of a reading disability requires specially trained per-sons. A student's inability to master the written word may be overcome,to some degree, by supplementing the textual material with auditory tapes,pictures, graphs, and the like.
Auditory Supplementation
Although listening has an advantage over reading because it con-tains intonational nuances, individual differences among listeners existinsofar as rate of listening comprehension is concerned. Listening totape recordings is somewhat like listening to a lecture, except for theformality of the classroom environment. However, the student can stopthe tape at any time to think over a point. And, he can return the tapeat an earlier point in case he has missed the point. Most research in-dicates that, no student will do worse with taped material than with writtentexts and that some students will do better with taped materials as com-pared to written texts.
31
In preparing the material for a recording, the problem of makingthe material comprehensible to the learner is as important as for writ-ten material. The subject matter must be efficiently organized if thematerial is to be meaningful to the listener. Learning from tape record-ings will be enhanced by the following:
1. limit the amount of information on any one tape
2. repeat key points at strategic places in the recording
3. keep the material at the level of comprehension ofthe students
4. permit discussions and questions before and afterthe student listens to the recording
5. prepare the student for the recording by telling himof the subject matter to be covered, its purposes, andhow he will be able to use the information
6. save the most important points for presentation nearthe end
7. compliment the students by leaving them with a noteof encouragement
The speaker on a tape recording should be skilled in the technique'sof verbal delivery; i.e. , someone trained in the area. Since the speakeron a recording cannot take advantage of his body and other audio-visualaids, he will have to concentrate on different types of verbal techniquesto project his ideas and purposes. Some of the key points in making agood delivery are:
1. present the nterial in a lively and animated fashion
2. convey a sense of enthusiasm and belief in the im-portance of the subject matter. The material mustbe learned and understoa by the speaker in orderto deliver the talk with conviction.
3. use a pleasant tone and vary the pitch and force. Therate should be carefully controlled as should the degreeof loudness.
32
4. use pitch, tone, and rate variation for emphasis
5. use inflections--a downward inflection at the endof a complete thought (as is typically used at theend of sentences); an upward inflection at the endof a question (often associated with surprise, doubt,uncertainty). Do not use rising inflections at theend of complete thought units, as it presents a sing-song quality
6. use short, crisp sentences
7. use a voice that is free from any unusual quality(e.g., harshness, nasality, breathiness, hoarse-ness, throatiness)
Illustration Supplementation
There are other alternative techniques available for presentinginformation that one might normally present in prose. These alterna-tive approaches may result in improved comprehension and/or perform-ance. Although an instructor may use what he considers to be simpleand concise language (and the writer or reading material does the same)communication might be less than complete without visual presentation.Visual presentations show the learner exactly what the writer. is talkingabout or what is being referred to in the written material. Illustrationsare held to be essentially demonstrations but in a remote fashion. Theysave time because fewer words have to be written to describe a situationand, accordingly, fewer words have to be read.
The following should be taken into consideration in preparing il-lustrations:
1. avoid unnecessary artwork since it may seiveonly to distract
2. colors, if used, should be in sharp contrast witheach other
33
3. if temporal sequencing is involved (e. g. , as in astep-by-step procedure) and only that part of theequipment is to be visible which applies to a par-ticular step, then overlays on transparanciesshould be used
4. limit the number of callouts or identifiers
5. connect identifiers to the correct point in theillustration with a leader line
6. limit the callouts to those referenced to in thetext
7. use directional arrows to help the reader orienthimself with respect to the illustration
8. all text appearing on the illustration should be inan area which will not affect the readability of theillustration, its callouts, or identifiers
9. simplify as much as possible (an overcrowded il-lustration is confusing)
10. use size and placement for emphasis
11. discuss the important aspects of each illustrationin the text and state how to apply the information
12. keep the illustration interesting--from the reader'sviewpoint
13. choose a simple, eas3 to read letter style
Preparing effective illustrations, a field in itself, requires pro-fessionals and experts. Commercial and industrial artists, engineers,and draftsmen are often consulted in these matters. However, sincethese professionals may not be subject matter experts, effective illus-trations will represent the merging of instructors, who know the subjectmatter and message to be delivered by the illustration, with profession-als who have the artistic talent.
34
Flowcharts and Tables
Flowcharts and schematics also represent forms of textual sup-plementation. The flowchart is one form of illustration which is help-ful when several complex, serialized points have to be made or whenthe subject matter pertains to interactive sequences of acts that shouldbe performed under different conditions. The flowchart is more easilyunderstood, more accurately used, and more quickly interpreted thanits prose counterpart. Figure 1 presents a set of instructions in proseform.
Instructions in Prose
Place switch S-1 on the tester in the "on" position.Probe connections are required from scope input A to TP3on the LLM. Adjust the OUTPUT LEVEL on the LLM sothat the amplitude is 5 v. P/ P. If there is no output at TP3,check for a signal at V101, pin 1. If there is a signal atV101, pin 1, then check for a signal at V102, pin 2. Ifthere is no signal at V102, pin 2, there is trouble in thefilter network. There is trouble in either V102, T101, orassociated circuitry if there is a signal at V102, pin 2. Ifthere is no signal at V101, pin 1, check for a signal atV101, pin 2. If there is a signal at pin 2, the trouble is inthe ..., but if there is no signal at pin 2, then check V103,pin 3, for a signal
Figure 1. Sample of instructions in prose.
These same instructions are also presented as Figure 2 in theform of a flowchart. It, can be seen, from Figure 2, that the flowchartwill help the reader who is searching for a specific piece of information.It also provides an integrated picture so that relationships and orderingsamong the parts can be readily perceived.
35
STEP I
SWITCHS-I ONTESTER
BEST COPY AVAILABLE
PROBECONNECTIONS
WAVEFORM
.
FROMSCOPE
TOLLM
CH-I INPUT TP3
NOTE: ADJUST OUTPUTLEVEL CONTROL ON
LLM SO THAT AMPLITUDEIS 5V. P/P
iftV111...ib
a= 5Vb=513:t 26/As
SUGGESTED TROUBLE ISOLATION PROCEDURE
IF NO OUTPUT AT TP3
1
(CHECK FOR SIGNAL y....AT V101 PIN 1
CHECK FOR NSIGNAL ATV101 PIN 2 /
los --- el- 6160 ps .
ii
CHECK V103PIN 3
.N"-.61601rs =01
CHECK FORSIGNAL AT 1----V102 PIN 2 /
/ / TROUBLE
/TROUBLE \IN EITHER
TROUBLE IN V102, T 101 , OR
IN FILTER ASSOCIATED
NETWORK CIRCUITRYM 11 41110
CHECK V103PIN 2
1
/ TROUBLIN
Mak 11.0, ale a
rn:51.--44 its 1*----
IMAIMIIIMM I-26V7
CHECK TIMINGONE INPUT FROMTESTER ATJUNCTION (FC5) )
TROUBLETROUBLEIN
Figure 2. Example of flowchart to represent instructions of Figure 1.36
Another alternate to prose is tabular presentation. Figure 3 isa two-dimensional table, presenting the same information as in Figures1 and 2. While complex information is difficult to present, regardlessof the format, it is evident that for the information presented in Figures1, 2, and 3, the prose version is the least comprehensible.
The advantages of presenting quantitative data in a tabular for-mat are that tables are: (1) economical, (2) systematic, (3) concise, and4) comparisons are facilitated within and among tables. The following
will help in preparing comprehensible tables:
I. title the table with a brief descriptive statementof the contents
2. legend each column
3. legend each row
4. separate classifications (i. e., between rows andbetween columns) by ruling in lines or leavingwide spaces
5. identify the units that are used for the numbersin the column (e. g., dollars in thousands, etc. )
6. use footnotes to point out any unusual circum-stances or information that cannot be placed inthe body of the table
37
LI.
,N1
Tes
tP.
roce
durf
-
1. T
este
r S-
1 to
ON
2. P
robe
con
nect
ions
rro
m s
cope
inpu
t A to
LI,
M, T
1'3
3. I
f no
out
put a
t TP3
, the
n
Step
Sign
al-N
o Si
gnal
WA
VE
FOR
M
if J0=
5v
b =
513
t 260
.sA
ctio
nIn
dica
tion
1. C
heck
V10
1, p
in 1
2. C
heck
V10
2, p
in 2
3. C
heck
V10
1, p
in 2
4. C
heck
V10
3, p
in :3
5. C
heck
V10
3, p
in
Step
1
Step
2
Sign
alG
o to
ste
p 2
No
sign
alG
o to
ste
p 3
Sigr
.;.!.
1N
o si
gnal
Tro
uble
in V
102,
T10
1, o
r as
soci
ated
cir
cuitr
yT
roub
le in
filt
er n
etw
ork
Sign
alT
roub
le in
No
sign
alG
o to
ste
p 4
Sign
alT
roub
le in
No
sign
alG
o to
ste
p 5
No
sign
alT
r.ou
ble
inC
heck
Tim
ixo
ON
E in
put f
rom
test
er a
t jun
ctio
n(FC
5)
S-L
ep 3
Step
4
15 V
8160
p-s
44-7
f
Figu
re 3
.E
xam
ple
of ta
bula
r pr
esen
tatio
n of
insl
.ruc
tions
.
Graphs
Graphs are of particular value when the writer wishes to displaythe general shape of a function, or where trends or relationships betweenquantities are beig described and contrasted. There are several differ-ent forms of graphs. The three most popular are the line graph, the bargraph or histogram, and the circle or pie graph. The graph should beplaced close to the place where it is mentioned in the text.
In the typical line graph, when the functions portrayed are con-tinuous, a set of values is described on the horizontal or x axis andanother set of values is described on the vertical or z axis. When plot-ting the results of an experiment, the independent variable, by conven-tion, is laid off on the horizontal axis and the dependent variable is laidoff on the vertical axis. The following will help in the preparation ofcomprehensible line graphs:
1. draw the horizontal and vertical axes with heavyand distinct lines
2. label the axes in terms of the units used (e. g. , dol-lars in thousands, percentage, etc.)
3. title the graphs descriptively but as concisely aspossible
4. place the title below the graph; never place it withinthe grid or plotting area
5. select convenient scales for both axes; the scale se-lected must rema4. the same from the beginning ofthe axis to the enc.
6. select units that are easy to divide (e.g., 1, 2,5, 10,50, 100, etc.) so that intermediate points will be rela-tively easy to read
7. construct the axes so that the vertical axis is 2/3 to3/4 as long as the horizontal axis
8. arrange the graph so that it is read from left to rightand from the bottom to the top
39
9. assign the value of zero wo the point where thetwo axes meet; if the data are such that there isa large gap between zero and the next value, theaxis is discontinued between zero and the firstvalue by using the symbol (see Figure 4)
10. use dots or crosses to plot values correspondingto the intercept points on each axis
11. If several curves of different importance are plot-ted in the same graph, make the principal curvethe most prominent (e.g., use a thicker line)
12. limit the number of functions plotted (curves on the .
graph) to four; try to avoid a cluttered appearance
13. provide legends wherever an explanation or an iden-tification of different lines is required
An example of a line graph is presented in Figure 4.
The second most commonly used type of-graph is the bar graph.The bar graph is a representation of discrete numerical quantities and isuseful for displaying differences among these numerical quantities. Forexample, if the frequency of students from various states were to be plot-ted on a bar graph, the states would be listed on one axis and frequenciesor percentages on the other. There are no rules regarding the axis to usefor the different variables, or the lengths or heights of the bars. If thereis no logical or ordinal relationship among the discrete entities plo+ted,then they may be arranged from lowest to highest. An example of a bargraph is displayed in Figure 5. The graph should be titled. By conven-tion, the title is placed below the graph. Provide supplementary descrip-tive information thatis required for understand'ng the graph in a key orlegend.
40
EX
AM
PLE
OF
LIN
E G
RA
PH
190
ME
N18
0
170
160
150
0H
0-
140
WO
ME
Net
.
013
0
120
110
100 ot
a-i so
111
1111
1111
111
6162
6364
65.6
6676
8697
0717
2737
4H
EIG
HT
(IN
CH
ES
)
Figu
re 4
.W
eigh
t as
a fu
nctio
n of
hei
ght f
or m
en a
nd w
omen
bet
wee
nth
e ag
es o
f 18
and
64.
41
12 10 8 6 4 2 0
MO
M
iiYE
AR
196
5
YE
AR
195
6
EX
AM
PLE
OF
BA
R G
RA
PH
LES
S T
HA
N8
I
9-11
13-1
516
or M
OR
E
Figu
re 5
.A
nnua
l mea
n in
com
e of
mal
es (
25 y
ears
and
old
er)
in19
56 a
nd 1
965
as a
fun
ctio
n of
year
s of
edu
catio
n.
The circle or pie graph displays differences among the data interms of different size sectors or segments. It is useful for demon-strating the relationship among parts of a whole. It is a simple graphto draw, and easy to understand. To prepare an effective circle or piegraph:
1. title the graphbelow the circle
2. legend each sector
3. include in each sector the proportion of the totalcircle that is represented by the sector
4. make the size of each sector directly proportionalto its proportion of the whole circle
5. use a protractor for drawing the circle and forplotting the percentages of each segment
An example of a circle graph is presented in Figure 6.
Schematic Diagrams
Schematics have been broadly used for equipment maintenanceinstruction and on-the-job maintenance support in both military andcommercial settings. The IMM (Integrated Maintenance Manual) and theSIMM (Symbolic Integrated Maintenance Manual) by the Navy and CoastGuard are examples of military manuals that have incorporated the useof concise, consistent, and well-laid out schematics of equipment. PIMO(Presentation of Information for Maintenance and Operation) is the AirForce version for presenting technical order type instructions and in-formation. The information in PIMO emphasizes how to do something,rather than what should be done.
43
BEST COPY AVAILABLE
EXAMPLE OF PIE GRAPH
4 YEARS HIGn SCHOOL31.2 S
1-3 YEARS HIGH SCHOOL
18.2 %
1 -3 YEARS COLLEGE
8.9%
4 YEARS COLLEGEOR MORE
9.8%
LESS THAN 5 YEARSELEMENTARY SCHOOL
6.5%
8 YEARS ELEMENTARYSCHOOL
15.1%
Figure 6. Percentage of United States, population completingvarious years of schooling, 1966.
Comprehensible schematics will include:
1. easy to understand layouts
2. clearly demarked circuits and components
3. maintenance data
4. the same functional divisions as the equipment
5. aid for locating components (e. g. , grid underlay)
6. cross-sectional schematic views of mechanicaldevices
7. coding (e. g. , flow arrows to show direction offlow)
8. wiring and cabling interconnections
When text is included with the schematic, it should be:
1. placed on the page facing the schematic
2. readily associated with the schematic portionto which it pertains
3. short but complete
Figure 7 presents a schematic that incorporates many of theprinciples outlined above.
4.5
GE
N C
SA
P/0
OM
NI
IMM
HY
BR
ID B
LO
CK
ED
SCH
EM
AT
IC)
PIA
P3A
M
TS
uct_
oar
n
n. 4
441
wai
lluki
LIC
AM
MO
LU
NE
OIL
.
Figu
re 7
.E
xam
ple
of s
chem
atic
inco
rpor
atin
g co
mpr
ehen
sibi
lity
prin
cipl
es.
Thi
s ex
cerp
t pre
sent
s de
taile
d sc
hem
atic
leve
l cov
erag
e of
a h
ybrid
equ
ipm
ent c
onsi
stin
g of
ele
ctric
al a
nd fl
uid
cont
rol c
or.t.
pone
nts.
Fea
ture
s in
clud
e:
PR
EC
ISE
HA
RD
WA
RE
DE
FIN
ITIO
NH
ardw
are
boun
darie
s ar
e de
fined
by
gray
shad
ing.
PR
EC
ISE
FU
NC
TIO
NA
L D
EF
INIT
ION
Fun
ctio
nal e
lem
ents
(ci
rcui
ts a
nd m
echa
ni-
cal c
ompo
nent
s) a
re d
efin
ed b
y bl
ue s
hadi
ng.
INT
EG
RA
TIO
N O
F E
LEC
TR
ICA
L A
ND
ME
CH
AN
ICA
L D
EV
ICE
SE
lect
rical
, mec
hani
cal,
hydr
aulic
, and
ane
w,
mat
ic in
form
atio
n is
inte
grat
ed to
incr
ease
unde
rsta
ndin
g.
IMP
RO
VE
D P
RE
SE
NT
AT
ION
OF
ME
CH
AN
ICA
L D
EV
ICE
SV
alve
s an
d ot
her
mec
hani
cal d
evic
es a
repr
esen
ted
usin
g cr
oss-
sect
iona
l sch
emat
icvi
ews
to e
nhan
ce u
nder
stan
ding
.
IMP
RO
VE
D S
IGN
AL
CO
DIN
GC
oded
flow
arr
ows
show
con
tent
of f
luid
flow
line
s. C
oded
arr
ow-h
eads
den
ote
type
of e
lect
rical
sig
nals
.
IMP
RO
VE
D T
EC
HN
ICIA
N IN
TE
RF
AC
EF
ront
pan
el c
ontr
ols
'and
indi
catio
ns a
rede
note
d by
n c
utaw
ay w
hite
bac
kgro
und.
In a
dditi
on, t
his
diag
roln
has
o fa
cing
pag
ebl
ock
text
dia
gram
to p
rovi
de c
ompl
ete
desc
riptio
n of
ope
ratio
n.
The
use
of c
ross
-sec
tiona
l sch
emat
ic v
iew
sis
als
o ap
plic
ob:e
to e
quip
men
ts u
sing
mic
ro-
wov
e de
vice
s.
The
tota
l hyb
rid d
iagr
am r
epre
sent
s a
com
-pl
ete
func
tiona
l ent
ity, T
hus,
the
syst
em is
team
ed in
an
inte
grat
ed m
anne
r en
ablin
g th
ete
chni
cian
to e
asily
see
the
rela
tions
hips
betw
een
the
diffe
rent
com
pone
nts
of a
hyb
ridsy
stem
.
Additional Reading List
A manual of style. Chicago: The University of Chicago Press, 1964.
Anderson, U. A. Training the speaking voice. New York: Oxford UniversityPress, 1942.
Bruning, J. , & Kintz, B. Computational handbook of statistics. Glenview,Ill.: Scott, Foresman, 1968.
deKieffer, R. E., & Cochran, L. W. Manual of audio-visual techniques. NewYork: Pr-ritice Hall, 1962.
Gunning, R. The technique of clear writing. New York. McGraw-Hill, 1952.
Haas, K. B., & Packer, H. Q. Preparation and use of audio-visual aids.1955.
Prochnow, H. V., & Prochncw, H. V., Jr. The public speaker's treasurechest. New York: Harper & Row, 1964.
Tate, Merle W. Statistics in education. New York: Macmillan, 1964.
Thomas, P. E. L. .A guide for autiiors. Springfield, Ill.: Charles CThomas, 1061,
Wilmut, H., Chubb, G., Tabachnick, B. PIMD data preparation guidelines.Norton Air Force Base, Calif. , AFSC TR 69-155, May, 1969.
Wright, P. Writing to be understood: Why use sentences? Applied Ergo-nomics, 1971, 2, 207-209.
147 jair
III. "NONANALYTIC" METHODS FOR EVALUATINGTEXTUAL METHODS
While use of the word counting (e. g. , the Flesch or Dall-Challtechnique) or associated (e. g., cloze) methods will be useful in evaluat-ing the readability of textual materials, there are a number of nonanal-ytic techniques which will also be helpful to the material developer whowishes to evaluate his product. These nonanalytic methods are perform-ance oriented. They consider the actual performance of the person whoreads the material rather than the characteristics of the text itself. Gen-erally, the nonanalytic techniques involve administering the written ma-terials to a group and, then, testing the group to determine how muchthey have learned, o.. how much they comprehended about the subjectmatter from reading the materials.
It might be taken for granted that modifying the textual materialsto produce a lower (and presumably easier) reading level will necessar-ily produce learning benefits. But it is also logical to assume that a twopound weight will fall twice as fast as a one pound weight. In the finalanalysis, some test of the effects of a materials modification on actuallearning may be desirable. This chapter presents concepts and consid-erations important to such an endeavor. Such concepts and considera-tions include the design and conduct of such tests as well as the charac-teristics of the actual measurement instruments to be involved in theevaluation.
Design of Materials Evaluations
The evaluation of textual materials through nonanalytic methodswill rely on the learning they produce, as measured by tests of the per-formance of the reader. Such tests may be administered immediatelyafter the materials are read or after a time interval has elapsed. Inthe latter case, the final result will be confounded by the interveningactivity of the reader. Thus, the intervening activity must be careful-ly controlled, or it will not be known whether any actual learning canbe attributed to the materials themselves.
Because the design and analysis of a nonanalytic evaluation isa complex process, it is best to consult experts iii research methodsand statistical analysis before conducting such an evaluation. To per-form a nonanalytic test, study guidelines, which are specific and not
49
subject to personal interpretation, should be developed in advance. Theguidelines must describe fully the methods, procedures, data collectioninstruments, and statistical data treatment methods to be employed:Estimates of the time needed to conduct each aspect of the study shouldalso be included. Example methods for conducting such evaluations aredescribed below.
Single Group Methods
Assume that the textual materials for a course have been modi-fied and that the materials developer wishes to know whether or not themodification produces any significant change in the amount of studentlearning. Assume that there are four blocks in the course involved.The original materials (0) could be administered to students when theyare in blocks 1 and 3, while the modified (M) materials could be admin-istered to the same students when they are in blocks 2 and 4. The classscores on end of block 1 and 3 tests (0 materials) could then be com-pared with their scores on block 2 and 4 tests (M materials). One wouldanticipate higher scores on the M materials than on the 0 materials, andthe difference between the two could be used as an index of the gain achiev-ed by the modification.
In this type of study, each reauer is exposed to each type of mate-rial. This avoids certain problems associated with control group studies.However, note that this type of design assumes that learning block 1 doesnot affect learning in block 2, etc., and that blocks 1 and 3 are at thesame difficulty level as blocks 2 and 4. Such assumptions may not al-ways be appropriate, and when they are not, this method should be avoid-ed.
Control Group Studies
When the assumptions of the single group study cannot be met,control group studies can be employed. In the prior example, the stu-dents could have been split into two groups. In making such a split, itis essential that the two groups be equated on all variables which couldaffect the evaluative outcome. In the present context, reading gradelevel and background in the field of study might represent the criticalvariables on which the two groups would be equated.
5O
There are two methods for establishing equivalent groups. Oneis to select the persons for assignment to each group from the totalpool of available persons at random. For example, if an alphabeticalroster is available, every other person might be assigned to one of thegroups and the remainder would be assigned to the other group. Thesecond method is to match, for the variables thought important for thestudy, as closely as possible the persons assigned to each group. Thus,if there is available a pool of 20 persons of known and varying readinggrade levels and they are to be assigned to equivalent groups, the finalmatching might look as follows:
Matched PairReading Grade Level of Matched Subjects
Group A Group B
1 12.1 12.12 11.9 11.83 10.5 10.64 10.0 10.05 9.7 9.86 9.8 9.77 8.9 8.98 8.8 . 8.79 8.5 8.6
10 8.0 8.0
Having matched the groups, the control group technique involvesadministering the 0 materials (for all four blocks) to one group and theM materials (for all four blocks) to the other group and admiriisteringthe appropriate tests. Possibly, such tests would he administered atthe end of each block as well as at the end of all four blocks. Any dif-ferences between the test scores of the two groups (either on the indi-vidual block tests or the final examination) would then be attributable tothe textual materials themselves.
Factorial Design
A still more powerful nonanalytic evaluative approach is throughthe factorial design. Using the same example, assume that the evaluatorwishes to know whether or not there is a difference in the learning pro-duced by the two types of materials and also whether or not this differ-ence holds regardless of the general intelligence (AFQT score) of thestudent. Factorial designs permit the evaluator to simultaneously look
51
at the effects of two or more factors on learning. In the following exam-ple, AFQT is considered to be a factor, while difficulty of reading materi-als is considered to be a second factor.
HighAFQT
LowAFQT
Original Modified
( 1 )
..............Si. Si ........
s2s2
11(2)S
nsn
(3)
s1
---- -- s ........s....2
sl2
. -............. .
. .
. .
. ........
sn sn (4)
In this example, the total group is split into two groups: high AFQTscorers and low AFQT scorers. Half of the high AFQT scorers receive the0 materials and half receive the M materials. The same holds for the lowAFQT scorers--half receive the 0 materials and half receive the M materi-als. The analysis (called analysis of variance) of the test scores of the fourgroups will tell the evaluator: (1) whether or not a difference in learning isproduced by the two versions of the text, (2) whether or not high AFQT scor-ers make higher test scores than low AFQT scorers, and, most importantly,(3) whether or not there is a differential effect on the benefit to be gained byeach type of material as a function of AFQT score.
Within single group, control group., and factorially designed evalua-tions, there are a large number of sub approaches. Accordingly, an expertshould probably be consulted during the formulation of nonanalytic studies toinsure proper experimental design.
52
Conduct of Nonanalytic Studies
No matter how powerful the design of the nonanalytic study, usabledata will not be obtained unless most, if not all, of the following adequacy con-ditions are satisfied:
control--all conditions in the situation and within thesubjects which might affect the evaluation must be con-trolled; i.e., eliminated from the evaluation conditionsor distributed equally over all conditions
.samplethe readers in the experiment should be fullyrepresentative of those who will eventually use the ma-terials and a sufficient number of subjects must be in-cluded to allow stability in the resultant descriptive andinferential statistics employed
question realism--the question(s) asked by the study mustbe drawn from sound concepts or theory
assumptionstheeassumptions must be clearly stated andacceptable
objectives - -the study objectives must be unambiguous andclearly defined
personnel--the personnel who perform the study mustpossess sufficient training, experience, and sophistica-tion to properly conduct the experiment
stimuli--the reading materials included (stimulus repre-sentations) must be meaningful to or mirror the problemon hand, and the stimuli and stimulus combinations mustcover an appropriate range of materials
criterion weighting--if subscores are combined they mustbe appropriately weighted
alternate methods--backup or alternate methods should beavailable, in case the favored approach must be abandoned
time and costs--the time schedule must be realistic andcompatible Nv ith available manpower; the cost estimate mustbe compatible with manpower, facility, and equipment re-quirements; contingency time and costs must be allowed
53
Characteristics of Useful Measure of Performance
Aside from concise and specific methods and procedures, read-ability evaluations should employ performance measuring instrumentswhich possess sound measurement qualities. Recommendations from anevaluation can be no better than the data on which the recommendationsare based. Decisions have to be made on the basis of the readability eval-uative data. In order for a data set to be scientifically defensible, themeasures on which the data set are based should meet most, if not all,of the following criteria:
1. Reliability - -A measure is reliable to the extent thatit will yield the same score when a person is meas-ured with it on two separate occasions. If a measuredoes not possess this attribute, it is worthless forevaluation purposes. There are several methods forassessing the reliability of an evaluative measure andall of them involve the calculation of a correlation co-efficient. Any elementary statistics textbook will in-struct the reader in methods for calculating and in-terpreting a correlation coefficient.
2. Validity--Validity refers to the extent that a measureassesses what it intends or purports to measure. Itis possible for a measure to be highly reliable, yethave little or no validity.
3. Com relp_..ieris--Any useful performance evalu-ative measure must sample from the total range ofreading material and not from a part of the range.
4. Objectivity- -Only those aspects which can be objec-tively evaluated should be included in the measurementinstruments. If the evaluator's personal biases canenter into the data, the measurements are subjectiverather than objective. A measure is objective to theextent that different persons who use the same meas-ure will give a student the same score.
5. DifferentiationEvaluation measures must be able toreflect differences in the variable(s) being measured.This allows decisions and comparisons to be trade onthe basis of known amounts by which one presentationof a written selection differs from another presentationof the same selection.
54
6. Relevance and Appropriateness - -Only those know-ledge or performance aspects which are appropriatefor the course objectives should be included. Too of-ten evaluation studies focus on the measurement ofreader reactions and not on learning or behavior onthe job.
Examples of Dependent Variables
In evaluative research, the reader's response is always consideredthe dependent variable; i. e., that which is observ4.and measured. Theindependent variable is the variable that is manipulated to see what effectchanges in that variable will have on the dependent variable(s). In a readabil-ity study, the dependent variable could ;Je the trainee's responses on anachievement test. An example of an independent variable in a readabilitystudy is the variation of the textual materials under test.
In order to obtain a dependent variable measure which reflects stu-dent learning on the basis of reading textual material, tests will haVe to beconstructed. To accomplish this, an outline of the written material shouldbe prepared first. The outline need only be in terms of major headings orcategories. The test(s) should measure a representative sample of thesecategories. Decide, in advance of constructing the test(s), on the numbir ofitems that should be written in each area. The considerations that go intothe selection of the number of items to include in the various areas are: theimportance of each area in the total learning picture, the amount of spacedevoted to each area in the written material, and the objectives that havethe greatest value. It is also known that as the number of items in a testincreases, the reliability of the test increases.
There are several different types cf tests that can be constructed foruse in a readability study. Several of the most appropriate are presentedbelow. For a more thorough discussion of these tests, refer to the Handbookfor the Evaluation of Technical Training Courses and Students, AFIIRL-TR
55
Written Tests
When preparing a written test (also referred to as a paper andpencil test), an item pool of more than the finally required number ofitems should be developed so that if replacement items are needed forthe test, they will be available. There are several types or forms oftest items that can be used. The advantages and disadvantages of eachitem type should be considered as well as the learning objective to whichthe item is aimed. A test item should measure the learning objectiveas directly as possible and, depending on purpose, certain item typesare better than others.
The multiple-choice item, along with true-false and matchingitems, are referred to as "selection". items because the student selectshis response from alternatives that are offered to him. The selectionitems have certain advantages over "supply" items (items for which thestudent has to furnish his own response as in the essay item). The ad-vantages of the selection type item are: (1) test scorer personal biasesand subjective idiosyncracies are removed from the scoring, (2) easeof scoring (scoring keys or machines can be used), (3) more aspects ofthe material can be tested in a given period of time, (4) with increasesin comprehensiveness impruved validity may be expected, and (5) easeof application of statistical item analytic procedures to resultant data.
Criterion and Norm Referenced Testing
If a criterion referenced test is employed to judge the readabil-ity ox textual materials, the effects of reading the materials are com-pared with an absolute standard to determine whether or not acceptable(criterion) performance has been achieved. The absolute standard maybe passing a performance item which is at a given level of difficulty orachieving an absolute score. By contrast, the students' results on anorm referenced test are compared with the performance of a peer orreference group; e. g., a given percentile.
The characteristics of criterion referenced tests are that theyindicate the degree of competence attained from reading the materialwithout reference to the performance of others, and they measure stu-dent performance with regard to specified absolute standards (criteria)of performance.
56
Generally, criterion referenced tests tell us how effective thewritten materials are with regard to a specified standard of behavior.Criterion objectives, expressed in terms of specific behaviors, mustbe identified. Either performance testing or paper and pencil testingcan be used with criterion referencing. In criterion referenced meas-urement, grades are usually S (satisfactory) or U (unsatisfactory).
Performance Tests
In performance tests, the adequacy of the materials being eval-uated is tested through the performance of the reader on tasks whichare relevant to his present or future job. Performance tests can rangefrom simulated performance through performance of job tasks usingactual job equipment. Scoring can be based on measurement of per-formance in process, adherence of a final product to prescribed stand-ands, care and use of tools during performance, adherence to safetyprecautions, or some combination of these categories. Completion ofa prescribed task within an allotted time is also sometimes scored.One of the most popular methods of scoring is through a sequentialchecklist. In the sequential checklist, a task is broken down into thesequential elements which must be correctly performed if the task isto be completed. Each of these elements is then sequentially listedfor scoring by an examiner while he observes the performance of theperson being tested on the task.
The final score is determined by summing the points. A singleperformance test can be constructed which includes items dealing withsafety precautions, quality of the final product, and the manner in whichthe student performs.
There are variations to performance tests which avoid the needfor the student to perform a task. One such variation is the examineerecorded performance test. In this type of performance test, t'le stu-dent is niquired to write his response to a written question about a pic-ture or diagram of some performance aspect. His response is usuallya checkmark or a one-word answer. For example, in one test of thistype, the student is asked to identify an error in a picture of someoneengaged in an activity similar to that for which the student is training.
57
The Evaluative Milieu
It is difficult to complete an evaluative study in a training en-vironment which resists change. If the administration perceives re-vised materials as negatively affecting vested interests, many obsta-cles can be placed in the evaluator's path. In this case, the evaluator
*must meet with the persons who are resisting his efforts and attemptto persuade them to the benefits of the evaluation. Often, by makingthese persons a part of the planning committee for such a study, suchnegativity can be overcome.
58
Additional Reading List
Campbell, D., & Stanley, J. Experimental and quasiexperimentaldesigns for research. New York: Rand McNally, 1963.
Cochran, W. G., & Cox, G. R. Ex2erimental designs. New York:Wiley, 1957.
Gronlund, N. Constructing achievement tests. Englewood Cliffs, N. J.:Prentice-Hall, 1968.
Guilford, J. Fundamental statistics in psychology and education. NewYork: McGraw-Hill, 1965.
Siegel, A. I., Bergman, B. A. , Federman, P., & Sellman, W. S.Handbook for the evaluation of technical training courses andstudents. Lowry Air Force Base, Colorado, AFHRL-TR-72-15,February, 1972. (AD-753 094)
Suchman, E. Evaluative research: Principles and practice in publicservice and social action programs. New York: Russell SagesFoundation, 1967.
Tyler, R. Education evaluation: New roles, new means. Chicago: Uni-versity of Chicago Press, 1969.
Wittrock, M., & Wiley, D. (Eds.) The evaluation of instruction. NewYork: Holt, Rinehart & Winston, 1970.
Wood, D. Test construction. Columbus, Ohio: Merrill, 1960.
59
REFERENCES
Bormuth, J. R. Comparable cloze and multiple-choice comprehensiontest scores. Journal of Reading, 1967, 10, 291-299,
Buros, 0. K. !Ed.) The Sixth Mental Measurement Yearbook. HighlandPark, N. J.: Gryphon Press, 1965.
Caylor, J. S. , Sticht, T.G., Fox, L.C. , & Ford, J. P. Methodologies fordetei mining reading requirements of military occupational special-ties. Presidio of Monterey, Calif.: Human Resources ResearchOrganization, HumRRO Division No. 3, Technical Report 75-5,March 1973.
Dale, E., & Chall, J. S. A formula for predicting readability (and) instruc-tions. Educational Research Bulletin, 1948, 27. 11-20, 28, 37-54.
Dale, E., & Chall, J. S. The concept of readability. Elementary English1949, 26, 19-26.
Dale, E., & Tyler, R. W. A study.of the factors influencing the difficultyof reading materials for adults of limited reading ability. LibraryQuarterly, 1934, 4, 384-412.
Federman, P. J. , Macpherson, D. H., & Siegel, A. I. Normative develop-ment for submarine sonar manuals. Wayne, Pa.: Applied Psycho-logical Services, Inc. , 1970.
Flesch, R. Estimating the comprehension difficulty of magazine articles.Journal of General Psychology, 1943, 28, 63-80. (a)
Flesch, R. Marks of readable style. New York: Bureau of Publications.Teachers College, Columbia University, 1943. (b)
Flesch, R. The art of plain talk. New York: Harber & Bras., 1946.
Flesch, R. A new readability yardstick. Journal of Applied Psychology,1948, 32, 221-233.
Flesch, R. The art of readable writing. New York: Harper, 1949.
60
Flesch, R. How to test readability. New York: Harper, 1951.
Flesch, R. Reply to "Simplification of Flesch reading ease formula. "Journal of Applied Psychology, 1952, 36 54-55.
Flesch, R. How to make sense. New York: Harper, 1954.
Gray, W. S. , & Leary, B. E. What makes a book readable. Chicago:University of Chicago Press, 1935.
Grieme, R., Cleveland, D., Chubb, G., et al. Project PIMO Final Report:Volume IV - PIMO technical data format specifications. TR -69 -155,May 1969. Space and Missile Systems Organization, Norton Air ForceBase, Calif. (AD-852 104)
Lively, B. A., & Pressey, S. L. A method tor measuring the "vocabu-lary burden" of textbooks. Educational Administration and Super-vision, 1923, 9, 389-398.
Lorge, I. Predicting reading difficulty of selections for children. Ele-mentary English Review, 1939, 16, 229 -23 ?.
Lorge, I. Predicting readability. Teachers College Record, 1944, 45,404-419.
Lorge, I. The Lorge and Flesch readability formulae: a correction.School and Society, 1948, 67, 141-142.
Madden, H. L. & Tupes, E. C. Estimating reading ability level fromthe AQE General Aptitude Index. Lack land Air Force Base, Texas:Personnel Research Laboratory, Aerospace Medical Division,PRL-TR-66-1, AD-632 182, February 1966.
McLaughlin, G. H. SMOG gradi.ag: A new readability formula. Journalof Reading, 1969, 12, 639-646.
Ojemann, R. The readability of parents and factors associated with thereading difficulty of parent education materials. In G. D. Stoddard(Ed.), Researches in parent education II. Iowa City, Ia.: Univer-sity of Iowa., 1934. Cited by J. S. Chall, Readability - an appraisalof research and application. Columbus, Ohio: Ohio State University,Bureau of Educational Research, 1958.
Powers, R. D., Sumner, W. A., & Kearl, B. E. A recalculation of fouradult readability formulas. Journal of Educational Psychology,1958, 49, 99-105.
61
Smith, E. A., & Sentner, R. J. Automated readability index. Wright-Patterson Air Force Base, Ohio: Aerospace Medical Division,AMRL-TR-66-220, November 1970. (AD-667 273).
Spache, G. A new readability formula for primary grade reading ma-terials. Elementary School Journal, 1953, 53, 410-413.
Taylor, W. Cloze procedure: A new tool for measuring readability.Journalism Quarterly, 1953, 30, 415-433.
Thorndike, E. L. Improving the ability to read. Teachers College Re-cord, 1934, 36, 1-19, 123-144, 229-241.
The integrated maintenance manual. Anal eim, Calif.: Technical Opera-tions, Inc.
Tribe, E. B. A readability formula for the elementary school based uponthe Rinsland vocabulary. Unpublished doctoral dissertation, Uni-versity of Oklahoma, 1956. Cited by G. R. Klare, The measure-ment of readability. Ames, Ia.: Iowa State University Press, 1963,69-70.
Vogel, M., & Washburne, C. An objective method of determininggrade placement of children's reading material. ElementarySchool Journal, 1928, 28, 373-381. Cited by C. R. Klare, Themeasurement of readability, Ames, Ia.: Iowa Staff.:: -UniversityPress, 1963, 38-39.
Washburne, C., & Vogel, M. What books fit what children? School andSociety, 1926, 23, 22-24. Cited by G. R. Klare, The measure-ment of readability, Ames, Ia.: Iowa State University Press, 1963,38.39.
Wheeler, L. R., & Smith, E. H. A practical readability formula for theclassroom teacher in the primary grades. Elementary English,1954, 31, 397-399.
Wilmot, H. L. , Chubb, G. P. , Tabachnick, B. J. , et al. Project PI.M0Final Report: Volume VI PIMO technical data preparation guide-lines. TR-69-155, May 1969. Space and nssile Systems Organi-zation, Norton Air Force Base, Calif. (AD-852-106).
62
APPENDIX A
How to Perform a Cloze Analysis
The cloze procedure involves: deleting every nth word (usuallythe fifth word) in passages selected from the text under consideration,administering the mutilated passage to a sample of typical readers andasking them to fill in the blanks, and scoring the answers.
Sampling Reading Passages
Since i will be very difficult and impractical to test all the read-ing material in a particular manual, book, or course, you will have toresort to a sampling technique. The sampling technique employed shouldprovide samples that are representative of the reading matter and thatprovide adequate and sufficient data for analysis. To accomplish this,you should:
1. Select samples of approximately 500 words, butnot less than 250 words
2. Select samples from each chapter or section (orwhatever other divisions are used in the writtenmaterial). Try to take two samples from eachsection.
3. Use a random technique for selecting a particu-lar sample. A table of random numbers may beused (these tables are often included in standardstatistics text books) such that a digit from thetable would identify the paragraph to use in thesample. This would require prenumbering allthe paragraphs in the reading matter.
4. Begin each sample at the beginning of the para-graph. Use as much text (250-500 words) asnecessary to provide a minimum of 50 blanksper sample.
63
Preparing the Sample Reading Passages
The words can be deleted manually or by computer program.The computer program would be advantageous if the samples wereextensive in length. If alternate forms of the doze test are needed(e.g., to prevent compromising answers of two groups of subjects who
will be tested at different times, if the same reader will be testedat different times but on the same passages, or if retention over suc-cessive time periods is of interest) different deletion patterns can beused. This could be accomplished by deleting every fifth word startingwith the first, second, third, or fourth word. Other deletion patternssuch as deleting every sixth, seventh, or even tenth word could also beused. The five word pattern, starting with the first word in the para-graph is the most popular. The closer the blanks are to each other,the more difficult the task becomes. As the deletion pattern increasesfrom every fifth word, the passage length will necessarily increase inorder to have a passage with at least 50 blanks (e.g., if every sixthword is deleted, a 300 word passage will be needed; if every tenth word
is deleted, a 500 word passage will be needed). In applying the dozetechnique, the following also applies:
1. End each test passage at the end of a sentence.It is not imperative to use the same number ofwords in each test passage since comparisonswill be made on a percentage correct basis.
2. Words are defined as a unit with spaces on eith-er size: Thus, numbers or groups of lettersmay be considered as words. Examples of singlewords are: 100, AFB, isn't, on-the-job, and i. e.Punctuation marks such as hyphens, apostrophes,and commas are deleted along with the words.
3. Hyphenated words are not deleted as units if thetwo parts are not bound to each other, as in on-the-job, but are free forms, as in readability-comprehensibility. In the latter case, the hyphenis not deleted with the word.
4. All blanks should be of the same size, A 15 spaceblank is most commonly used.
64
Subjects
In order to be able to generalize the results of the readabilitystudy, a sample of subjects must be selected that is representativeof the population who normally read the material. Wherever possible,a random sample of the entire population of concern should be selected.This is often impossible to accomplish because the group may be sep-arated geographically, because of time and cost restrictions, or be-cause of other reasons.
Procedures
When administering the reading passages, a full set of directionsshould be provided to the subjects. A technique often used in this situ-ation is to have the subjects read the di.'ections to themselves while theadministrator reads them aloud. The eections should inform the sub-jects of the purposes of the study, the manner in which the blanks shouldbe filled in, and any restrictions in time limit (although rigid time re-strictions should not be used in a cloze analysis). the subjects shouldbe reassured that the study they are participating in involves a test ofthe material they are reading and not of them. An example passageshould be provided. The subjects should be given a few minutes to re-cord their responses in the example and then the correct answers shouldbe read to them. Some other points that may be made in the directionsare: (1) enter only one word or number in each blank space, (2) guess ifnecessary since there is no penalty for guessing, and (3) write down aword even if the spelling is incorrect. After providing the subjects withan opportunity to ask questions, they should be told to begin. A sampleset of instructions that was used in one study is presented below.
65
Instructions
This questionnaire is made up of six reading pas-sages. Some of the passages come from training materialsbeing used now. Others were specially written for compari-son. In each passage, about one-fifth of the words have beenleft out. We want you to write in each blank space the exactword you think is missing.
For example, in the sentence, "George LJa 11 i 9 VI
chopped down a ehe-yr4 tree," the person respondingthought that the words "'Washington" and "cherry" were themissing words, so hewrote them in the blank spaces.
Some of the missing words will be easy, and otherswill be quite hard. Sometimes the word will "pop" into yourmind; at other times clues in other parts of the passage willhelp you decide on the right word. In any case, you shouldtry to write in each blank the exact word you think was takenout.
As you finish each section, go right on to the next one.Do not rush through the materials. You will have about anhour and a half to finish, or about 15 minutes for each section.
In scoring responses, accept as correct those responses that areidentical to the deleted word. Synonyms are not correct, nor are changesin tense. However, words are accepted as correct even if they are mis-spelled.
Results
In analyzing the results you should:
1. Tabulate the number of correct responses oneach passage
2. Determine each subject's comprehension levelby calculating the percentage of correct re-sponses (in the cloze procedure, with everyfifth word being deleted, approximately 44 percent correct is considered acceptable for use
66
in supervised instruction and 57 per centcorrect for use in independent instruction)
3. Assign a reading grade level to each passage(in order to do this, data on many subjectsmust be obtained and statistical proceduresused which would allow for relating the dozescores of a group of subjects to a reading gradelevel). The statistical procedures involved arerather sophisticated and therefore reference toa statistical source is recommended. Readinggrade levels for the subjects may be obtained byusing the Madden and Tupes procedure, as de-scribed in Chapter I, Reading Level Determina-tion.
67
APPENDIX B
Examples of Before and After Material and AutomatedReadability Index (ARI) and Grade Level (GL) Analyses
The following three examples demonstrated the effect on readahil-ity of revised reading material. The excerpts were taken from Air Forcemanuals, revised in accordance with the principles for making written copymore readable, and then ARI and reading grade levels were calculated.
BeforeThe receiving inchecker's first
duty is to process incoming ship-ments from the transportation sec-tion, commercial carriers, or par-cel post. He insures that the totalnumber of boxes, drums, parcels,etc; , received are correctly re-flected on the receiving documentsthat accompany the shipments. Healso checks for physical damagethat may have occurred during trans-it or as the result of improper pack-aging or handling by the carrier. Heexplains noted discrepancies on thereverse side of the receiving docu-ment and obtains the signature ofthe carrier as acknowledgment ofthe discrepancies. Property is seg-regated by priority, cost category,and Federal supply class as it isunloaded. If this is not feasible,due to volume or time limitations,segregation is accomplished priorto processing the property throughthe receiving and inspection lines.
Classified items are not allow-ed to remain on the receiving line,awaiting unchecking and inspectionactions. Upon completion of receiv-ing checkings and inspection action,the receipt is input via the remotekeyboard printer.
ARI = 70.50GL = 15.01
68
AfterThe most important job of the
receiving inchecker is to processall incoming shipments. Shipmentsmay come from the transportationsection, commercial shippers, orparcel post. The inchecker makessure that the number of boxes,druins, or other containers receiv-ed is shown on the papers that comewith the shipment. He also looks tosee if the shipment is damaged. Ship-ments may be damaged during shippingor because of rough handling or badpackaging. If any of the numbers arewrong or if he finds any damage, hewrites it down on the back of the re-ceiving papers. The person who de-livered the material must also signthe bacx of the form. This provesthat the delivery man knew that theshipmelt was damaged, or was toobig or too small. While it is beingunloaded, the shipment is separatedby: (1) priority, (2) cost, and (3)Federal supply class. If the ship-ment is too big to operate then or ifyou don't have enough time, it is allright. But, it must be separated be-fore it goes to the receiving and in-spection lines. You must not makeclassified items wait on the receiv-ing line for inchecking and inspection.When the checking and inspecting isdone, the receipt of the shipment isinput via the remote keyboard printer,
ARI = 58.79GL = 8.95
Before
Radio interference suppressionis the elimination or minimizing ofthe electrical disturbances which in-terfere with radio reception, or dis-close the location of the vehicle tosensitive electrical detectors. It isimportant that vehicles with or with-out radios be properly suppressed toprevent interference with radio re-ception of neighboring vehicles.
The ignition and generating sys-tems have been designed to suppressradio interference. Ignition systeminterference suppression is effectedby a primary capacitor in the dis-tributor wiring harness receptacle,a resistor suppression in the sparkplugs, and by shielded spark plugcables. Radio interference suppres-sion in the generating system is ef-fected by a choke, capacitor, filter,and rectifier in the generator regu-lator assembly. The primary capaci-tator is housed in the distributor wir-ing harness receptacle and is an inte-gral part of the primary circuit.Spark plugs are shielded individuallyby metallic braid beneath the rubberinsulation. The generator regulatorassembly houses the capacitor, choke,filter, and rectifier.
ARI = 72.77GL = 16.73
69
After
Radio interference is caused byelectrical impulses. The interfer-ence can be stopped or cut down. De-tectors can locate vehicles. All vehicles should be checked so they willnot interfere with the radio receptionof other vehicles.
Ignition and generating systemswere designed so they won't inter-fere with radios. This is done through3 parts of the ignition system. One isa capacitor in the distributor wiringharnesk,.. The capacitor is part of theprimary circuit. Another is a resis-tor in the spark plugs. The third isthe shelded spark plug cables. Eachspark plug is shielded by metal braidsunder rubber insulation, There are 4items in the generator regulator as-sembly that cut radio interference.The: e are the choke, capacitor, filter,and rectifier.
ARI = 56. 145GL = 7.73
Before
The process described for de-veloping the human component of aweapon system has been combinedwith evolving educational and train-ing methodology to form an effec-tive process for designing and oper-ating training programs--the sys-tem process. An instructional sys-tem is an integrated combination ofresources (students, instructors,materials, equipment and facili-ties), techniques, and proceduresperforming efficiently the functionsrequired to achieve specified learn-ing qbjectives. Under this concept,we must ensure that training pro-grams are not bound by traditionalprocedures, techniques, or rou-tines of the day. Therefore, instruc-tional system designers must care-fully consider both current and fu-ture needs when identifying and de-signing facilities, equipment, anddevices. This same total systemsconcept must be applied when iden-tifying and planning instructional ac-tivities. When working in this area,you must develop and organize learn-ing experiences that will enable stu-dents to apply what they have learnedto new or changing situations. The in-structional system must employteach-ing techniques that will producegradu-ates who will not become the victimsof fixed habits, procedure& or atti-tudes, nor be bound by extreme speci-alization.
ARI = 74.783GL = 16.79
70
After
A plan to design and operatetraining courses was developed. Itis known as the system process. Itjoins the methods used to develop aman in a weapon system with thoseof education and training. A train-ing plan achieves its learning goalsby combining a few things. They areresources, techniques, and proce-dures. By resources, we mean stu-dentd, teachers, materials, tools,and facilities. Training coursesshould not be bound by old fashionedprocedures or techniques. Facil-ities, equipment, and techniquesshould be designed for current andfuture needs. Develop and plan thesystem so that students can applywhat they learn to new situations.The techniques used should not makestudents creatures of habits. Also,students should not be bound by speci-alization.
ARI = 58.457GL' = 8.91
APPENDIX C
How. to Perform a Flesch AnalysisAccording to Flesch, readability is composed of two componentparts, reading ease (an estimate of the case a reader will have in read-ing and understanding what was written) and human interest (an estimateof how interesting the reader will find the written material).
Sampling Reading Passages
Since it will be very difficult and impractical to test all the read-ing material in a particular manual, book, or course, you will have toresort to a sampling technique. The sampling technique employed shouldprovide samples that are representative of the reading matter and thatprovide adequate and sufficient data for analysis. To accomplish this, thefollowing should be considered:
1. Select three to five samples from an article and25 to 30 samples from a book.
2. Samples should be taken at random. Tables of ran-dom numbers may be used or a scheme such thatevery fifth paragraph, or every other page, orevery tenth page is used. Do not select introduc-tory paragraphs as samples (they are usually nottypical of the style of the rest of the section).
3. Each sample should have approximately 100 words.
4. Start each sample at the beginning of the para-graph.
71
Procedures
The following steps should be performed to determine reading
ease:
1. Count each word in the sample, marking offthe 100th word.
2. Words are defined as a unit with spaces oneither side. Thus, numbers or groups of let-ters may be considered as words. Examplesof single words are: 100, AFB, isn't, on-the-job, and i.e.
3. Calculate the average number of words in eachsentence. That is, count all the sentences inthe sample. Then, divide the number of wordsby sentences. For instance, if you have a 110
word passage and 12 sentences, the averagesentence length is 9 (rounding off to the nearestwhole number). If more than one sample is se-lected from a piece of writing, add the total num-ber of sentences in all the samples and divide thetotal number of words by the total number of sen-tenc es.
4. If you restrict the sample to exactly 100 words,then the sentence in which the 100th word falls iscounted in the sentence count if more than halfthe words in the sentence were included in the 100word count. For example, if the last sentencewas "I sang for my supper, " and the 100th wordin the sample was sang, this sentence would notbe counted.
5. The following punctuation marks define a sentence:period, question mark, exclamation point, ck.)lon,and semicolon. However, sentence counting be-comes complicated with colons and semicolons incertain situations. In order for the, clauses to becounted as separate sentences, they must be in-dependent and a separate unit of thought. For
72
example, the following is counted as one sentence;The advantages of this technique are: (1) ease ofuse; (2) validity. But, the next example is countedas two sentences; He is tall and dark she is shortand blond.
6. Count the average word length in syllables. Thisis done by counting all syllables and dividing thetotal number of syllables by the number of words.In the final formula, the average word length insyllables is expressed as the number of syllablesper one hundred words. Therefore, multiply theresult of the total sample by 100. When using 100word syllables, count the total number of syllablesin all the samples and divide by the number of sam-ples.
7. Syllables are counted the way the word is pronounced.For example, average has two syllables, and read-ability has five syllables. Syllables, for numbersand symbols, are counted the way they are pro-nounced; e.g. , AFB has three syllables, # (number)has two syllables, and 34 (thirty-four) has threesyllables.
8. The reading ease score is determined by the follow-ing formula;
Reading Ease Score = 206. 835 - [1. 015 (averagesentence length) + . 846 (number of syllables perone hundred words)]
This result will vary between 0 (unr9dable) and100 (ea o read). The table below may be used
Reading Ease Score90-10080- 8970- 7960- 6950- 5930-- 490- 29
73
Interpretation,Very EasyEasyFairly EasyStandardFairly DifficultDifficultVery Difficult
Grade Level567
8 and 910 to 1213 to 16
College graduate
To find the human interest score:
1. Count the number of personal words per 100words. In testing a whole piece of writing,divide the total number of personal words bythe total number of words and multiply by 100.If 100 word samples are used, count the per-sonal words in each sample and divide the tot-al number of personal words in all samples bythe number of samples.
2. Personal words are the first, second, and thirdperson pronouns. The neuter pronouns it, its,itself are not counted. The pronouns they, them,their, theirs, themselves are counted if they re-fer to people rather than things. He, him, his,she, her, and hers are always counted (even ifthey refer to animals or things).
3. Count all words that are of the masculine or femi-nine form, such as Betty, mother, brother, John,ballerina. However, words that are common toboth sexes are not counted; e.g., lawyer, parent,singer, child. Doctor John Smith has one personalword (John); Mr. John Smith has two personal words(Mr. and John).
4. Count group words; e.g., folks and people as per-.sonal words.
5. Count the number of personal sentences per 100 sen-tences. In testing a whole piece of writing, dividethe total number of personal sentences by the totalnumber of sentences and multiply by 100. In testingsamples, divide the number of personal sentencesin all the samples by the number of sentences in allthe samples and multiply by 100.
6. Personal sentences are spoken sentences markedby quotation marks or other indications that thesentence was spoken (spoken sentences are some-times set off by commas and colons). Do notcount quoted phrases when the quotes do not iii-dicate spoken phrases; e. g., The Flesch Itead-ability technique measures "reading ease" and"human interest. "
7. Count as personal sentences all sentence:. thatare addressed to the reader; e. g., Add 3 + 2,Count all personal sentences, Do not count wordsthat are common to both sexes. Do not count .
sentences that are vaguely addressed to the read-er; e. g., It is a good idea to arrange the dataneatly.
8. Exclamations are counted as personal sentences;e. g., Holy cow!
9. Incomplete sentences whose meaning has to be in-ferred are counted as personal sentences; e. g.,No, So, So.
10. The human interest score is determined by thefollowing formula:
3.635 (number of personal words per 100 words) +. 314 (number of personal sentences per 100 sentences)
This result will vary between 0 (no human interest) and100 (much human interest). Using the following tableto interpret the human interest score:
Human Interest Score Interpretation
60 to 100 Dramatic40 to. 59 Highly Interesting20 to 39 Interesting10 to 19 Mildly Interesting0 to 9 Dull
Example
When thinking of "What will the student be able to do? " wemust first answer the question "What can the student already do? "This has a direct impact on the analysis of each training standardtask/ subject knowledge statement and eventually affects the depthand scope of the learning objectives. For example, if the typicalstudent's previous work has involved related tasks, many elementarydetails may be omitted. This applies to the analysis of the trainingstandard and the development of objectives. AFM 50.5, USAF FormalSchools Catalog and paragraph 3 of the specialty description in AFM36-1 or AFM.
Reading Ease Score
Average sentence length =
Number of syllables = 179
no. of words 100 = 20no. of sentences 5
Reading Ease Score = 206.835 - [1. 015(20) + .846(179)1 =206. 835 - [20. 3 + 151. 434] =206. 835 - 171. 734 = 34. 101
Human Interest Score
Personal words = 1Personal sentences = 1
Human Interest Score = 3.635(1) + .314(1) = 3.949
76
APPENDIX D
How to Perform the FORCAST Analysis
The FORCAST technique provides a reading grade level basedon a monosyllabic word count. An example of the technique is demon-strated below. The sample reading passage was excerpted from a mili-tary guide for instructors.
An instructor cannot read the mind of a student toverify the extent of his understanding. It is only throughsome overt activity on the part of the student that the ex-tent of his knowledge or skill can be determined. To saythat a student will "develop an understanding of Ohm'sLaw" is often used as an example of an obscure objective.One person may feel that if the student writes the law itindicates understanding. Another may say that the studentshould explain the law. A third may contend that the onlyway the student can satisfy this objective is for him to usethe formula to solve problems in electrical circuitry.
Number of one syllable words in a 150 word passage = 76
Reading Grade Level = 20 ono. of one0syllable words) 20 - 7.6
1
= 12
77
APPENDIX E
How to Perform the SMOG Analysis
The following example demonstrates the procedure for calculat-ing a SMOG grade on a passage of 30 sentences.
Sample Test
1. GENERAL INFORMATION
The need for human engineering in the rational defense isbased on the fact that the science of man. and his capacitiesmust keep pace with technology if military effectiveness isto be achieved. Machines cannot fight alone; they requiremen to operate and maintain them. The task of human en-gineering, both generally and as an element of the Person-nel Subsystem (PS), is to get the best from man in the oper-ation of the highly complex equipment produced by moderntechnology. (Some of the concepts expressed in this and thefollowing paragraphs are based on Ref 193.) Human engi-neering proceeds on the assumption that the capacities ofman develop within certain limits and that by adapting thedesign of equipment to these capabilities, a more effectivesystem will result. A system that requires extreme, un-interrupted concentration over long periods of time wouldnot be "human engineered" nor could a system be reliedupon in combat if it were so complex that it could be main-tained only by the engineers and scientists who designed it.As used in this handbook, the term "human engineering" isnot synonymous with human factors. The term "human fac-tors" is more comprehensive, covering all biomedical andpsychosocial considerations applying to man in the system.It includes not only human engineering, but also life support,personnel selection and training, training equipment, jobperformance aids, and performance measurement and eval-uation. Human factors is closely related to and is oftenused as a synonum for PS. Sometimes the total human fac-
. .tors effort (including human engineering and other PS ele-ments) is called "bioastronautics" or "life sciences."
78
2. THE PRACTICE OF HUMAN ENGINEERING
System performance must not be degraded by improper de-sign which imposes limitations on the human operator ormaintenance technican. Consequently, the Air Force Sys-tems Command (AFSC) has been assigned responsibility forensuring that new systems are planned and designed so asto exploit known human capabilities, but, at the same time,not require any performance which exceeds those capabil-ities. Carrying out this responsibility constitutes the "prac-tice" of human engineering in the Air Force context.
2.1 Who Does Human Engineering?
No single professional group is responsible for research inhuman engineering or the application of human engineeringprinciples to the design of systems. Engineers, industrialengineers, anthropologists, physicians, physiologists, andpsychologists all participate. In a broad sense, everyoneconcerned with the design or improvement of devices whichpeople use or the ways in which they use them is practicinghuman engineering. However, as so often happens in a tech-nical field, a point is reached where every man can no longerbe his own specialist. The profession of human engineeringhas emerged to meet this demand. Human engineering drawsits basic data from several fields of knowledge, among them(1) the engineering sciences, (2) experimental psychology,(3) physical anthropology, and (4) psychophysiology. Theprofession calls for varied talents and broad training in boththe physical sciences and the sciences that deal with man. Itis understandable, therefore, that many human engineeringspecialists are experimental psychologists conversant withengineering or engineers with supplementary training in thebehavioral sciences.
79
2.2 What Do Human Engineers Do?
Human engineers conduct research, establish principles,and assist in designing and developing system hardwarethat is properly engineered for effective human use. Theystudy the hu:nan being as a user of equipment and equip-ment as a tool for a task. In collaboration with hardwareengineers and designers, human engineers seek to developnew and improved man-equipment interfaces that will sim-plify the operator's task and increase the probability ofmission accomplishment. They seek to achieve displaysthat will most effectively present information to the humansenses, to obtain the most efficient controls for humanuse, and to provide an optimum work environment. Theycontribute to the development of such complex man-equip-ment systems as combat-information centers, air defensesystems, space vehicles, and offensive weapon systems.In the design of military equipment, human engineers placeemphasis on efficiency, as measured by the speed and ac-curacy of performance, and on the safety and comfort ofthe human operator. Because the successful design ofequipment for human use requires consideration of man'sbasic characteristics, human engineers study man's sen-sory capacities, his muscular strength and coordination,his body dimensions, his perception and judgment, hisnative skills, his optimum workload, and his requirementsfor comfort, safety, and freedom from environmental stress.
3. HUMAN ENGINEERING IN THE AIR FORCE
Within the Air Force, AFSC is responsible for both the de-velopment and the application of human engineering know-ledge and techniques.
Sample Calculation
SMOG Grade = 3 + Jwords of 3 syllables or more in 30 sentences
= 3 + 4179 = 3 + 13.4
= 16
80
APPENDIX F
Glossary of Readability Measures
81
Pred
icte
dA
utho
r(s)
Dat
eC
rite
rion
Var
iabl
esV
alue
Liv
ely
bl4
23!u
dged
dif
fi-
Wei
ghte
d m
edia
n in
dex
Rel
ativ
ePr
ess
ecu
ltyno
., ba
sed
on T
horn
dike
wor
d lis
tdi
ffic
ulty
.
Dol
ch19
2bG
rade
leve
las
sign
ed b
ypu
blis
hers
Five
indi
ces
of v
ocab
u-la
ry "
load
Was
hbur
ne19
.4:8
Mea
n re
sted
X2:
Num
ber
of d
iffe
rent
Rea
ding
Vog
elre
adin
g le
vel
wor
ds in
IO
C°
wor
dssc
ore
on(W
inne
tka)
of th
ose
likin
gbo
okX
: N
umbe
r of
pre
posi
tions
3 in
IC
(( w
ords
Stan
ford
Ach
ieve
-m
ent T
est
(X1)
X4:
Num
ber
of w
ords
not
on T
horn
dike
list
of
,Ct(
wor
ds
X :
Num
ber
of s
impl
e5
sent
ence
s in
75
sent
ence
s
Lew
eren
z19
29O
rder
of
pre-
sent
atio
n on
Perc
enta
ge o
f w
ords
be-
Gra
de le
vel
ginn
ing
with
W, H
, B (
easy
)St
anfo
rd1,
and
E (
hard
)A
chie
vem
ent
Tes
t
Lew
eren
z19
301)
Rat
io o
f A
nglo
-Sax
on to
Gra
-.le
vel
1935
Gre
ek a
nd R
oman
wor
ds19
39(d
iffi
culty
)
;ohn
son
1930
Patty
&19
31Pa
inte
r
Gra
de le
vel
assi
gned
by
publ
ishe
rs
N a
ne (
relie
d(
valid
ity o
fT
horn
dike
wor
dlis
t)
2) R
atio
of
wor
ds in
"C
lark
'sfi
rst 5
06"
to to
tal d
iffe
rent
wor
ds (
dive
rsity
)
3) E
stim
ate
of im
age
bear
ing
or s
enso
ry w
. -as
(in
tere
st)
4) P
olys
ylla
bic
wor
d co
unt
5) V
ocab
ular
y m
ass
(num
l.er
of d
iffe
rent
wor
ds in
sam
-pl
e th
at a
re n
ot o
n C
lark
'slis
t of
50C
com
mon
wor
ds
Perc
enta
ge o
f po
lysy
llabl
eG
rade
leve
lw
ords
1) T
horn
dike
wor
d lis
t ind
ex R
elat
ive
num
bers
diff
icul
ty
2) N
umbe
r of
dif
fere
nt w
ords
in s
ampl
e
Form
ula
Wei
ghte
d m
edia
n in
dex
num
-be
r
Bas
e ju
dgm
ent o
f di
ffic
ulty
on "
seve
ral
of v
ocab
ular
yfa
ctor
s
X1=
.085
X2
+ .1
0X3
+ .6
04X
4 -
.411
X5
+ 1
7.43
Mea
n of
tabl
ed v
alue
s fo
r ea
chof
var
iabl
es
Mea
n of
tabl
ed v
alue
s fo
r ea
chof
var
iabl
es
Tab
led
valu
e of
per
cent
age
ofpo
lysy
llabl
es
Inde
x N
umbe
r=A
.
A. W
. W. V
. = m
ean
Tho
rndi
keIn
dex
Num
ber
R =
num
ber
of d
iffe
rent
wor
dsin
sam
ple
Typ
e of
Ra.
ige
ofM
ater
ial
Mat
eria
lSt
udie
dSt
udie
d
Met
hod
of c
om-
pari
son
with
crite
rion
Scho
ol,
Gra
de 2
-In
spec
tion
gene
ral,
colle
gesc
ient
ific
Rea
ding
Prim
er-
Insp
ectio
nte
xts
grad
e 4
Chi
ldre
n's
libra
rybo
oks
Gra
des
3-9
Cor
rela
tion
.845
(Thi
s va
lue
is c
on-
foun
ded
by e
ffec
tsof
pop
ular
ity o
fbo
oks)
Stan
ford
Gra
de 2
-G
raph
icA
chie
ve-
colle
gem
eat T
est
para
grap
hs
Wid
eva
riet
y
Scho
olte
xts
Prim
er to
Insp
ectio
ngr
ade
8
Gra
des
9-12
Aut
hor(
s)D
ate
Cri
teri
onV
aria
bles
Pred
icte
dV
alue
Form
ula
Typ
e of
Mat
eria
lSt
udie
d
Ran
ge o
fM
ater
ial
Stud
ied
Met
hod
of c
om-
pari
son
with
crite
rion
Oie
man
n19
34X
CSO
Dal
e19
34T
yler
McC
lusk
y19
34
Tho
rndi
ke19
34
Gra
y an
d19
35L
eary
Was
hbur
ne &
193
8M
orph
ett (
re-
vis:
on o
f V
ogel
& W
ashb
urne
,19
28)
Com
pre-
hens
ion
scor
eof
adu
lts o
flo
w r
eadi
ngle
vel
Ren
ding
spee
d, c
om-
preh
ensi
onsc
ore
Voc
abul
ary
diff
icul
ty
Mea
n co
m-
preh
ensi
onte
st s
core
of
adul
ts
Mea
n te
sted
read
ing
leve
lcf
thos
e lik
ing
book
. Als
ote
ache
r ju
dg-
men
t at g
rade
s1-
2.
6 m
easu
res
of v
ocab
ular
ydi
ffic
ulty
, 8 m
easu
res
of s
truc
ture
, 4 q
ualit
ativ
efa
ctor
s
X2
Num
ber
of d
iffe
rent
tech
nica
l ter
ms
X :
Num
ber
of d
iffe
rent
3 ha
rdno
n-te
chni
cal
wor
ds
X :
Num
ber
of in
dete
rmi-
4 na
te c
laus
es
Voc
abul
ary,
sen
tenc
est
ruct
ure
Num
ber
of w
ords
not
on
Tho
rndi
ke li
st in
10,
000
wor
d sa
mpl
e
X2:
Num
ber
of d
iffe
rent
wor
ds n
ot o
n D
ale
Lis
tof
769
XS:
Num
ber
of p
erso
nal
X6:
Mea
n se
nten
ce le
ngth
in w
ords
X7:
Per
cent
dif
fere
nt w
ords
X8:
Num
ber
of p
repo
sitio
nal
X2:
Num
ber
of d
iffe
rent
X3:
X4:
pron
ouns
phra
ses
wor
ds p
er 1
000
wor
ds
Num
ber
of d
iffe
rent
wor
ds p
er 1
000
not i
nT
horn
dike
's m
ost.
com
mon
150
0 w
ords
Num
ber
of s
impl
ese
nten
ces
in 7
5 se
nten
-ce
s
Perc
ent o
fad
ults
of
low
RG
L(3
-5)
who
will
und
er-
stan
d pa
ssag
e(X
1)
Gra
de le
vel
Mea
n co
m-
preh
ensi
onte
st s
core
of
adul
ts (
X1)
Gra
de le
vel
(X!)
16 p
assa
ges
are
pres
ente
d in
ord
er A
dult
of d
iffi
culty
, to
be c
ompa
red
educ
atio
nw
ith m
ater
ial t
o be
test
ed
X1
= -
9.4X
2-
. 4X
3 +
2.2
X4
+ 1
14. 4
1-
9.0
No
form
ula.
Des
crip
tion
ofvo
cabu
lary
and
sen
tenc
e st
ruc-
ture
cha
ract
eris
tics
in e
asy
and
hard
mat
eria
ls.
Tab
led
valu
e of
num
ber
of w
ords
not o
n T
horn
dike
list
.
Xi =
.010
29X
2 +
.009
012X
5-
.020
94X
6 -
.033
13X
7-
.014
85X
8 +
3.7
74
X1
=.0
0255
X2
+ .0
458X
3 -
. 030
7X4
+ 1
.294
Gra
de 6
-co
llege
Adu
lt he
alth
Bel
owed
ucat
ion
grad
e 8
mat
eria
l
Qua
ntita
tive
fact
ors:
corr
elat
ion.
Indi
vidu
alfa
ctor
s up
to .6
0Q
ualit
ativ
e fa
ctor
s:in
spec
tion.
Cor
rela
tion
.511
Wid
e va
riet
y A
bove
Insp
ectio
ngr
ade
8
Gen
eral
fict
ion
and
non-
fict
ion
Gen
eral
`adu
lt
Gra
de 4
-9
Gra
de 2
-C
3rre
latio
n .6
435
colle
ge
Chi
ldre
n's
Gra
des
1-9
Cor
rela
tion
. 86
libra
ry b
ooks
(Thi
s va
lue
is c
on-
foun
ded
by e
ffec
tsof
pop
ular
ity o
fbo
oks)
Aut
hor(
s)D
ate
Cri
teri
onV
aria
bles
Pred
icte
dV
alue
Form
ula
Typ
e of
Ran
ge o
fM
ater
ial
Mat
eria
lSt
udie
dSt
udie
d
Met
hod
of c
om-
pari
son
with
crite
rion
Ston
e19
38
De
Lon
g19
38
Lor
ge19
39M
cCal
l-19
44C
rabb
s te
st19
48no
rms
Mor
ris
ti19
38H
al v
erso
n
Yoa
kam
1939
Kes
sler
Fles
ch
1941
Judg
ed d
if-
fi c
ulty
1943
Judg
men
t and
McC
all-
Cra
bbs
test
norm
s
Rat
io o
f ne
w w
ords
to to
tal
wor
ds, p
erce
nt s
ente
nces
com
plet
e on
one
line
Perc
enta
ge o
f w
ds
in v
ar-
ious
leve
ls o
f St
one'
s w
ord
list.
X2:
Per
cent
of
wor
ds o
ut-
side
Dal
e lis
t of
769
wor
ds
X6:
Mea
n se
nten
ce le
ngth
X: P
erce
nt p
repo
sitio
nal
8 ph
rase
s
a Q
ualit
ativ
e vo
cabu
lary
c la
sses
:I)
wor
ds le
arne
d ea
rly
in li
fe
II)
loca
lism
s
III)
con
cret
e w
ords
IV)
abst
ract
wor
ds
Voc
abul
ary
inde
x: b
ased
on
Tho
rndi
ke w
ord
list.
Wor
dsb.
.twee
n 4t
h an
d 20
th th
ou-
sand
,in
dex
is th
e nu
mbe
rof
thou
sand
in w
hich
wor
dap
pear
s. A
bove
20t
h th
ou-
sand
inde
x =
20.
Mea
n se
nten
ce le
ngth
inw
ords
Mea
n nu
mbe
r of
dif
fere
ntha
rd w
ords
per
10G
wor
ds
Xs:
Mea
n se
nten
ce le
ngth
X: N
umbe
r of
aff
ixes
per
10G
wor
ds
Xh:
Num
ber
of p
erso
nal
refe
renc
es p
er 1
0Gw
ords
Rel
ativ
edi
ffic
ulty
Six
leve
lsof
rel
ativ
edi
ffic
ulty
Rea
ding
grad
e le
vel
need
ed to
answ
er 5
0%of
que
stio
nsco
rrec
tly(C
so)
Rel
ativ
edi
ffic
ulty
Rel
ativ
e di
ffic
ulty
bas
ed o
nva
riab
les
stat
ed.
Tab
led
valu
e of
per
cent
ages
of w
ords
at v
ario
us le
vels
of S
tone
's w
ord
list
cso
-=1
OX
2 +
.06X
6 +
.10X
8
+ 1
.99
Tab
led
norm
s re
late
cou
nt o
fw
oods
in 4
cla
sses
to d
iffi
culty
Rea
ding
Gra
de 1
text
s
Rea
ding
Pre-
pri-
text
sm
er to
grad
e 2
McC
all-
Gra
des
Cra
bbs
3-12
test
pas
-sa
ges
Gra
de le
vel
Mea
n in
dex
num
ber
com
pari
son
Scho
olw
ith ta
bula
ted
valu
este
xts
Gra
de le
vel
Fact
ors
com
pare
d to
Gra
y an
dL
eary
sta
ndar
ds in
depe
nden
tly
Rea
ding
C75
= .1
338X
s +
.064
5Xm
-gr
ade
leve
lne
eded
to. 0
659X
h +
4.2
498
answ
er 7
5%of
que
stio
nsL
ater
cor
rect
ed to
;co
rrec
tlyC
= .0
7Xm
+ .0
7X -
.05X
(C75
)(c
orre
c-/5
sh
tion
for
high
+ 3
.27
grad
e le
vels
)
Cor
rela
tion
.67
Mul
tiple
cor
rela
-tio
n of
.74
was
fou
ndbe
twee
n cl
asse
s I,
III,
IV
, and
McC
all-
Cra
bbs
test
nor
ms
( L
orge
, 193
9)
2nd
-14t
hIn
spec
tion
grad
e
35 te
xt-
Gra
de 2
-In
spec
tion
book
sco
llege
Adu
lt m
ag-
Gra
des
3-az
ines
and
12M
cCal
l-C
rabb
s pa
s+sa
ges
Cor
rela
tion
.74
Aut
hor(
s)D
ate
Cri
teri
on
Fles
ch(r
evis
ion)
1948
Judg
men
t and
McC
all-
Cra
bbs
test
norm
s
Dal
e &
1948
McC
all-
Cha
llC
rabb
s te
stno
rms,
com
-pr
ehen
sion
test
sco
res
Dol
ch19
48G
rade
leve
las
sign
ed b
ypu
blis
hers
CO
c&
Whe
eler
1948
Non
e (r
elie
don
val
idity
of
Tho
rndi
kew
ord
list)
Fles
ch19
50C
ompr
e-he
nsio
n te
stsc
ores
Farr
,19
51Fl
esch
rea
ding
;enk
ins,
Fea
se s
core
Patte
rson
Gun
ning
1952
Fles
ch s
core
s,M
cCal
l-C
rabb
sno
rms,
and
judg
men
t
Var
iabl
es
wl
Mea
n w
ord
leng
th (
syl-
labl
es p
er 1
00 w
ords
)
sl =
Mea
n se
nten
ce le
ngth
pw =
Num
ber
of p
erso
nal
wor
ds p
er 1
00 w
ords
ps =
Num
ber
of p
erso
nal
sent
ence
s pe
r 10
Cse
nten
ces
X1:
Perc
ent o
f w
ords
not
on D
ale
list o
f 30
06co
mm
on w
ords
X2:
Mea
n se
nten
ce le
ngth
in w
ords
X1:
Med
ian
sent
ence
leng
th
X2:
90t
h pe
rcen
tile
sent
ence
leng
thX
3: P
erce
nt h
ard
wor
ds (
not
on D
olch
list
of
100C
wor
:Ls)
Perc
ent o
f w
ords
in e
ach
thou
sand
of
Tho
rndi
ke20
,000
wor
d lis
t
wl:
Wor
d le
ngth
(sy
llabl
espe
r 10
0 w
ords
)dw
: per
cent
age
of "
defi
nite
wor
ds."
nosw
: Num
ber
of o
ne-s
yl-
labl
e w
ords
Mea
n se
nten
ce le
ngth
Mea
n se
nten
ce le
ngth
, per
-ce
nt o
f w
ords
of
3 or
mor
esy
llabl
es
Pred
icte
dV
alue
Form
ula
Rel
ativ
eR
eadi
ng E
ase
2(.6
.835
-84
6w1
diff
icul
ty,
1.01
5si
rela
tive
inte
rest
Hum
an I
nter
est =
3.6
35+
. 31
4pw
pS
Typ
e of
Mat
eria
lSt
udie
d
McC
al 1
-C
rabb
s pa
s-sa
ges
Rea
ding
C50
= .1
579X
1 +
.049
6X2
+gr
ade
leve
lne
eded
to3.
6365
answ
er 5
0%of
que
stio
nsco
rrec
tly (
Cso
)
Gra
de le
vel
Loo
k up
thre
e va
riab
les
in ta
bles
,av
erag
e th
e ta
bled
val
ues.
Inst
ruct
iona
l, T
abul
ate
wor
ds in
pass
ages
by
Inde
pend
ent,
Tho
rndi
ke th
ousa
nds.
For
in-
and
aver
age
stru
ctio
nal l
evel
, 90%
of
wor
dsgr
ade
leve
lm
ust b
e be
low
stu
dent
's R
GL
.Fo
r u:
depe
nden
t lev
el 9
5%.
R:
Lev
el o
f R
= 1
68.0
95 +
.532
dw-
.811
w1
abst
ract
ion"
on a
rbitr
ary
scal
e
Rel
ativ
eN
ew R
eadi
ng E
ase
Inde
x =
diff
icul
ty1.
599
- 1.
015s
1 -
31.5
17
Fog
Inde
xFo
g In
dex
= .4
x (
mea
nse
n-(t
he r
eadi
ngte
nce
leng
th +
per
cent
of
grad
e le
vel
wor
ds o
f 3
or m
ore
sylla
bles
)ne
eded
tore
ad a
nd u
nder
-st
and
mat
eria
l)
Ran
ge o
fM
ater
ial
Stud
ied
Gra
des
3-12
Met
hod
of c
orn-
pari
son
with
crite
rion
McC
all-
Gra
des
3-C
rabb
s pa
s-12
, 3-1
6sa
ges,
sch
ool f
or h
ealth
mat
eria
l,pa
ssag
eshe
alth
art
i-cl
es
Rea
ding
text
s
Scho
ol te
xts
Adu
lt, p
ub-
lic r
elat
ions
Gen
eral
adul
t and
scho
ol
Cor
rela
tion
R.E
.: .7
0H
. L :
. 43
Cor
rela
tion
.70
Gra
des
1-In
spec
tion
6 Gra
des
3-C
orre
latio
n .7
212 Fu
ll ra
nge
Cor
rela
tion
.95
of F
lesc
h"r
eadi
ngea
se"
valu
es
Gra
des
6-In
spec
tion
12
Aut
hor(
s)D
ate
Cri
teri
onV
aria
bles
Pred
icte
dV
alue
Form
ula
Typ
e of
Mat
eria
lSt
udie
d
Ran
ge o
fM
etho
d of
coi
n-M
ater
ial
pari
son
with
Stud
ied
crite
rion
McE
lroy
1953
(AF
Man
ual
11-3
)
Forb
es &
1953
Mea
n va
lue
Cot
tleof
fiv
e ex
ist-
ing
read
abili
-ty
for
mul
as
Spac
he19
53G
rade
leve
las
sign
ed b
publ
ish-
trs
Fles
ch19
54O
bser
ved
char
acte
rist
ics
of e
asy
and
hard
mat
eria
l
W:e
eler
&19
54G
rade
leve
lSm
ithas
sign
ed b
ypu
blis
hers
ribe
1956
McC
all-
Cra
bbs
test
scor
es (
re-
vise
d fo
rm)
Gil
lie19
57Fl
esch
"le
vel
of a
bstr
actio
n '
scor
e.
Eas
y w
ords
(pr
onou
nced
in1-
2 w
ords
), a
ssig
ned
valu
e 1
hard
wor
ds (
pron
ounc
ed in
3or
mor
e sy
llabl
es),
ass
igne
dva
lue
of 3
Fog
Cou
ntFo
g C
ount
per
sen
tenc
e =
sum
of
(arb
itrar
yl's
and
3's
in s
ente
nce
valu
e) a
ndR
GL
= s
um o
f l's
and
Ts
divi
ded
read
ing
grad
e by
num
ber
of s
ente
nces
.If
val
uele
vel
is o
ver
20, d
ivid
e by
2.If
bel
ow20
, sub
trac
t 2, t
hen
divi
de b
y2.
Tho
rndi
ke v
ocab
u-R
eadi
ngL
ook
up m
ean
voca
bula
ry in
dex
Stan
dard
-la
ry d
iffi
culty
inde
x, w
ords
in 4
th th
ousa
nd a
nd a
bove
.gr
ade
leve
lin
tabl
es to
obt
ain
RG
Liz
ed te
sts
X1:
Mea
n se
nten
ce le
ngth
Gra
de le
vel
Gra
de le
vel =
.141
X +
.086
X2
1G
rade
scho
olX
: Pe
rcen
tage
of
wor
ds n
ot+
.839
test
s2
on D
ale
list o
f 76
9 ea
syw
ords
- N
umbe
r of
ref
eren
ces
toA
rbitr
ary
hum
an }
..z.lu
gs, e
tc. ,
per
100
sca
les
ofw
ords
real
ism
("r
")-
Num
ber
of in
dica
tions
of
and
ener
gyvo
ice
com
mun
icat
ion
per
(" e
")10
0 w
ords
.
Mea
n le
ngth
of
"uni
ts"
(sim
- R
eadi
nglia
r to
sen
tenc
es),
per
cent
age
grad
eof
mul
ti-sy
llabl
e w
ords
.le
vel
X :
Mea
n se
nten
ce. l
engt
hR
eadi
nggr
ade
leve
lne
eded
toX
5: P
erce
ntag
e of
wor
ds n
ot a
nsw
er 5
0%on
Rin
slan
d w
ord
list,
mul
tiplie
d by
100
.
1
of q
uest
ions
corr
ectly
(C
so
"r"
scor
e: ta
bled
val
ue c
orre
spon
d-in
g to
obs
erve
d nu
mbe
r of
ref
eren
c-es
to h
uman
bei
ngs,
etc
."e
" sc
ore:
tabl
ed v
alue
cor
resp
ond-
ing
to o
bser
ved
num
ber
of in
dica
-tio
ns o
f vo
ice
com
mun
icat
ion
Gen
eral
adul
tm
ater
ial
Inde
x =
10
('neo
n le
ngth
of
units
x S
choo
lpe
rces
ai. z
rrul
ti-sy
llabl
e w
ords
).te
xts
Loo
k up
lade
.in
tabl
e to
obt
ain
RG
L
Gra
de 5
-C
orre
latio
n .9
6co
llege
Gra
des
1-C
orre
latio
n .8
23 A
dult
Insp
ectio
n
Gra
des
1-In
spec
tion
4
C50
= 0
719X
1+
.104
3X + 5
McC
all-
Gra
des
3-C
rabb
s pa
s-12
2.93
47sa
ges,
195
0-C
orre
ctio
n is
then
app
lied
thro
ugh
revi
sion
refe
renc
e to
tabl
e of
Dal
e an
d)
Cha
ll (1
948)
Xi:
Num
ber
of f
inite
ver
bsA
bstr
actio
nA
bstr
actio
n le
vel =
36
+ X
2 +
Xpe
r 20
0 w
ords
leve
l (ar
-bi
trar
y sc
ale)
X2:
Num
ber
of d
efin
ite a
rti-
cles
with
nou
ns p
er 2
00w
ords
X,:
Num
ber
of n
ouns
of
3 ab
stra
ctio
n pe
r 20
0 w
ords
.
- (2
X3)
Scho
olG
rade
4-
Cor
rela
tion
.83
text
s,co
llege
gene
ral
adul
t ma-
teri
al
Aut
hor(
s)D
ate
Cri
teri
on
Col
eman
1965
Clo
ze s
core
(des
crib
edby
Sza
lay
1965
)
Smith
&Se
nter
( A
RI)
Fry
Var
iabl
es
1966
Gra
de le
vel
assi
gned
by
publ
ishe
rs
1968
Gra
de le
vel
assi
gned
by
publ
ishe
rs
Met
.a.u
4;11
11a
1969
McC
all-
Cra
bbs
test
.(1
00%
sco
reeq
uire
d) (
1961
evis
ion)
Cok
eR
othk
opf
1970
Fles
ch"R
eadi
ng E
ase"
scor
es
Cay
lor,
1972
Clo
ze s
core
Stic
ht, F
ox,
Ford
X1:
per
cent
age
of o
ne-
sylla
ble
wor
ds
X :
Num
ber
of s
ente
nces
2pe
; 100
wor
ds
X3:
Num
ber
of p
rono
uns
per
100
wor
ds
X :
Num
ber
of p
repo
si-
4 tio
ns (
not i
nclu
ding
"to"
) pe
r 10
0 w
ords
.
w /s
: Mea
n se
nten
ce le
ngth
in w
ords
s/w
: Mea
n w
ord
leng
th in
stro
kes
(let
ters
)
Mea
n nu
mbe
r of
sen
tenc
espe
r 10
0 w
ords
Mea
n nu
mbe
r of
syl
labl
espe
r 10
0 w
ords
Pred
icte
dV
alue
1X
: C
loze
scor
e
Arb
itrar
yva
lue
and
grad
e le
vel
Gra
dele
vel
Num
ber
of w
ords
of
thre
eG
rade
leve
lor
mor
e w
ords
in 3
0 se
nten
ces
- W
ords
per
sen
tenc
e-
Vow
els
per
wor
d (a
n in
-dc
x of
syl
labl
es p
er w
ord)
Num
ber
of o
ne-s
ylla
ble
wor
ds p
er 1
50 w
ord
sam
pl:.
Fles
ch"r
eadi
ngea
se"
scor
es
Rea
ding
grad
e le
vel
requ
ired
for
35%
clo
zesc
are
Form
ula
a)X
1=
- 38
.45
4 1.
29X
1
b) X
1= -
37.9
51.
16X
1 4
1.48
X2
c) X
1= -
34.0
2 +
1.0
7X1
+ 1
.18X
2
+ .7
6X3
d) X
1= -
26.0
1 +
1.0
4X1
+ 1
.06X
2
+ .
56X
3 -
. 36X
4
Aut
omat
ed R
eada
bilit
y In
dex
(w /s
) +
9(s
/w)
Gra
de L
evel
= 0
.50(
w /s
) +
4.71
(s/w
) -2
1.43
Plot
obs
erve
d va
lues
of
the
stat
ed v
aria
bles
on
the
grap
hsu
pplie
d. T
hen
read
off
gra
dele
vel.
SMO
G G
rade
= 3
+ s
quar
e ro
ot o
fpo
lysy
llabl
e co
unt
Not
sta
ted.
Con
tain
ed in
com
pu-
ter
prog
ram
.
Typ
e of
Mat
eria
lSt
udie
d
Not
sta
ted
Scho
olte
xts
Scho
olte
xts
Ran
ge o
fM
ater
ial
Stud
ied
Met
hod
of c
om-
pari
son
with
crite
rion
McC
all-
Cra
bbs
pas-
sage
s
Gen
eral
adul
t and
McC
all-
Cra
bbs
pas-
sage
s
RG
L =
20
-of
wor
dsA
rmy
10te
chni
cal
liter
atur
e
Gra
de 2
-pr
ofes
sion
al
Gra
des
1-7 G
rade
s 1-
colle
ge
Gra
des
1-pr
ofes
sion
-al N
ot s
tate
d.Pr
obab
lylo
w g
rade
sth
roug
had
ult.
Gra
dele
vels
6-13
Cor
rela
tion
a) .8
6b)
. 88
c) .
d) .
90
Cor
rela
tion
(For
gra
dele
vel o
nly)
.98
Insp
ectio
n.C
orre
latio
nsw
ith o
ther
form
ulas
and
stud
ent c
om-
preh
ensi
on te
stsc
ore
rang
edfr
om .7
8 -
. 98
Cor
rela
tion
appr
oxim
atel
y .7
Cor
rela
tion
.M
Cor
rela
tion
.77
