Food-Based Validation of a Dietary Questionnaire: The Effects of Week-to-Week Variation in Food Consumption

International Journal of Epidemiology©International Epidemiological Association 1989

Vol. 18, No. 4Printed in Great Britain

Food-Based Validation of a DietaryQuestionnaire: The Effects of Week-to-Week Variation in FoodConsumptionSIMONETTA SALVINI,** DAVID J HUNTER,*t LAURA SAMPSON/ MEIR J STAMPFER/ GRAHAM ACOLDITZ,« BERNARD ROSNER'" AND WALTER C WlLLETT,«t

Salvini S (Channing Laboratory, Harvard Medical School, Boston, MA 02115, USA) Hunter D J , Sampson L, StampferM J, Coldrtz G A, Rosner B and Willett W C. Food-based validation of a food frequency questionnaire: the effects ofweek-to-week variation in food consumption. International Journal of Epidemiology 1989; 18: 858-867.The reproducibility and validity of responses for 55 specific foods and beverages on a self-administered foodfrequency questionnaire were evaluated. One hundred and seventy three women from the Nurses' Health Studycompleted the questionnaire twice approximately 12 months apart and also recorded their food consumption forseven consecutive days, four times during the one-year interval. For the 55 foods, the mean of correlation coefficientsbetween frequencies of intake for first versus second questionnaire was 0.57 (range = 0.24 for fruit punch to 0.93 forbeer). The mean of correlation coefficients between the dietary records and first questionnaire was 0.44 (range = 0.09for yellow squash to 0.83 for beer and tea) and between the dietary records and the second questionnaire was 0.52(range = 0.08 for spinach to 0.90 for tea). Ratios of within- to between-person variance for the 55 foods were computedusing the mean four one-week dietary records for each person as replicate measurements. For most foods this ratiowas greater than 1.0 (geometric mean of ratios = 1.88), ranging from 0.25 (skimmed milk) to 14.76 (spinach).Correlation coefficients comparing questionnaire and dietary record for the 55 foods were corrected for the within-person variation (mean corrected value = 0.55 for dietary record versus first questionnaire and 0.66 versus thesecond). Mean daily amounts of each food calculated by the questionnaire and by the dietary record were alsocompared; the observed differences suggested that responses to the questionnaire tended to over-represent sociallydesirable foods. This analysis documents the validity and reproducibility of the questionnaire for measuring specificfoods and beverages, as well as the large within-person variation for food intake measured by dietary records.Differences in the degree of validity for specific foods revealed in this type of analysis can be useful in improvingquestionnaire design and in interpreting findings from epidemiological studies that use the instrument.

The importance of collecting dietary information inepidemiological studies of chronic disease, has beenincreasingly recognized.1"6 To study large numbers ofindividuals, dietary assessment methods must be rela-tively simple and inexpensive. Hence, manyresearchers have used self-administered food fre-quency questionnaires which do not require trained

T h e Charming Laboratory, Department of Medicine, HarvardMedical School and Brigham and Women's Hospital, Boston, MA,USA."Department of Preventive Medicine and Clinical Epidemiology,Harvard Medical School, Boston, Boston, MA, USA.tDepartments of Epidemiology and Nutrition, Harvard School ofPublic Health, Boston, MA, USA.tlstiruto Nazionale dei Tumori (National Cancer Institute), Milan,Italy.

Reprint requests: Dr. Walter Willett, Channing Laboratory, 180Longwood Avenue, Boston, MA 02115, USA.

interviewers, thus reducing costs and making large-scale studies feasible. Such questionnaires should beevaluated for reproducibility and validity,reproducibility referring to the ability of the instru-ment to provide similar measurements on repetition,and validity to the ability of the instrument to recordwhat it is intended to measure. To assess validity, adifferent, and presumably superior, method of mea-suring diet is used for comparison.7"10 Validationstudies of food frequency questionnaires generallyhave shown this tool to be reliable and valid for rankingthe nutrient intakes of individuals,7 although therehave been exceptions." The reproducibility of ques-tions regarding single foods has been examined inseveral studies,12"16 but the validity of such questionshas been evaluated in only one investigation thatassessed recall over a one-week period.17

Epidemiological interest may focus on individual

858

at UB

Kaiserslautern on M

ay 11, 2014http://ije.oxfordjournals.org/

Dow

nloaded from

http://ije.oxfordjournals.org/

VALIDATION OF A DIETARY QUESTIONNAIRE 859

foods both to identify those contributing most tospecific nutrients, and also to identify associationsbetween individual foods and disease.13'18"26 It is there-fore desirable to have information about the perform-ance of a food frequency questionnaire at the level ofindividual foods. In particular, information aboutsingle items may identify foods for which a frequencyquestionnaire performs poorly, and for which changesin the formulation of the questions may improvevalidity.

The role of within- and between-person variation innutrient consumption has been studied extensively.27"33

Day-to-day variation in both the frequency of consum-ing foods and in the portion sizes of foods consumed byfree-living subjects34 can greatly affect the validity ofdietary records or short-term recalls as measures oflong-term intake. These sources of variation render asingle or a few days of diet-recording inadequate as anestimate of usual diet and attenuate associations withother measures. Quantification of the within- andbetween-person components of variation in nutrientintake assessed by dietary records can be used to cor-rect observed correlation coefficients between suchmeasurements and other variables.35

The availability of multiple one-week dietaryrecords for 173 women allowed us to compare foodintake assessed by these records with the responses to asemi-quantitative food frequency questionnaire. Wewere particularly interested in:1. The reproducibility of food intake measured by two

questionnaires administered 12 months apart.2. The validity of the food frequency questionnaire for

individual foods, as compared with 28 days of die-tary records.

3. The influence of within-person variation in foodconsumption determined by dietary records on thecorrelations between dietary records andquestionnaires.

MATERIALS AND METHODSStudy PopulationThe study population consisted of 173 members of theNurses' Health Study cohort who participated in apreviously reported dietary questionnaire validationstudy.9 All women were 34 to 59 years of age in 1980and had responded to a mailed semi-quantitative foodfrequency questionnaire.

The Food Frequency QuestionnaireThe self-administered semi-quantitative food fre-quency questionnaire used in 1980 included a list of 61foods. For each food a standard serving size was spec-ified; 'natural' portion sizes (eg slices of bread or

bacon, one fruit) were used whenever possible, other-wise an amount suggested by the nutritionists' experi-ence was specified (eg 6-8 oz of meat or fish). Subjectscould choose from nine frequency categories of con-sumption, as shown in Figure 1. Daily amounts of eachfood were calculated by multiplying the specified por-tion unit by the frequency of use, using the followingvalues for reported frequencies: never or less thanonce a month = 0, 1-3 per month = 0.07, one perweek = 0.14, 2-4 per week = 0.43, 5-6 perweek = 0.80, one per day = one, 2-3 per day = 2.5,4-6 per day = 4.5, six or more per day = 6.0. The samequestionnaire was administered a second time approxi-mately one year later at the end of either the third orfourth week of dietary recording (see below).

Dietary RecordsBetween June 1980 and June 1981, participants com-pleted a seven-day dietary record four times atintervals of approximately three months. Women wereprovided with a Pelouze dietetic scale for weighingfood servings. Alternative methods employing com-mon household measures were taught for use whenaway from home. All food and beverages consumedfor a one-week period were recorded in a speciallydesigned booklet. Participants provided detaileddescriptions of each food, including brand and methodof preparation and recipes whenever possible. Therecords were then coded by trained dietitians andreviewed by LS to minimize variability of interpreta-tion. Participants were telephoned to resolve any dis-crepancies or to obtain further information whennecessary.

The focus of this analysis was to compare the twoquestionnaires and the diet records of each subjectwith respect to single foods or food groups. Hence, weanalysed the diet records as follows:1. To correspond with questions on the food fre-

quency questionnaire, codes representing variationof the same food were grouped together. The foodson the questionnaire corresponded to approxi-mately 840 codes used for the dietary records, ie amean of 15 different codes for each food on thequestionnaire, ranging from one (bananas) to 146codes (meat). For example, 'skimmed or low fatmilk' had ten different codes on the diet records,representing different percentages of fat thatranged from 0 to 2 per cent and differences invitamin fortification or addition of milk solids.Likewise, 'cold breakfast cereal' was representedby 54 different codes for specific brands consumedduring the 28 days. These codes were collapsed intotwo groups corresponding to the questionnaire

at UB

Kaiserslautern on M


Dow

nloaded from


DIET ASSESSMENT

II"a.

For each food listed, check the box Indicating how often, on average, you have used the amount specified during the past year. If yourintake of a food item has greatly Increased or decreased during the past 10 years, indicate this in the last 2 columns.

FOOD AND AMOUNTS

Oalry FoodsSkim or low fat milk (8 oz. glasses)

Whole milk (8 oz. glasses)

Yoghurt. (1 c.)

Ice cream (ftc.)Cottage cheese (Vic)

Hard cheese, plain or as part ofa dish (slice or servings)

Margarine (pats added to food or bread)

Butter (pats added to tood or bread)

Avera6+perday

i

i

i

i

i

i

i

i

4-6perday

7

7

7

7

7

7

7

7

2-3perday

3

a

a

s

a

a

1perday

4

4

4

4

4

4

A

4

5-6per

week

s

5

S

s

s

«

S

ie use last year2-4per

week

fi

ft

fl

j_

_t

B

£

1per

week

7

r

7

7

7

7

7

7

1-3per

month

•

A

a

a

i

A

A

AlmostNever

g

9

9

a

A

s

4

A

(9)

(10)

(11)

(12)

(13)

(U)

(15)

(16)

My use during thepast 10 years has:

GreatlyIncreased

GreatlyDecreased

o>

(42)

(«)

(44)

(45)

(46)

(47)

(48)

(49)

at UB

Kaiserslautern on M


Dow

nloaded from



items, 'skimmed or low fat milk' and 'cold breakfastcereal'.

2. To permit comparison with the semi-quantitativefood frequency questionnaire, we converted theamounts reported in the dietary records into thesame standard portion size units used in thequestionnaire.

For each individual we computed both the meandaily amount of food consumed during each of the fourweeks and the mean daily amount for the full fourweeks of recording.

Two of the 61 foods listed in the food frequencyquestionnaire ('other fruits', 'home fried food'), havebeen omitted in the analysis because data from the dietrecords were not comparable. Data for 'low-caloriecarbonated drink' and 'artificial sweetener' were notavailable at the time of analysis. Furthermore, on thequestionnaire we asked about meat from mixed dishesseparately from meat from a main dish. Exactlyequivalent information was not available from the die-tary records, and thus the two questions for meat havebeen combined. The portion reported for meat as amixed dish has been divided by two, since the portionsize for this group is estimated to be half the portion ofmeat as a main dish. The food frequency questionnairealso included separate questions about home made andready made pie. These were combined by adding thetwo categories of pie consumption. As a result, weconsidered 55 foods in this analysis.

Mixed dishes from the dietary records were dividedinto their components and allocated to the appropriatecategories of the questionnaire as would routinely bedone in analysing complex dishes. For example, when'peas and carrots' was reported, we assigned half of theportion to carrots and half to peas. Analogous alloca-tions of components were used for other mixed dishesusing typical recipes.

Statistical AnalysisSpearman and Pearson product-moment correlationcoefficients were used to compare mean daily amountsmeasured by the dietary records and the first and thesecond questionnaires. Since Spearman correlationcoefficients were either very close to or slightly higherthan the equivalent Pearson correlation coefficients,only Pearson coefficients are presented.

Extreme outliers were excluded; in no instance didwe exclude more than five observations for any fooditem. Since the distribution of the individual meandaily amounts for most of the foods was log^-normal(skewed, long tail to the right), these were loge-trans-formed. For uncommonly eaten foods some subjectsreported no use, so that 1.0 was added to all values

before transformation; although the resulting distri-butions were not strictly normal, they did not includeserious outliers. Subjects leaving a blank answer for afood in either of the questionnaires were consideredmissing for that particular food item.

Within-person variation in food intake measured bythe dietary records can attenuate the correlationsbetween questionnaires and records.28"32-36 We there-fore calculated correlation coefficients corrected forwithin-person variation in dietary records using thefollowing formula.27"35

r, = r0 Vwhere:

r, = estimated true (corrected) correlation betweenthe average daily intake for an individual measuredby dietary records (x) and the average daily amountfor an individual measured by a single food fre-quency questionnaire (y), r0 = observed correla-tion coefficient between x (as estimated by fourone-week dietary records) and y, X, = (estimatedwithin-person variance in x) / (estimated between-person variance in x), k, = number of repeatedobservations of x (in this study k = 4).

For these corrections, variance components wereestimated using the mean daily portion of the fourseparate weeks for each individual and the StatisticalAnalysis System PROC VARCOMP procedure.36 Wedid not correct correlations for within-person variationin the questionnaire since the correlation between themean of many questionnaire administrations and die-tary record values is of no practical interest. The stan-dard errors for corrected correlation coefficients werecalculated using the formula given by Rosner andWillett.35

RESULTSCompared with the mean daily amounts of foods esti-mated from the diet records, most foods wereoverestimated on the food frequency questionnaires(Table 1). However, higher intakes were measured bydietary records for a group of animal products (wholemilk, butter, processed meats, eggs), and for coldbreakfast cereal. In most instances the mean dailyamount from the dietary records was closer to thesecond questionnaire, which referred to a period oftime corresponding to the dietary records, than to thefirst questionnaire.

The Pearson correlation coefficients between thedietary records and the first food frequency question-naire ranged from 0.09 for yellow squash to 0.83 forbeer and tea, with a mean value of 0.44. The correla-tions with the second questionnaire ranged from 0.08for spinach to 0.90 for tea (mean = 0.52).

at UB

Kaiserslautern on M


Dow

nloaded from


862 INTERNATIONAL JOURNAL EPIDEMIOLOGY

TABLE 1 Comparison of mean daily consumption of 55 food items estimated by dietary records (DR) and by food frequency questionnaires (Ql andQ2). Data were provided by 173 Boston Area female registered nurses from the Nurses' Health Study

Food

Dairy productsSkimmed milkWhole milkYoghurtIce creamCottage cheeseHard cheeseMargarineButter

Fruit and vegetablesApplesOrange*Orange or

grapefruit juicePeachesBananasString beamBroccoliCabbageCarTOtt

ComSpinachPeas/lima beansYellow squashSweet potatoesBeans/lentilsTomatoes

Meat, fish and eggsChicken without

skinChicken with skinHamburgersHot dogsProcessed meatsBaconMealFishEg*s

Sweets, baked goods,ChocolateSweeuPieCakeBiscuitsCold breakfast

cerealWhite breadDart bread

MiscellaneousPeanut butterCrispsChipsNutsPotatoesPasta/rice

BeveragesCoffeeTeaBeerWtoeSpiritsCoke/Pepsi

carbonatedbeverages

Fnnt punch

Portionunit

1 cup1 cup1 cupicupicup1 slicelpatlpat

1 fruit1 fruit

small glass1 fruit1 fruiticupicupicup1 raw or

icup ckdicupicupicupicupicupicup1 torn.

6-8 oz6-8 oz111 slice1 slice6-8 o i l6-8 ozi1

cereals1 oil1 oil1 slice1 slice1

Icup1 slice1 slice

ltbsl o z i4oz l1 oziicupicup

1 cup1 cupbcKtk/can1 glass1 drink1 glass

1 glass•glass

DRf

dailymean

OJ0 30.020 20 10 51.21.0

0 20 1

0.70 10 10.10101

0.20.10.10.10.040.0030.10.3

0.10.040.10.10.20.10.40.10.4

0.10.10.03010.6

0 40.90 5

0.10.10.030.10.20J

1.80.70.10J0J0.1

010.1

(SD)t

(0.4)(0.4)(0.04)(0.2)(0 1)(0.3)(1.1)(1.0)

(0.2)(0.1)

(0.6)(0.1)(0.2)(0.1)(0.1)(0.1)

(0 2)(0 1)(0 1)(0 1)(0 1)(0 01)(0.1)(0.1)

(0.1)(0.1)(0.1)(0.1)(0.2)(0.1)(0.2)(0.1)(0.2)

(0.1)(0.1)(0.1)(0.1)(0.7)

(0.6)(0.6)(0.4)

(0.2)(0 1)(0 04)(0 1)(0.1)(0.2)

(13)(0 9)(0 2)(0 5)(0.5)(0J)

(0 1)(0 1)

01*daily

mean

0.50.20.070 20 20 51.30.6

0.40.3

0 70 10 20.30.20 1

0.20.10.40.20.10 0 20.10.5

0 20.20.20.10.10.10.40 20.3

0.10.10.10 10.7

0.30.70.8

0.20.10.10.10.30.3

2.41.00.10.20.30.2

0.10.1

(SD)t

(0.8)(0.5)(0.1)(0.2)(0.2)(0.4)(1.3)(1.0)

(0.5)(0.4)

(0.6)(0.2)(0J)(0.2)(0.2)(0.2)

(0.2)(0.1)(0.5)(0.2)(0 1)(0 03)(0 1)(0 3)

(0 2)(0.2)(0.2)(0.1)(0 2)(0.1)(0.3)(0.2)(0.3)

(0.2)(0.2)(0.1)(0.1)(1.0)

(0.3)(0.9)(1.0)

(0.3)(0.2)(0.1)(0.2)(0.3)(0.2)

(1.8)(1.3)(0.2)(0.3)(0.6)(0.5)

(0.2)(0.2)

o:dally

mean

05020060.20.10.51.50 6

0.30.3

0.70.1020.2020.1

0.20.10.30.20.10.020.040 3

0.20.10.20 1010 10.40.20.3

0 20.1010.10.8

0.30.90.8

0.20.10.10.10.30.3

2.41.00.10 30.30 2

0.1004

(SD)t

(0.8)(0.4)(0 1)(0.2)(0 2)(0.4)(1.5)(1.1)

(0.4)(0.3)

(0.6)(0.2)(0.3)(0.2)(0.2)(0.1)

(0.2)(0.1)(0 3)(0.2)(0 1)(0 03)(0 1)(OJ)

(0.2)(0.1)(0.1)(0.1)(0.2)(0.1)(0.3)(0.2)(OJ)

(0 3)(0 2)(0 1)(0.1)(1 1)

(0 4)(1 1)(1.1)

(0.4)(0.2)(0.1)(0.2)(0J)(0.2)

(1.8)(12)(02)(0J)(0-5)(0.6)

(0.2)(0.1)

Correlation Coefficient^

DRvsQl

0.670 5 40.770.610.590.330.580.67

0 610.40

0.590.250.660.350.350.30

0.300.280.140.310 0 90 240220 2 9

0380.220.120.200.330.440 350 3 4064

0.310.460230250.48

0690 430.50

0.560.530.330.330.550.19

0.740.830.830.790.740.81

0J10.25

DRviO2

0.790.620.740.610.680.490.710.79

0.660.59

0.800.280.700.470.490.31

0.310.200.080.380.270.240.380.44

0500440.260 420.440.550.380.430.68

0 310.430 3 9038050

0 750 61066

0.670.590.360.440.520.27

0.750 9 00.870.850.800.81

0.280.41

QlvsQ2

0.750.510.720.770.610.470.700 74

0.660.58

0.680 510.640580 580.61

0.430.510.410580.420.58028037

0460 350380590 530600 450.460.70

0.370.510.530.560.60

0.710.610.57

0.570.630.530.450.640.43

0.800.860.930.760.740.85

0.500.24

«T0.250.322.431.731.561.380.570.64

1.862.31

0.38

1.053.233.988 74

2.77—

14 76299906

279686

3.172.854573.162282.391.937.671.08

2.853.409.165001.73

0.431 401.45

1.022.176.967.772.362.64

0.300.290.650.440.370.32

2.733.49

CorrectedCoeffick

DRvjQl

0.69±0.O40.56±0.O60.97±0.O70.73±0.O60 7O±0.060 38±0 080.62±0.O50.72±0.O5

0.74±0 070.5O±0.08

O.62±0.O5

0.74±0.O50.47±0.100.49±0.100 53±0 16

0 39±0 09—

0 30±0.190 4110 090 16±0 14

_0 29±0 100.4810.14

0.5110.090.2910.100.1810.010.2710.100.4110.090.5610 080.4310 080.5210 120.72±0 05

0 4110 090.6310.100.4210.110.3710.12O.5710.O7

0.7310.040.5010.07O.5810.O7

0.6310 060.6610.080.5510 140.5710.150.6910.080.2410.10

0.7710.040.8610.030.8910.030.8310.030.7710.040.8410.03

0.4010.090J410.10

:nt+SE//

DRvsO2

0 8110.030.6210.050.9410.070.7310.060.8010.060 5710.070 7610.040 8510.03

0.8010.060.7410.08

0 8410.03—

0 7910.050.6310.090.6910.100.5510.16

0.4010.09—

0.1710.180.5010.090.4910.16

—0.5010.090.73±0.15

0.6710.090 5810.090 3810.110 5610.090 5510.080.70±0.080.4610.080.6610.120 7710.05

0.4110.090 5910.100.7110.180.5710.120.6010.07

0 7910.040.7110.060.7710.06

0.7510.050.7310.070.6010 140 7510.170.6610.080.3510.09

0.7810.030.93±0.020.94±0.020.9010.020.8410.030.8410.03

0.3610.090.5610.10

* DR - dietary record, Ql - flm questionnaire, Q2 - second questionnairet Daily means and SD (standard deviaoons) computed from untransfonncd dita.t Pearson correlation coefficients computed from log,-transformed data to improve normality. Correlations over 0.15 are 'stansricalry significant' (p < 0.05).I Variance component! mrmlatrrl from loge-tramformed data,// Standard Error.— Dash indicates omitted ratios because of very large within-person variance. 1 one <a — 28 J 5 grams.

at UB

Kaiserslautern on M


Dow

nloaded from



Reproducibility correlations comparing the first andsecond questionnaires ranged from 0.24 for fruit punchto 0.93 for beer (mean = 0.57).

The ratio of the within- to between-person variationfor the mean daily amount of food consumed in each ofthe four weeks of dietary recording was almost alwaysgreater than 1.0. This ratio ranged from 0.25 for skim-med milk to 14.76 for spinach, with a geometric meanof 1.88. For peaches, corn, and sweet potatoes, thewithin-person variance was so large that the estimatedbetween-person variability becomes negative; theseratios were thus omitted.

The correlation coefficients between the dietaryrecords and the first food frequency questionnaire cor-rected for the within- versus between-person varianceranged from 0.16 for yellow squash to 0.97 for yoghurt(mean = 0.55). For comparisons with the second ques-tionnaire, the corrected correlation coefficients rangedfrom 0.17 for spinach to 0.94 for yoghurt and beer(mean = 0.66).

Foods consumed regularly (eg skimmed and wholemilk, margarine, butter, orange juice, cold breakfastcereal, coffee, tea and Coke/Pepsi) tended to have alower within-person week-to-week fluctuation in con-sumption. Uncorrected correlation coefficientsbetween diet assessment methods were generally highfor these foods and correction for within-person varia-tion improved the coefficients only slightly. Forexample, we observed that coffee consumption waswell estimated in this population by the food frequencyquestionnaires (r = 0.74 for dietary record versus firstfood frequency questionnaire and r = 0.75 for dietaryrecord versus second food frequency questionnaire).The ratio of within- to between-person variance forcoffee (0.30) indicates more variation in consumptionbetween individuals than within individuals, so thatcorrection of the correlation coefficients had only avery slight effect. In contrast, for foods consumed lessregularly the observed correlation coefficients weresubstantially attenuated by within-person variation,thus the corrected coefficients provide a more accurateestimate of the true correlation between the question-naire and long-term food intake. As an example,tomatoes had a ratio of within- versus between-personvariance of 6.86; with correction, the correlationcoefficient between dietary records and first question-naire improved from 0.28 to 0.48, and for the com-parison with the second food frequency questionnairefrom 0.44 to 0.73.

We also calculated the mean daily amounts of eachfood using data from the dietary records, for eachresponse category on the second food frequency ques-tionnaire. This analysis thus provides an assessment of

the actual values for the categories of the surrogatevariable (ie the food frequency questionnaire). Theresults for the five foods with highest and the five foodswith lowest corrected correlation coefficients betweenthe dietary records and the second food frequencyquestionnaire are presented in Table 2; observationsbased on fewer than five subjects are shown in paren-theses and will not be discussed.

For the foods with the highest corrected correlationcoefficients, actual intake from the dietary recordsincreased with higher frequency categories reportedon the questionnaire. For example, the mean tea con-sumption based on the diet records was 0.06 cups perday for those reporting no use on the questionnaire and2.95 cups per day for those reporting 4-6 cups per dayon the questionnaire. Although actual yoghurt intakealso clearly increased with higher reported frequency,the absolute values for this food were substantiallylower than those expected on the basis of question-naire responses.

The foods with the lowest corrected correlationcoefficients, tended to have less variation betweenindividuals; only three subjects ate any of these foodsmore often than once a day. These data provide anindication of the nature of the relatively low correla-tions. For example, low reported frequency of carrotintake corresponded to low actual intake, however,higher questionnaire reports of carrot intake representa substantial overestimation of actual intake. Never-theless, in spite of the low correlation for carrots, thequestionniare was able to identify a substantial propor-tion of subjects with more than a twofold difference inactual carrot intake. The questionnaire item on 'spin-ach and other greens' both overestimated actual intakeand provided no useful discrimination among subjects.

DISCUSSIONReliability or reproducibility is a minimal requirementfor any measurement instrument. In this study theaverage of correlation coefficients between repeatedquestionnaire determinations at an interval of aboutone year was 0.57. For 23% of the food items thecorrelation coefficient was greater or equal to 0.70, andfor 73% of foods the coefficient was greater or equal to0.50. This level of reproducibility is comparable to thatof many biological measurements (eg serum cho-lesterol and blood pressure level), that are strong pre-dictors of disease in epidemiological studies.

Nomura et aJP- assessed the reproducibility of foodintake responses to a food questionnaire. Kendall'scorrelation coefficients ranged from —0.08 to 0.71(mean = 0.30) for comparison of the original measurewith a questionnaire repeated after a six-month inter-

at UB

Kaiserslautern on M


Dow

nloaded from



TABLE 2 Actual mean daily intake from dietary records, by categories of food consumption reported in the second food frequency questionnaire.Data are shown for the five foods with highest and lowest corrected correlation coefficients between dietary records and second food frequencyquestionnaire. Data were provided by 173 Boston area female registered nurses from the Nurses' Health Study.

Food groups,portion unit

(corrected correlationcoefficient)

Expected values*

never

0.0

1-3month

0.07

1week

0.14

Frequency categories

2-4week

0.43

5-6 1week day

0.80 1.0

2-3day

2.5

4-6day

4.5

6+day

6.0

Highest correlation coefficientsYoghurt, 1 cup mean 0.004 0.03 0.05 0.14 (0.27)

(r = 0.94) n 99 38 18 7 (2)

Tea, 1 cup mean 0.06 0.21 0.20 0.40 0.46( r»0 .93) n 44 19 9 19 9

Wine, 1 glass mean 0.04 0.17 0.28 0.40 0.67(r = 0.90) n 48 40 18 40 9

Butter, 1 pat mean 0.35 0.70 0.95 0.92 1.36(r = 0.85) n 81 19 6 13 11

Coffee, 1 cup mean 0.24 (0.80) 1.68 1.09 1.10(r = 0.78) n 22 (3) (4) 11 5

Lowest correlation coefficientsCarrots, 1 or 1 cup mean 0.06 0.10 0.17 0.23 0.22

(r = 0.40) n 6 49 66 42 7

Hamburger, 1 mean (0.07) 0.09 0.15 0.18 (0.07)(r = 0.38) n (4) 46 92 29 (1)

Non-Cola carbonatedbeverages mean 0.05 0.09 0.13 0.18 (0.14)(r = 0.36) n 104 37 14 10 (4)

Pasta, i cup mean 0.18 0.25 0.29 0.38 (0.61)(r = 0.35) n 5 33 64 66 (2)

Spinach, other greens,Icup mean 0.01 0.07 0.07 0.10 0.05(r - 0.17) n 17 44 50 33 10

(0.26) (0.12) —(1)0.85281.2491.85140.9717

(0.16)(3)

(2)1.6033

2.077

2.4920

1.8569

2.958

(2.60)(1)2.7452.3926

(5.21)(1)

(4.01)(1)4.0916

(0.52)

(2)

0.0415

(0.74)(1)

(0.03)

(2)

* Expected values are the anticipated dietary record intakes (in terms of specified portion units) if questionnaire responses were completelyaccurate, based on one/day = 1.0.— Dash indicates no observations.

val (mean = 0.30) and from -0.05 to 0.54(mean = 0.27) for comparison with a questionnaireadministered two years later. Graham et aP reportedexact agreement ranging from 57.1% for coleslaw con-sumption to 98% for frequency of drinking vodka,when re-interviewing 99 subjects within 18 months.Acheson and Doll14 analysed responses from 63 sub-jects who were asked to describe their frequency of useof several foods and beverages as one of five frequencyoptions, and were re-interviewed after a three-monthperiod. Ninety per cent of responses were in the identi-cal or immediately adjacent category. Colditz et a/15

compared a questionnaire completed by 1497 womenin 1979 for the pilot phase of diet assessment in theNurses' Health Study with the 1980 version of thequestionnaire. Spearman correlation coefficientsbetween food frequency responses ranged from 0.34for ready made pie to 0.76 for tea. Focusing on the

reliability of retrospective dietary reports, Byers et al[6

found Pearson correlation coefficients ranging from0.18 for roast beef to 0.71 for coffee (mean = 0.44),after a mean time of 6.5 years.

We assessed questionnaire validity of food intake inthree ways. First, we examined the correlation for eachfood between the food frequency questionnaire andthe dietary records and corrected these coefficients forwithin-person variation in the dietary records. Cor-relation for the repeat questionnaire, which referred tothe period of diet recording, might have been influ-enced by an increased awareness of diet acquiredduring the process of record keeping. However, thedifferences between correlations for the first andsecond questionnaire tended to be small, suggestingany effect of increased awareness was not great. Cor-relation coefficients varied substantially among foods;in particular, correlations were high for dairy products,

at UB

Kaiserslautern on M


Dow

nloaded from



bread, cold breakfast cereal and most of the items inthe beverage group. One explanation of these differ-ences is that it is easier for people, as others havefound,37 to report the consumption of frequently eatenfoods, especially those not subject to seasonal varia-tion. In addition, for mathematical reasons, correla-tions will tend to be higher for foods with a largerbetween-person component of variance, and less fre-quently consumed foods tend to have a lower between-person component.

As a second measure of validity we compared theabsolute mean amounts of each food estimated by thetwo different methods. Although a ranking of subjectsby intake is sufficient for many epidemiological ana-lyses, an accurate estimate of the absolute amountconsumed is desirable to formulate recommendations,and compare different studies. Focusing on the secondquestionnaire, we found that butter, whole milk, eggs,processed meat and cold breakfast cereal were under-estimated by 10 to 30% on the questionnaire. In con-tast, a number of fruits and vegetables, skimmed orlow fat milk, yoghurt and fish were overestimated by atleast 50%. These findings for specific foods suggestthat participants over-reported consumption of foodsoften considered desirable or healthy, such as fruit andvegetables, and underestimated foods considered lessdesirable. A similar bias was found in an Australianstudy.38 Hauser and Bebb39 found similar over- andunderestimations in a dietary history compared withdietary records; citrus fruits, 'other cooked vegetables'and tea were overestimated by the dietary history andluncheon meat was underestimated. On the otherhand, they obtained opposite results for cereals andeggs (overestimated) and cooked green and yellowvegetables (underestimated).

It is possible that the over- or underestimation mayreflect some difficulties in comparing the portion sizeoffered as standard with the portion that is actuallyconsumed. This observation cannot explain the over-estimation of most vegetables, since mean and medianportions were generally close to half a cup,34 which isconsistent with the food frequency questionnaire stan-dard portion size. Yoghurt was one of the few foods forwhich the portion on the questionnaire (one cup) wasdifferent from the mean and median values observedon the dietary records (1/2 cup). We therefore com-pared the mean daily frequency of yoghurt consump-tion from the dietary records (without regard toamount) which was 0.04 instead of the portion size-weighted frequency of 0.02. The frequency value 0.04is closer to the result obtained from the questionnaires(0.07 for the first and 0.06 for the second), indicatingthat a misspecified portion may explain some, but not

all of the overestimation for this specific food by thequestionnaire. For skimmed milk, which was over-reported on the questionnaire, the overestimation wasexplained by a lower frequency of use on the dietaryrecords. This general tendency to over-report sociallydesirable foods, whether conscious or unconscious,will probably be difficult to eliminate by an alterationof questionnaire design.

Based on weekly averages, we found that the ratio ofwithin- to between-person variance was greater thanone for most foods. As a result, methods of dietaryassessment based on a single day of dietary intake willgrossly misrepresent an individual's usual intake ofspecific foods. High day-to-day within- to between-person variance ratio have been reported for nutrientintake27-28-30"32 and, using data from the persent study,for portion sizes.34 Had we calculated variance com-ponents for days, rather than weeks, the ratios wouldhave actually been approximately seven times higherthan those in Table 1, and thus much higher than theratios generally observed for nutrients in the previousstudies. Thus a single day of dietary data measuresintake of specific foods even less well than it measuresnutrient intake.

Correction of the observed crude correlationsbetween dietary records and questionnaires for thewithin-person variation in the dietary records providesmore accurate information regarding the validity of thequestionnaire. Substantial improvement in the cor-relation coefficients was seen for foods with highwithin-person variance, such as most vegetables, somefruits, and other items including hamburgers, pro-cessed meats and nuts.

The third approach used to evaluate the validity ofthe questionnaire was to calculate the actual consump-tion based on the dietary records for categories ofreported food frequency. This provides a simultaneousevaluation of misclassification and error in estimationof absolute intake, which proved to be very useful inappreciating the origin of over- or underestimation ofmean intakes. For example, we found that for carrotsand several other foods with lower correlations, lowfrequencies of use reported on the questionnaire weregenerally accurate, but high frequencies tended to beoverestimated. This approach also highlighted the roleof variation of consumption among individuals, sincecorrelations were higher for foods* consumed fre-quently by a substantial portion of tlie population.

Many authors, both in the past13 and morerecently,1*"26 have studied relationships betweenspecific foods or group of foods and disease, such ascancer and myocardial infarction. Information on thevalidity of data for intake of specific foods can greatly

at UB

Kaiserslautern on M


Dow

nloaded from



enhance the interpretation of findings from suchstudies.

These analyses based on specific foods can be used todetect weaknesses in the questionnaire and evaluateremedial changes. For example, it is clear that thequestionnaire item 'spinach and other greens' per-formed poorly since absolute intake was overestimatedby a factor of three or four and responses providedessentially no information on the classification of par-ticipants. While 'other greens' was intended to includekale, chard, and collards, we hypothesized that someparticipants also interpreted this question to includelettuce, which was not included in the 1980 question-naire since, in preliminary analyses, it did not seemimportant for any of the nutrient intakes of interest.We therefore reanalysed this item by including theconsumption of lettuce from the dietary records. Theobserved correlation coefficients increased just slightly(from 0.08 to 0.13 for the second questionnaire), sug-gesting that not everyone interpreted the question inthe same way. Since we had already suspected that thewording of this question was misunderstood, we used amore complete and specific list of foods in subsequentversions of our questionnaire. In particular, the ques-tion about spinach was split into two separate ques-tions: 'spinach, cooked (1/2 cup)' and 'spinach, raw (asin salads)'; and two separate questions about lettucewere added. Since this validation study is part of anongoing cohort, we also had data available from a 1982and 1984 dietary questionnaire for most participants.The 1982 questionnaire asked about food consumptionin the previous year, and 1984 questionnaire about thediet three to four years in the past.* When comparedwith 1980-1981 dietary records and corrected forwithin-person variation in the records, correlationcoefficients for cooked spinach were 0.51 using the1982 questionnaire, and 0.77 using the 1984 question-naire. For raw spinach the corrected coefficients were0.38 for the 1982 questionnaire and 0.22 for the 1984questionnaire. When the average dietary record valuesfor categories of the 1982 and 1984 questionnaireresponses were evaluated, substantially improved dis-crimination was observed. Overreporting on the ques-tionnaire" was markedly reduced, although notcompletely (data not shown).

These analyses document that the semi-quantitativefood frequency questionnaire provides reasonablyreproducible and valid measurements of relative con-sumption for most foods. Our findings apply directlyonly to women in the large cohort of registered nursesfrom which the participants were sampled. Thus,similar analyses among other populations would bedesirable. These data also demonstrate the utility of a

validation study to evaluate remedial changes for foodswith low reproducibility and validity. Furthermore,since structured dietary questionnaires will inevitablyprovide less than perfect measures of dietary intake,dietary records may be used to estimate the actualintake associated with specific responses to the ques-tionnaire items.

ACKNOWLEDGEMENTSThis work was supported by Research GrantsCA16686, CA23645, CA26560, CA4O935 andCA40356 from the National Institutes of Health.Simonetta Salvini was sponsored by the U.S.-ItalyCancer Program.

The authors thank the registered nurses who partici-pated in the study, Karen Corsano and Pradip Rana forcomputer programming, and Susan Newman for helpin preparing the manuscript.

REFERENCES1 Trillion M F. Subcommittee on methodology for dietary appraisal.

Am J Public Health 1960; 50: 39-52.2 Young C M, Trulson MF. Methodology for dietary studies in epi-

demiological surveys. H—Strengths and weaknesses of exist-ing methods. Am J Public Health 1960; 50: 803-14.

3Wiehl D, Reed R. Development of new or improved dietarymethods for epidemiological investigation. Am J PublicHealth 1960; 50: 824-8.

4 Stefanik P, Trulson M F. Determining the frequency intakes of foodsin large group studies. Am J Clin Nutr 1962; 11: 335-43.

5 Bazzarre T L, Myers M P. The collection of food intake data incancer epidemiology studies. Nutr Cancer 1979; 1: 22-45.

* Mettlin C J, Graham S. Methodological issues in etiologic studies ofdiet and colon cancer. Nutr Cancer 1979; 1: 46-55.

'Sampson L. Food frequency questionnaires as a research instru-ment. Out Nutr 1985; 4: 171-8.

8 Block G. A review of validations of dietary assessment methods. A mJ Epidemiol 1982; 115: 492-505.

' Willett C W, Sampson L, Stampfer M J el al. Reproducibility andvalidity of a semi-quantitative food frequency questionnaire.Am J Epidemiol 1985; 122: 51-65.

10 Willett W C, Reynolds R D, Cottrell-Hoehner S, Sampson L,Browne ̂ 1 L. Validation of a semiquantitative food frequencyquestionnaire; comparison with a one-year diet record. J AmDiet Assoc 1987; 87: 43-7.

11 Stuff J E, Garza C, Smith E O, Nichols B L, Montandon C M. Acomparison of dietary methods in nutritional studies. Am JClin Nutr 1983; 37: 300-6.

12 Nomura A, Hankin J H, Rhoads G G. The reproducibility of dietaryintake data in a prospective study of gastrointestinal cancer.Am J Clin Nutr 1976; 29: 1432-6.

u Graham S, Lilienfeld A M, Tidings J E. Dietary and purgationfactors in the epidemiology of gastric cancer. Cancer 1967; 20:2224-34.

14 Acheson E D, Doll R. Dietary factors in carcinoma of the stomach.A study of 100 cases and 200 controls. Gut 1964; 5: 126-31.

° Colditz G A, Willett W C, Stampfer M J a al. The influence of age,relative weight, smoking, and alcohol intake on thereproducibility of a dietary questionnaire. Int J Epidemiol1987; 16: 392-8.

at UB

Kaiserslautern on M


Dow

nloaded from



"Byers T, Marshall, J Anthony E, Fiedler R, Zielezny M. Thereliability of dietary history from the distant past. Am J Epi-demol 1987; 125: 999-1011.

17 Hankin J H, Rhoads G G, Glober G A. A dietary method for anepidemiological study of gastrointestinal cancer. Am J ClinNutr 1975; 28: 1055-61.

u Mettlin C, Graham S. Dietary risk factors in human bladder cancer.Am J Epidemiol 1979; 110: 225-63.

™ UiWn J H, Burns P E, Blot W J, Ziegler R G, Lees A W, FraumeniJ F. Dietary factors and breast cancer risk. IntJ Cancer 1981;28: 685-9.

"Graham S, Marshall J, Mettlin C, Rzepka T, Nemoto T, Byers T.Diet in the epidemiology of breast cancer. Am J Epidemiol1982; 116: 68-75.

21 Kvale G, Bjelke E, Gart J J. Dietary habits and lung cancer risk. InlJ Cancer 1983; 31: 397-405.

22 Colditz G A, Branch L G, LJpnick R J et al. Increased green andyellow vegetable intake and lowered cancer deaths in anelderly population. Am J Clin Nutr 1985; 41: 32-36.

DSlattery M, Sorenson A, Mahoney A, French T, Kritchevsky D.Diet and colon cancer: assessment of risk by fiber type andfood source (Abstract) Am J Epidemiol 1988; 128: 889.

24 McLaughlin J K, Gridley G, Block G, Blot W J, Ershow A, Winn DThe protective effect of fruit in the etiology of oral and phar-yngeal cancer (Abstract) Am J Epidemiol 1988; 128: 890.

25 Fraser G E, Strahan T M, Kissinger D G. Dietary and other nskfactors for definite myocardial infarction in Seventh-dayAdventists (Abstract) Am J Epidemiol 1988; 128: 916.

24 Morgan J W, Fraser G E, Philips R L, Andress M H. Dietary factorsand colon cancer incidence among Seventh-day Adventists(Abstract) Am J Epidemiol 1988; 128: 918.

27 Beaton G H, Milner J, Corey Petal. Sources of variance in 24-hourdietary recall data: implications for nutrition study design andinterpretation. Am J Clin Nutr 1979; 32: 2546-59.

28 Beaton G H, Milner J, McGuire V, Feather T E, Little J A. Sourcesof variance in 24-hour dietary recall data: implications fornutrition study design and interpretation. Carbohydratesources, vitamins and minerals. Am J Clin Nutr 1983; 37:986-95.

BTodd K S, Hudes M, Calloway D H. Food intake measurement:'problems and approaches. Am J Clin Nutr 1983; 37: 139-46.

30 Van Staveren W A, Deurenberg P, Katan M B d i l . Validity of thefatty acid composition of subcutaneous fat tissue microbiop-sies as an estimate of the longterm average fatty acid composi-tion of the diet of separate individuals. Am J Epidemiol 1986;123: 455-63.

31 McGeeD, Rhoads G, Han kin J, YanoK.Tillo tsonJ. Within-personvariability of nutrient intake in a group of Hawaiian men ofJapanese ancestry. Am J Clin Nutr 1982; 36: 657-63.

33 Sempos C, Johnson N E, Smith E L, Gffligan C. Effects of intrain-dividual and interindividiial variation in repeated diet records.Am J Epidemiol 1985; 121: 120-30.

33 Morgan K J, Johnson S R, Goungctas B. Variability of food intakes.An analysis of a 12-day data series using persistance measures.Am J Epidemiol 1987; 126: 326^*5.

"Hunter D, Sampson L, Stampfer M J, Rosner B, Colditz G A,Willett W C. Variability in portion sizes of commonly con-sumed foods among a population of US women. Am J Epi-demiol 1988; 127: 124(M9.

33 Rosner B, Willett W C. Interval estimates for correlation coeffi-cients corrected for within-person variation: Implications forstudy design and hypothesis testing. Am J Epidemiol 1988;127: 377-86.

*SAS User's Guide, SAS Institute, Inc., 1982.37 Jain M, Howe G R, Johnson K C, Miller A B. Evaluation of a diet

history questionnaire for epidemiologic studies. Am J Epi-demiol 1980; 111: 212-9.

38 Worsley A, Baghurst K I, Leitch D R. Social desirability and dietaryinventory responses. Human Nutr Appl Nutr 1984; 38A:29-35.

39 Houser H B, Bebb H T. Individual variation in intake of nutrients byday, month and season and relation to meal pattern: impli-cations for dietary survey methodology. In: Assessing chang-ing food consumption patterns. National Research CouncilCommittee on food consumption patterns. Washington DC,Nat Academy Press, 1981.

"Willett W C, Sampson L, Browne M et al. The use of a self-administered questionnaire to assess diet four years in thepast. Am J Epidemiol 1988; 127: 188-99

(Revised version received January 1989)

at UB

Kaiserslautern on M


Dow

nloaded from


Documents

Food-Based Validation of a Dietary Questionnaire: The Effects of Week-to-Week Variation in Food Consumption