Meta-Analyses of Experimental Data

Animal (2008), 2:8, pp 1203–1214 & The Animal Consortium 2008doi:10.1017/S1751731108002280

animal

Meta-analyses of experimental data in animal nutrition*

D. Sauvant1-, P. Schmidely1, J. J. Daudin2 and N. R. St-Pierre3

1UMR Physiologie de la Nutrition et Alimentation, INRA-AgroParisTech, 16 rue Claude Bernard, 75231 Paris Cedex 05, France; 2ENGREF, Mathematique etInformatique Appliquee, INRA-AgroParisTech, 16 rue Claude Bernard, 75231 Paris Cedex 05, France; 3Department of Animal Sciences, The Ohio State University,2029 Fyffe Rd., Columbus, OH-43210, USA

(Received 6 April 2007; Accepted 14 March 2008)

Research in animal sciences, especially nutrition, increasingly requires processing and modeling of databases. In certain areasof research, the number of publications and results per publications is increasing, thus periodically requiring quantitativesummarizations of literature data. In such instances, statistical methods dealing with the analysis of summary (literature) data,known as meta-analyses, must be used. The implementation of a meta-analysis is done in several phases. The first phaseconcerns the definition of the study objectives and the identification of the criteria to be used in the selection of priorpublications to be used in the construction of the database. Publications must be scrupulously evaluated before being enteredinto the database. During this phase, it is important to carefully encode each record with pertinent descriptive attributes(experiments, treatments, etc.) to serve as important reference points for the rest of the analysis. Databases from literaturedata are inherently unbalanced statistically, leading to considerable analytical and interpretation difficulties; missing data arefrequent, and data structures are not the outcomes of a classical experimental system. An initial graphical examination of thedata is recommended to enhance a global view as well as to identify specific relationships to be investigated. This phase isfollowed by a study of the meta-system made up of the database to be interpreted. These steps condition the definition of theapplied statistical model. Variance decomposition must account for inter- and intrastudy sources; dependent and independentvariables must be identified either as discrete (qualitative) or continuous (quantitative). Effects must be defined as either fixedor random. Often, observations must be weighed to account for differences in the precision of the reported means. Once modelparameters are estimated, extensive analyses of residual variations must be performed. The roles of the different treatmentsand studies in the results obtained must be identified. Often, this requires returning to an earlier step in the process. Thus,meta-analyses have inherent heuristic qualities.

Keywords: meta-analysis, mixed models, nutrition

Introduction

The research environment in the animal sciences, especiallynutrition, has markedly evolved in the recent past. Inparticular, there is a noticeable increase in the numberof publications, each containing an increasing number ofquantitative measurements. Meanwhile, treatments oftenhave smaller effects on the systems being studied than inthe past. Additionally, controlled and non-controlled factors,such as the basal plane of nutrition, vary from studyto study, thus requiring at some point a quantitative sum-marization of past research.

Fundamental research in the basic animal science dis-ciplines generates results that increasingly are at a muchlower level of aggregation than those of applied research(organs, whole animals), thus supporting the necessity ofintegrative research. Research stakeholders, those who ulti-mately use the research outcome, increasingly want morequantitative knowledge, particularly on animal responseto diet, and of better precision. Forecasting and decision-support software require quantitative information. Addition-ally, research prioritization by funding sources may forceabandoning active research activities in certain fields. In suchinstances, meta-analyses can still support discovery activitiesbased on the published literature.

The objectives of this paper are to describe the applicationof meta-analytic methods to animal nutrition studies, includ-ing the development and validation of literature-derived

*Salaries and research support were provided by state and federal fundsappropriated to the Ohio Agricultural Research and Development Center, theOhio State University.- E-mail: [email protected]

1203

databases, and the quantitative techniques used to extractthe quantitative information.

Definitions and nature of problems

Limits to classical approachesResults from a single classical experiment cannot be thebasis for a large inference space because the conditionsunder which observations are made in a single experimentare forcibly very narrow, i.e. specific to the study inquestion. Such studies are ideal to demonstrate cause andeffect, to test specific hypothesis regarding mechanismsand modes of action. In essence, a single experimentationmeasures the effects of one or a very few factors whilemaintaining all other factors as constant as possible. Often,experiments are repeated by others to verify the generalityand repeatability of the observations that were made,as well as to challenge the range of applicability of theobserved results and conclusions. Hence, it is not uncom-mon that over time, tens of studies are published even on arelatively narrow subject. In this context, there is a need tosummarize the findings across all the published studies.Meta-analytic methods are concerned with how best toachieve this integration process.

The classical approach to synthesizing scientific knowl-edge has predominantly been based on qualitative litera-ture reviews. A limitation of this approach is the obvioussubjectivity involved in the process. The authors subjectivelyweigh outcomes from different studies. Criteria for theinclusion or non-inclusion of studies are ill defined at best.Different authors can draw dramatically different conclu-sions from the same initial set of published studies. Addi-tionally, the limitation of the human brain to differentiatethe effects of many factors becomes very apparent once thenumber of publications involved exceeds 12 to 15 studies.

Definitions and objectives of meta-analysesMeta-analyses use objective, scientific methods based onstatistics to summarize and quantify knowledge acquiredthrough prior published research. Meta-analytic methodswere initially developed in psychology, medicine and socialsciences a few decades ago. Meta-analytic reviews havebeen more recent and much less frequent in nutrition.In general, meta-analyses are conducted for one of thefollowing four objectives:

> For global hypothesis testing, such as testing for theeffect of a certain drug or of a feed additive using theoutcomes of several publications that had as an objec-tive the testing of such an effect. This was by far thepredominant objective of the first meta-analyses pub-lished (Mantel and Haenszel, 1959; Glass, 1976). Earlyon, it was realized that many studies lacked statisticalpower for statistical testing, so that the aggregation ofresults from many studies would lead to much greaterpower (hence lower type II error), more precise pointestimation of the magnitude of effects and narrowerconfidence intervals of the estimated effects.

> For empirical modeling of biological responses, such as theresponse of animals to nutritional practices. Because thedata extracted from many publications cover a much widerset of experimental conditions than those of each individualstudy, conclusions and models derived from the wholeset have a much greater likelihood of yielding relevantpredictions to assist decision-makers. These meta-analyticactivities have led to a new paradigm suggested bySauvant (1992 and 1994) as a law of multiple responses tochanges in feeding practices. For example, alteration infeeding practices impact feed efficiency, product quality,the environment, animal welfare, etc., and significantprogress can be made in this type of holistic research usingmeta-analyses. There are numerous examples of suchapplications of meta-analytical methods in recent nutritionpublications, such as the quantification of the physiologicalresponse of ruminants to types of dietary starch (Offner andSauvant, 2004), to the supply of dietary N (Rico-Gomezand Faverdin, 2001), to dietary fats (Schmidely et al., 2008)and rumen defaunation (Eugene et al., 2004). Othershave used meta-analyses to quantify phosphorus flux inruminants (Bravo et al., 2003) or carcass characteristics tovarious factors (McPhee et al., 2006).

> For collective summarizations of measurements thatonly had a secondary or minor role in prior experiments.Frequently, results are reported in publications with theobjective of supporting the hypothesis or observationsrelated to the effect of one or a few experimental factors.For example, ruminal volatile fatty acid (VFA) concentra-tions are reported in studies investigating the effects ofdietary starch, or forage types. None of these studieshave as an objective the prediction of ruminal VFAs. Butthe aggregation of measurements from many studiescan lead to a better understanding of factors controllingVFA concentrations, or allow the establishment of newresearch hypotheses.

> In mechanistic modeling, for parameter estimates andestimates of initial conditions of state variables. Mechanisticmodels require parameterization, and meta-analyses offer amechanism of estimation that makes parameter estimationmore precise and more applicable to a broader range ofconditions. Meta-analyses can also be used for externalmodel evaluation (Lovatto and Sauvant, 2002; Sauvant andMartin, 2004), or for a critical comparison of alternatemechanistic models (Offner and Sauvant, 2004; Offneret al., 2003).

Types of data and factors in meta-analysesAs in conventional statistical analyses, dependent variables inmeta-analyses can be of various types such as binary [0, 1](e.g. for pregnancy), counts or percentages, categorical-ordinal (good, very good, excellent), and continuous, which isthe most frequent type in meta-analyses related to nutrition.

Independent factors (or variables) may be modeled usinga fixed or random effect. In general, factors related tonutrition (grain types, dry matter intake (DMI), etc.) shouldbe considered as fixed effects factors. The ‘study’ effect can

Sauvant, Schmidely, Daudin and St-Pierre

1204

be considered as either random or fixed. If a datasetcomprised many individual studies from multiple researchcenters, the ‘study’ effect should be considered randombecause each study is conceptually a random outcome froma large population of studies to which inference is to bemade (St-Pierre, 2001). This is especially important if themeta-analysis has for objective the empirical modeling ofbiological responses, or the collective summarizations ofmeasurements that only had a secondary or minor role inprior experiments, because it is likely that the researcher inthose instances has a targeted range of inference muchlarger than the limited conditions represented by the spe-cific studies. There are instances, however, where eachexperiment can be considered as an outcome each from adifferent population. In such instances, the levels of ‘study’or ‘trial’ are, in essence, considered arbitrarily chosen bythe research community, and the ‘study’ effect must then beconsidered fixed. In such an instance, the range of inferencefor the meta-analysis is limited to the domain of the specificexperiments in the dataset. This is of little concern if theobjective of the meta-analysis is that of global hypothesistesting, but it does severely limit the applicability of itsresults for other objectives.

Difficulties inherent to the dataThe meta-analytic database is best conceptualized withrows representing treatments, groups or lots, while thecolumns consist of the measured variables (those for whichleast-squares means are reported) and characteristics (classlevels) of the treatments or trials. A primary characteristicof most meta-analytic databases is the large frequencyof missing data in the table. This reduces the possibility ofusing large multi-dimensional descriptive models, andgenerally forces the adoption of models with a small subsetof independent variables, frequently two. Additionally, thedesign of the meta-analytic data, sometimes referred to asthe meta-design, is not determined prior to the data col-lection as in classic randomized experiments. Consequently,meta-analytic data are generally severally unbalanced andfactor effects are far from being orthogonal (independent).This leads to unique statistical estimation problems similarto those observed in observational studies, such as leveragepoints, near collinearity, and even complete factor discon-nectedness, thus prohibiting the testing of the effects thatare completely confounded with others.

Table 1 shows an example of factor disconnectedness,where two factors, each taking three possible levels, arebeing investigated. In this example, factors A and B aredisconnected because one cannot join all bordering pairs ofcells with both horizontal and vertical links. Consequently,even in a model without any interaction terms, the effect ofthe third level of A cannot be estimated separately from theeffect of the third level of B. This would be diagnoseddifferently depending on the software used with differentcombination of error messages in the log, zero degrees offreedom for some effects in the ANOVA table, or a missingvalue for the statistic used for testing.

In general, the variance between studies is large com-pared to the variance within studies, hence underlying theimportance of including the study effect into the meta-analytical model. The study effect represents the sum of theeffects of many factors that differ between studies, butfactors that are not in the model because they either werenot measured, or have been excluded from the model, orfor which the functional form in the model is inadequatelyrepresenting the true but unknown functional form (e.g. themodel assumes a linear relationship between the depen-dent and one independent continuous variable whereas thetrue relationship is nonlinear). In the absence of interactionsbetween design variables (e.g. studies) and the covariates(e.g. all model variables of interest), parameter estimatesfor the covariates are unbiased, but the study effect can adda large uncertainty to future predictions (St-Pierre, 2001).The presence of significant interactions between studiesand at least one covariate is more problematic since thisindicates that the effect of the covariate is dependent onthe study, implying that the effect of a factor is dependenton the levels of unidentified factors.

Steps in the meta-analytic processThere are several inherent steps to meta-analyses, theimportant ones are summarized in Figure 1. An importantaspect of this type of analyses is the iteration process,which is under the control of the analyst. This circularpattern where prior steps are re-visited and refined is animportant aspect of meta-analyses and contributes much totheir heuristic characteristic.

Objectives of the studyEstablishing a clear set of study objectives is a critical stepthat guides most ulterior decisions such as the databasestructure, data filtering, weighing of observations andchoice of the statistical model. Objectives can cover a widerange of targets, ranging from preliminary analyses toidentify potential factors affecting a system, thus serving animportant role to the formulation of research hypotheses infuture experiments, to the quantification of the effect of anutritional factor such as a specific feed additive.

Data entryResults from prior research found in the literature must beentered in a database. The structure and coding of thedatabase must include numerous variables identifyingthe experimental objectives of all experiments selected.Hence, numerous columns are potentially added to code

Table 1 Example of factor disconnectedness

Factor A 1 2 3

Factor B1 x x2 x x3 x

Meta-analyses in nutrition

1205

each objective found in all the sources of experimentaldata. This coding is necessary so as to avoid the improperaggregation of results across studies with very differentobjectives. During this coding phase, the analyst may choseto transform a continuous variable to a discrete variablewith n levels coded in a single column with levels of thediscrete variable as entries, or in n columns with 0 to 1entries to be used as dummy variables in the meta-analyticmodel. Different criteria can guide the selection of classes,such as equidistant classes, or classes with equal fre-quencies or probability of occurrence. The important point isthat the sum of these descriptive columns must entirelycharacterize the objectives of all studies used.

Data filteringThere are at least three steps necessary to effectivedata filtering. First, the analyst must ensure that the studyunder consideration is coherent with the objectives of themeta-analysis. That is, the meta-analytic objectives dictatethat some traits must be measured and reported. If, forexample, the objective of meta-analysis is to quantify therelationship between dietary neutral detergent fiber (NDF)concentration and DMI, then one must ensure that bothNDF concentration and DMI were measured and reportedin all studies. The second step consists of a thorough andcritical review of each publication under consideration,focusing on the detection of errors in the reporting ofquantitative results. This underlines the importance ofhaving a highly trained professional involved in this phaseof the study. Only after publications have passed this‘expert’ quality filter should their results be entered in thedatabase. Verification of data entries is then anotheressential component to the process. In this third step, it isimportant to ensure that a selected publication does notappear to be an outlier with respect to the characteristicsand relations under consideration.

Preliminary graphical analysesA thorough visual analysis of the data is an essential step tothe meta-analytic process. During this phase, the analystcan form a global view regarding the coherence and hetero-geneity of the data, as well as to the nature and relativeimportance of the inter- and intrastudy relationships ofprospective variables taken two at a time. Systematic gra-phical analyses should lead to specific hypotheses and tothe initial selection of alternate statistical models. Graphicscan also help in identifying observations that appear uniqueor even outliers to the mass of all other observations. Thegeneral structure of relationships can also be identified,such as linear v. nonlinear relationships as well as thepresence of interactions. As an example, Figure 2 shows afictitious example of an intrastudy curvilinear relationshipbetween two variables in the presence of a negativeinterstudy effect. In this example, the negative interstudyeffect associated with the X variable indicates the presenceof a latent variable that differed across studies, and thatinteracts with the X variable. A review of the specificcharacteristics of studies (3) and (2) in this example mighthelp to identify this latent variable. This ‘visualization’phase of the data should always be taken as a preliminarystep to the statistical analysis and not as conclusive evi-dence. The reason is that as the multi-dimensions of thedata are collapsed into two- or possibly three-dimensionalgraphics, the unbalance that clearly is an inherent char-acteristic of meta-analytic data can lead to false visualrelationships. This is because X–Y graphics do not correctthe observations for the effects of all other variables thatcan affect Y.

Graphical analyses should also be done with regard tothe joint coverage of predictor variables, identifying theirpossible ranges, plausible ranges and joint distributions, allbeing closely related to the inference range. Similar X–Ygraphics should be drawn to explore the relationships

OBJECTIVES

SELECTION OF EXPERIMENTS

& DATA ENTRYo o

DETERMINATION OFMETA-DESIGN

SELECTION OFSTATISTICAL

MODEL

POST-ANALYSISEVALUATION

CONCEPTUAL Basis

Literature search,ExperimentsSurvey,

USERS«Adjustments»

GRAPHICALANALYSIS

Figure 1 Schematic representation of the meta-analytic process.


1206

between predictor variables taken two at a time. In suchgraphics, the presence of any linear trends indicatescorrelations between predictor variables. Strong positiveor negative correlations of predictor variables have twoundesirable effects. First, they may induce near collinearity,implying that the effect of one predictor cannot be uniquelyidentified (i.e. is nearly confounded with the effect ofanother predictor). In such instances, the statistical modelcan include only one of the two predictors at a time.Second, the range of a predictor X1 given a level of asecond predictor X2 is considerably less than the uncondi-tional range of predictor X1. In these instances, althoughthe range of a predictor appears considerable in a uni-variate setting, its effective range is actually very muchreduced in the multivariate space.

Figure 3 illustrates some of these concepts using anactual set of data on chewing activity in cattle and the NDFcontent of the diet. Visually, one concludes that both theintra- and interstudy relationships between chewing activityand diet NDF are nonlinear. This observation can be formallytested using the statistical methods to be outlined later.In this example, the possible NDF range is 0% to 100%,whereas its plausible range is more likely between 35% and60%. Interestingly, the figure illustrates that experimentalmeasurements were frequently in the 20% to 40% and45% to 55% ranges, leaving a hole with very few obser-vations in the 40% to 45% range.

Study of the experimental meta-designThe meta-design is determined by the structure of theexperiments for each of the predictor variables. To char-acterize the meta-design, numerous steps must takeplace before and after the statistical analyses. The specificsteps depend on the number of predictor variables in themodel.

One predictor variable> The experimental design used in each of the studies

forming the database must be identified and coded,

and their relative frequencies calculated. This informationcan be valuable during the interpretation of the results.

> Frequency plots (histograms) of the predictor variable canidentify areas of focus of prior research. For example,Figure 4 shows the frequency distributions of NDF for the517 treatment groups for the meta-analytic dataset usedto draw Figure 3. Figure 4 indicates a substantial researcheffort towards diets containing 30% to 35% NDF, an areaof dietary NDF density that borders the lower limit ofrecommended dietary NDF for lactating dairy cows.Because of the high frequency of observations in the 30%to 35% NDF range, the a priori expectation is that theeffect of dietary NDF will be estimated most precisely inthis range.

> One should also consider the intra- and interstudyvariances for the predictor variable. Small intrastudyvariances reduce the ability of assessing the structuralform of the intrastudy relationship between the predictorand the dependent variable. Large intrastudy variances,but with only two levels of the predictor variable, canhide any potential curvilinear relationships. Figure 5

Figure 2 Example of a curvilinear intrastudy effect in the presence of alinear negative interstudy effect.

20 30 40 50 60 70

300

400

500

600

700

800

900

1000

1100

Che

win

g (M

in/d

)

NDF (% of DM)

Figure 3 Effect of dietary neutral detergent fiber (NDF) content onchewing activity in cattle. Data are from published experiments where theNDF content of the diet was the experimental treatment.

10 20 30 40 50 60 70 80

0

10

20

30

40

50

60F

req

uen

cyn = 517m = 38.4s = 13

NDF (% of DM)

Figure 4 Frequency plot of neutral detergent fiber (NDF) content of dietsacross experimental treatments.


1207

shows the intrastudy root mean square error (S) of NDFas a function of the mean dietary NDF for experimentswith two treatments, and for those with more thantwo treatments. The analyst can determine a minimumthreshold of S to exclude experiments with little intrastudyvariation in the predictor variable. In instances where thestudy effect is considered random, this is not necessary. Insuch instances, the inclusion of studies with little variationin the predictor variable does little in determining therelationship between the predictor and the dependentvariable, but adds observations and degrees of freedom toestimate the variance component associated with the studyeffect. When looking at a possible curvilinear intrastudyrelationship, it is intuitive to retain only those experimentswith three or more levels of the predictor variables in thedataset. However, in Figure 5, this eliminates 40% of all theobservation in the study. In instances where the curvilinearrelationship is only apparent interstudy, one may chose toretain only experiments with three or more treatments, orwork on the intrastudy deviation between the control andtreatment groups, as suggested by Rico-Gomez andFaverdin (2001).

> Another important aspect at this stage of the analysis isto determine the significance of the study effect on thepredictor variables. As explained previously, one must bevery careful regarding the statistical model used toinvestigate the intrastudy effect when there is not a goodagreement between the intra- and interstudy effects. Inthese instances, the relationship between the predictorvariable and the dependent variable depends on thestudy, which itself represents the sums of a great manyfactors such as measurement errors, systematic differ-ences in the methods of measurements of the dependentvariable across studies, and, more importantly, latentvariables (hidden) not balanced across experiments. Inthose instances, the analyst must exert great caution in

the interpretation of the results, especially regarding theapplicability of these results.

> At this stage, even before statistical models are to befitted and effects tested, it is generally useful to calculatethe leverage of each observation (Tomassone et al.,1983). Traditionally, in statistics, leverage values arecalculated after the model is fitted to the data, butnothing prohibits the calculation of leverage values at anearlier stage because their calculations depend only onthe design of the predictor variable in the model. Forexample, in the case of the simple linear regression withn observations, the leverage point for the ith observationis calculated as

hi ¼ 1=nþ ðXi � XmÞ2=SðXi � XmÞ

2; ð1Þ

where hi is the leverage value, Xi is the value of the ithpredictor variable and Xm is the mean of all Xi.

Equation (1) clearly indicates that the leverage of anobservation, i.e. its weight in the determination of theslope, grows with its distance from the mean of the pre-dictor variable. The extension of the leverage pointcalculations to more than one predictor variables isstraightforward (St-Pierre and Glamocic, 2000).

> In a final step, the analyst must graphically investigatethe functional form of the relationship between thedependent variable and the predictor variable.

Two or more predictor variables: In the case of two ormore predictor variables, the analyst must examine gra-phically and then statistically the inter- and intrastudyrelationships between the predictor variables. Leveragevalues should be examined. With fixed models (all effects inthe models are fixed with the exception of the error term),variance inflation factors (VIFs) should be calculated foreach predictor variable (St-Pierre and Glamocic, 2000).An equivalent statistic has not been proposed for mixedmodels (e.g. when the study effect is random), butasymptotic theory would support the calculation of the VIFsfor the fixed effect factors in cases where the total numberof observations is large. The objective in this phase isto assess the degree of inter-dependence between thepredictor variables. Because predictor variables in meta-analyses are never structured prior to their determination,they are always non-orthogonal and, hence, show variabledegrees of inter-dependency. Collinearity determinations(VIFs) assess one’s ability to separate the effects of inter-dependent factors based on a given set of data. Collinearityis not model driven, but completely data driven.

Weighing of observations

Because meta-analytic data are extracted from the resultsof many experiments conducted under many differentstatistical designs and number of experimental units,the observations (treatment means) have a wide range ofstandard errors. Intuition and classical statistical theorywould indicate that observations should be subjected to

number of treatments >2

15 25 35 45 55 65 75

0

10

20

Mean NDF (%DM)

Intra-experiment root mean square error (%DM)

number of treatments = 2

Figure 5 Intrastudy root mean square error of dietary neutral detergentfiber (NDF) and mean NDF of experiments designed to study the effects ofNDF in cattle.


1208

some sort of weighing scheme. Systems used for weighingobservations form two broad categories.

Weighing based on classical statistical theoryUnder a general linear model where observations haveheterogeneous but known variances, maximum likelihoodparameters estimates are obtained by weighing eachobservation by the inverse of its variance. In the contextof a meta-analysis where observations are least-squares(or population marginal) means, observations should beweighed by the inverse of the squares of their standarderrors, which are the standard errors of each mean (s.e.).Unfortunately, when such weights are used, the resultingmeasures of model errors (i.e. standard error, standard errorof predictions, etc.) are no longer expressed in the originalscale of the data. To maintain the expressions of dispersionin the original scale of the measurements, St-Pierre (2001)suggested dividing each weight by the mean of all weights,and to use the resulting values as weighing factors in theanalysis. Under this procedure, the average weight usedis algebraically equal to 1.0, thus resulting in expressionsof dispersion that are in the same scale as the originaldata. The application of this technique is software depen-dent. In the Statistical Analysis System (SAS), for example,weights are automatically rescaled so that their sums mustbe equal to 1.

Weighing based on other criteriaOther weighing criteria have been suggested for theweighing of observations, such as the power of an experi-ment to detect an effect of a size defined a priori, theduration of an experiment, etc. The weighing scheme canactually be based on an expert assessment, partially sub-jective, of the overall quality (precision) of the data. Theopinion of more than one expert may be useful in thiscontext. From a Bayesian statistical paradigm, the use ofsubjective information for decision-making is perfectlycoherent and acceptable, as subjective probabilities areoften used to establish prior distributions in Bayesiandecision theory (De Groot, 1970). Traditional scientific objec-tiveness, however, may restrict the use of this weighingscheme in scientific publications.

Predictably, the importance of weighing observationsdecreases with the number of observations used in theanalysis, especially if the observations that would receivea small weight have relatively small leverage values. Forexample, we conducted a meta-analysis to quantify the effectof concentrate intake on milk production and intake by dairycows (Sauvant et al., personal communication). The data,consisting of 208 treatment means from 85 experiments thathad as a primary objective to study the effect of concentrateintake, were analyzed with and without a weighing schemebased on the reciprocals of the s.e. The resulting weightsranged between 0.25 and 12.5. The functional structure ofthe response was a quadratic function with fixed studyeffect. In the example, the dependent variable was DMI (kg/day). The independent variable (predictor) was concentrate

DMI (CI, kg/day). For DMI, the estimated function usingunweighed observations was

DMI ¼ 16:7 ð0:41Þ þ 0:64 ð0:09Þ CI

� 0:018 ð0:004Þ CI2 ðRMSE ¼ 1:02Þ: ð2Þ

The same function estimated using weighed observationswas:

DMI ¼ 17:5 ð0:41Þ þ 0:48 ð0:09Þ CI

� 0:012 ð0:005Þ CI2 ðRMSE ¼ 1:56Þ: ð3Þ

Clearly, the regression coefficients of (2) and (3) are veryclose to each other and do not lead to much difference inthe predicted response to CI.

Whether the analyst should weight the observationsbased on the s.e. for each individual treatments or thepooled s.e. from the studies is open to debate. There aremany reasons why the s.e. of each treatment within a studycan be different. First, the original observations themselvescould have been homoscedastic (homogeneous variance)but the least-squares means would have different s.e. dueto unequal frequencies (e.g. missing data). In such a case,it is clear that the weight should be based on the s.e. ofeach treatment. Second, the treatments may have inducedheteroscedasticity, meaning that the original observationsdid not have equal variances across sub-classes. In suchinstances, the original authors should have conducted a testto assess the usual homoscedasticity assumption in linearmodels. The problem is that a lack of significance (i.e.P . 0.05) when testing the homogeneity assumption doesnot prove homoscedasticity, but only that the null hypo-thesis (homogeneous variance) cannot be rejected at aP , 0.05. In a meta-analytic setting, the analyst may deemthe means with larger apparent variance to be less credibleand reduce the weight of these observations in the analysis.Unfortunately, most publications lack the informationnecessary for this option.

Among the more subjective criteria available for weigh-ing is the quality of the experimental design used in theoriginal study with regard to the meta-analytic objective.Experimental designs have various trade-offs due to theirunderlying assumptions. For example, the Latin square isoften used in instances where animal units are relativelyexpensive, such as in metabolic studies. The double ortho-gonal blocking used to construct Latin square designscan remove a lot of variation from the residual error. Thusvery few animals can be used compared to a completelyrandomized design for an equal power of detecting treat-ment effect. The downside, however, is that the periodsare generally relatively short to reduce the likelihood ofa period by treatment interaction (animals in differentphysiological status across time periods), thus reducing themagnitude of the treatment effects on certain traits, such asproduction and intake, for example. In those instances, theanalyst should legitimately weigh down observations from


1209

experiments whose designs limited the expression of thetreatment effects.

Statistical modelsThe independent variable can be either discrete or con-tinuous. With binary data (healthy/sick, for example), gen-eralized linear models (GLM) based on the logit or probitlink functions are generally recommended (Agresti, 2002).Because of advances in computational power, the GLM hasbeen extended to include random effects in what is calledthe generalized linear mixed model (GLMM). In its version9, the SAS system includes a beta release of the GLIMMXprocedure to fit these complicated models. In nutrition,however, the large majority of the dependent variablessubjected to meta-analyses are continuous, and their ana-lyses are treated at length in the remainder of this paper.

St-Pierre (2001) made a compelling argument to includethe study effect in all meta-analytic models. Because of thesevere imbalance in most databases used for meta-ana-lyses, the exclusion of the study effect in the model leads tobiased parameter estimates of the effects of other factorsunder investigation, and severe biases in variance esti-mates. In general, the study effect should be consideredrandom because it represents, in essence, the sum of theeffects of a great many factors, all with relatively smalleffects on the dependent variable. Statistical theory indi-cates that these effects would be close to Gaussian(normal), thus much better estimated if treated as randomeffects. Practical recommendations regarding the selectionof the type of effect for the studies are presented in Table 2.In short, the choice depends on the size of the conceptualpopulation, and the sample size (the number of studies inthe meta-analysis).

The ultimate (and correct) meta-analysis would be onewhere all the primary (raw) data used to perform theanalyses in each of the selected publications were availableto the analyst. In such an instance, a large segmentedmodel that includes all the design effects of the originalstudies (e.g. the columns and rows effects in Latin squares)plus the effects to be investigated by the meta-analysiscould be fitted by least-squares or maximum likelihoodmethods. Although computationally complex, such hugemeta-analytic models should be no more difficult to solvethan the large models used by geneticists to estimatethe breeding values of animals using very large national

databases of production records. Raw data availabilityshould not be an issue in instances where meta-analysesare conducted with the purpose of summarizing research ata given research center. This, however, is very infrequent,and meta-analyses are almost always conducted usingobservations that are themselves summaries of prior experi-ments (i.e. treatment means). It seems evident that a meta-analysis conducted on summary statistics should lead to thesame results as a meta-analysis conducted on the raw data,which itself would have to include a study effect becausethe design effects are necessarily nested within studies (e.g.cow 1 in the Latin square of study 1 is different from thecow 1 of study 2), which itself would likely be consideredrandom. Thus, analytical consistency dictates the inclusionof the study effect in the model, generally as a randomeffect. The study effect will be considered random in theremainder of this paper, with the understanding thatunder certain conditions explained previously it should beconsidered a fixed effect factor.

Besides the study effect, meta-analytic models includeone or more predictor variables that are either discrete orcontinuous. For clarity, we will initially treat the case ofeach variable type separately, understanding that a modelcan easily include a mixture of variables of both types, aswe describe later.

Model with discrete predictor variable(s)A linear mixed model easily models this situation as follows:

Yijk ¼ mþ Si þ tj þ Stij þ eijk; ð4Þ

where Yijk 5 the dependent variable, m 5 overall mean,Si 5 the random effect of the i th study, assumed , iidN(0, s2

S), tj 5 the fixed effect of the j th level of factor t,Stij 5 the random interaction between the i th studyand the j th level of factor t, assumed , iidN (0, s2

St),and eijk 5 the residual errors, assumed , iidN (0, s2

e).eijk, Stij and Si are assumed to be independent randomvariables.

For simplicity reasons, model (4) is written without weighingthe observations. The weights would appear as multi-plicative factors of the diagonal elements of the errorvariance-covariance matrix (Draper and Smith, 1998). Model(4) corresponds to an incomplete, unbalanced randomizedblock design with interactions in classic experimental

Table 2 Guidelines to establish whether an effect should be considered fixed or random in a meta-analytic model*

Population Experiment Effect of t in the model

Case 1 T is small- tffi T Fixed effectCase 2 T is large t5 T Random effectCase 3 T is large tffi T Should be fixed but random works betterCase 4 T is large t5 T, and t is very small Should be random but fixed may work better

(i.e. variance components can be poorly estimated)

*Adapted from Milliken (1999) for mixed models.-T represents the number of studies in the population (conceptual), and t is the number of studies in the meta-analysis.


1210

research. The following SAS statements can be used tosolve this model:

PROC MIXED DATA ¼ Mydata CL COVTEST;

CLASSES study tau;

MODEL Y ¼ tau;

RANDOM study study � tau;

LSMEANS tau;

RUN; ð5Þ

Standard tests of significance on the effect of t are easilyconducted and least-squares means can be separated usingan appropriate mean separation procedure. Although it maybe tempting to remove the study effect from the model ininstances where it is not significant (also called pooling ofeffects), this practice can lead to biased probability estima-tions (i.e. final tests on fixed effects are conditional on testsfor random effects) and is not recommended. This is becausenot being able to reject the null hypothesis of no study effect(i.e. variance due to study is not significantly different fromzero) is a very different proposition than proving that theeffect of the study is negligible. At the very least, the prob-ability threshold for significance of study should be muchlarger than the traditional P 5 0.05 (say P , 0.25). Ideally, theanalyst should state before the analysis is performed whatsize of estimated variance due to study should be considerednegligible, such as s2

so0:1 s2e.

Model with continuous predictor variable(s)A linear mixed model also easily models this situation.

Yij ¼ B0 þ Si þ B1Xij þ biXij þ eij; ð6Þ

where Yij 5 the dependent variable, B0 5 overall (inter-study) intercept (a fixed effect equivalent to m in (4)),Si 5 the random effect of the i th study, assumed , iidN(0, s2

S), B1 5 the overall regression coefficient of Y on X(a fixed effect), Xij 5 the value of the continuous predictorvariable, bi 5 the random effect of study on the regressioncoefficient of Y on X, assumed , iidN (0, s2

b), and eij 5 theresidual errors, assumed , iidN (0, s2

e). Also, eij, bi and Si

are assumed to be independent random variables.The following SAS statements can be used to solve this

model:


CLASSES study;

MODEL Y ¼ X=SOLUTION;

RANDOM study study � X;

RUN; ð7Þ

Using a simple Monte Carlo simulation, St-Pierre (2001)demonstrated the application of this model to a syntheticdataset, showing the power of this approach, and theinterpretation of the estimated parameters.

Model with both discrete and continuous predictorvariable(s)Statistically, this model is a simple combination of (4) and(6) as follows:

Yijk ¼ mþ Si þ tj þ Stij þ B1Xij þ biXij þ BjXij þ eijk;

ð8Þ

where Bj 5 the effect of the jth level of the discrete factor ton the regression coefficient (a fixed effect).

The following SAS statements would be used to solvethis model:


CLASSES study tau;

MODEL Y ¼ tau X tau � X;

RANDOM study study � tau study � X;

LSMEANS tau;

RUN; ð9Þ

In theory, (8) is solvable, but the large number of variancecomponents and interaction terms that must be estimatedin combination with the imbalance in the data makes itoften numerically intractable. In such instances, at leastone of the two random interactions must be removed fromthe model.

In (4), (6) and (8), the analyst secretly wishes for theinteractions between study and the predictor variables to behighly non-significant. Recall that the study effect repre-sents an aggregation of the effects of many uncontrollableand unknown factors that differed between studies. Asignificant study by factor t interaction (Stij) in (4) impliesthat the effect of t is dependent on the study, hence offactors unaccounted for. Similarly, a significant interactionof study by X in (6) (bi Xij) indicates that the slope of thelinear relationship of Y on X is dependent on the study,hence of unidentified factors. In such a situation, theanalysis produces a model that can explain very well theobservations, but predictions of future outcomes are gen-erally not precise because the actual realization of a futurestudy effect is unknown. The maximum likelihood predictorof a future observation is produced by setting the study andthe interaction of study with the fixed effect factors to theirmean effect values of zero (McCulloch and Searle, 2001),but the standard error of this prediction is very muchamplified by the uncertainty regarding the realized effect ofthe future study.

When the study effect and its interaction with fixed effectis correctly viewed as an aggregation of many factors notincluded in the model, but that differed across studies, thedesirability of including as many fixed factors in the modelas can be uniquely identified from the data becomesobvious. In essence, the fixed effects should ultimatelymake the study effect and its interactions with fixed effectspredictors small and negligible. In such instances, theresulting model should have wide forecasting applicability.


1211

Imagine, for example, that much of the study effect on bodyweight gain of animal is in fact due to the large differencein the initial body weight across studies. In such instances,the inclusion of initial body weight as a covariate wouldremove much of the study effect, and the diet effects(as continuous or discrete variables) would be estimatedwithout biases, with a wide range of applicability (i.e. afuture prediction would require a measurement of initialbody weight as well as measurements of the other predictorvariables).

Whether one chooses (4) or (6) as a meta-model issomewhat arbitrary when the predictor variable has aninherent scale (i.e. is a measured number). The assumptionsregarding the relationship between Y and the predictorvariable are, however, very different between the twomodels. In (4), the model does not assume any functionalform for the relationship. In (6), the model explicitlyassumes a linear relationship between the dependent andthe predictor variables. Different methods can be used todetermine whether the relationship should have a linear ornonlinear structure.

> The first method consists in classifying observations intofive sub-classes based on the quintiles for the predictorvariable, and performing the analysis according to (4) withfive discrete levels of the predictor variable. Althoughthe selection of five sub-classes is somewhat arbitrary,there are substantive references in the statistical literatureindicating that this number of levels generally works well(Cochran, 1968; Rubin, 1997). A visual inspection of thefive least-squares means or the partitioning of the fourdegrees of freedom associated with the five levels of thediscrete variable into singular orthogonal polynomialcontrasts can rapidly identify an adequate functional formto use for modeling the Y–X relationship.

> The second method can be directly applied to the data, orcan be a second step that follows the identification of anadequate degree for a polynomial function. Model (6) isaugmented with the square (and possibly higher orderterms) of the predictor variable. In the MIXED procedureof SAS, this can be done simply by adding an X *X termto the model statement. It is important to understandthat in the context of a linear (mixed or not) model, thematrix representation of the model and the solutionprocedure used are no different when X and X2 are in themodel compared to a situation where two differentcontinuous variables (say X and Z) are included in themodel. The problem, however, is that X and X2 areimplicitly not independent; after all, there is an algebraicfunction relating the two. This dependence can result in alarge correlation between the two variables, thus leadingto possible problems of collinearity.

> A third method can be used in more complex situationswhere the degree of the polynomial exceeds two, or theform of the relationship is sigmoid, for example. Therelationship can be modeled as successive linear segments,an approach conceptually close to the first method

explained previously. Martin and Sauvant (2002) used thismethod to study the variation in the shape of the lactationcurves of cows subjected to various concentrate sup-plementation strategies, using the model of Grosmanand Koops (1988) as its fundamental basis. Using thisapproach, lactation curves were summarized by a vector ofnine parameter estimates, whose estimates could becompared across supplementation strategies.

In (8), the interest may be in the effect of the discretevariable t after adjusting for the effect of a continuousvariable X as in a traditional covariate analysis, or theinterest may be inverse, i.e. the interest is in the effect ofthe continuous variable after adjusting for the effect of thediscrete variable. The meta-analyses of Firkins et al. (2001)provide examples of both situations. In one instance, theeffect of grain processing (a discrete variable) on milk fatcontent was being investigated while correcting for theeffect of DMI (a continuous variable). In this instance, theinterest was in determining the effect of the discrete vari-able. In another instance, the effects of various dietaryfactors such as dietary NDF, DMI and the proportion offorage in the diet (all continuous variables) on starch andNDF digestibility, and microbial N synthesis were investi-gated, while correcting for the effect of the grain proces-sing. In this case, the interest was much more towards theeffects of the continuous variables exempt from possiblebiases due to different grain processing across experiments.

Accounting for interfering factorsDifferences in experimental conditions between studies canaffect the treatment response. The nature of these condi-tions can be represented by quantitative or qualitativevariables. In the first instance, the variable and possibly itsinteraction with other factors can be added to the modelif there are sufficient degrees of freedom. The magnitudeof treatment response is sometimes dependent on theobserved value in the control group. For example, the milkfat response in lactating cows to dietary buffer supple-mentation as a function of the milk fat of the control group(Meschy et al., 2004). The response was small or non-existent when the milk fat of control cows was near 40 g/l,but increased markedly when the control cows had low milkfat, possibly reflecting a higher likelihood of sub-clinicalrumen acidosis in these instances.

When the intrastudy response depends on the levels of thepredictor variable, it is often because of the existence of anonlinear interstudy relationship. Applying model (6) with theaddition of a square term for the Xij to the data shown inFigure 3 results in a relatively good quantification of therelationship between chewing time and dietary NDF, asshown in Figure 6. In this type of plot, it is important to adjustthe observations for the study effect, or the regression mayappear to poorly fit the data because of the many hiddendimensions represented by the studies (St-Pierre, 2001).

In instances where the interacting factor is discrete, theexamination of the sub-classes least-squares means can


1212

clarify the nature of the interaction. For example, the effectof a dietary treatment may be dependent on the physiolo-gical status of the animals used in the study. This physio-logical status can be coded using multiple dummy variables,as explained previously.

Post-optimization analyses

As when fitting conventional statistical models, numerousanalyses should follow the fitting of a meta-analytic model.These analyses are used to assess the assumptions underlyingthe model, and to determine whether additional meta-analyticmodels should be investigated (the heuristic process illustratedin Figure 1).

Structure of residual variationIn (4), (6) and (8), the residuals (errors) are assumedindependent, and identically distributed from a normalpopulation with a mean of zero and a variance s2

e. Thenormality assumption can be tested using a standard x2

test, or a Shapiro-Wilks test, both available in the UNI-VARIATE procedure of the SAS system. The residuals canalso be expressed as Studentized residuals, with absolutevalues exceeding 3 being suspected as outliers (Tomassoneet al., 1983). In meta-analysis, the removal of a suspectedoutlier observation should be done only with extremecaution. This is because the observations in a meta-analysisare the calculated outcomes (least-squares means) ofmodels and experiments that should themselves be nearlyfree of the influence of outliers. Thus, meta-analytic outlierscan be much more likely indicative of a faulty model than ofa defective observation. In addition, the removal of onetreatment mean as an observation in a meta-analysis mightbe removing all the variation in the predictor variable forthe experiment in question, thus making the value of theexperiment in a meta-analytic setting nearly worthless. Inaddition, the analyst should examine for possible intra- and

interstudy relationships between the residuals and thepredictor variables.

Structure of study variationWhen a model of the type described in (4) is being fitted, itis possible to examine each study on the basis of its ownresiduals. For example, Figure 7 shows the distribution ofthe residual standard errors for the different studies used inthe meta-analysis of chewing time in cattle (Figures 3and 6). Predictably, the distribution is asymmetrical andfollows the law of Raleigh for the standard errors, whilevariances have a x2 distribution. Studies with unusualstandard errors, say those with a x2 probability exceeding0.999, could be candidates for exclusion from the analysis.Alternatively, one could consider using the inverse of theestimated standard errors as weights to be attached to theobservation before re-iterating the meta-analysis.

Other calculations such as leverage values, Cook’s dis-tances and other statistics can be used to determine theinfluence of each observation on the parameter estimates(Tomassone et al., 1983).

Conclusion

Meta-analyses produce empirical models. They are invalu-able for the synthesis of data that at first may appearscattered without much pattern. The meta-analytic processis heuristic and implicitly allows returning to prior steps.Extensive graphical analyses must be performed prior to theparameterization of a statistical model to gain a visualunderstanding of the data structure as well as to validatedata entries.

The increased frequency of meta-analyses published inthe scientific literature coupled with scarce funding forresearch should create an additional need for scientificjournals to ensure that published articles provide sufficientinformation to be used in a subsequent meta-analysis.There may be a time when original data from publishedarticles will be available via the web in a standardizedformat, a current practice in DNA sequencing research.

20 30 40 50 60 70

500

600

700

800

900

1000

NDF (% DM)

Che

win

g (m

in/d

)

Y = 55.9 + 29.4 X - 0.242 X2(n = 216, nexp = 88, RMSE = 43.4)

Figure 6 Effect of dietary neutral detergent fiber (NDF) content onchewing activity in cattle. Data are from published experiments where theNDF content of the diet was the experimental treatment. Observationswere adjusted for the study effect before being plotted, as suggested bySt-Pierre (2001). n 5 number of treatments; nexp 5 number of experiments.

3210

10

5

0

Standard errors of studies (min/d)

Fre

qu

ency

?

Figure 7 Frequency distribution of the residual standard errors of thestudies used in the meta-analysis of chewing activity in cattle.


1213

Lastly, new meta-analytic methods should assist theexpansion of mechanistic modeling efforts of complexbiological systems by providing conceptual models as wellas a structured process for their external evaluation.

Acknowledgment

The authors are thankful to Philippe Faverdin for his commentson an earlier draft of this paper.

ReferencesAgresti A 2002. Categorical data analysis, 2nd edition. John Wiley & Sons,Hoboken, New Jersey.

Bravo D, Sauvant D, Bogaert C and Meschy F III 2003. Quantitative aspects ofphosphorus excretion in ruminants. Reproduction, Nutrition, Development 43,285–300.

Cochran WG 1968. The effectiveness of adjustment by subclassification inremoving bias in observational studies. Biometrics 24, 295–313.

De Groot MH 1970. Optimal statistical decision. Mc Graw-Hill, New York.

Draper NR and Smith H 1998. Applied regression analysis. John Wiley & Sons,Inc., New York.

Eugene M, Archimede H and Sauvant D 2004. Quantitative meta-analysis onthe effects of defaunation of the rumen on growth, intake and digestion inruminants. Livestock Production Science 85, 81–91.

Firkins JL, Eastridge ML, St-Pierre N and Noftsger SM 2001. Invited: Effects ofgrain variability and processing on starch utilization by lactating dairy cattle.Journal of Animal Science 79(E. Suppl.), E218–E238.

Glass GV 1976. Primary, secondary and meta-analysis of research. EducationResearch 5, 3–8.

Grosman M and Koops WJ 1988. Multiphasic analysis of lactation curves indairy cattle. Journal of Dairy Science 71, 1598–1608.

Lovatto PA and Sauvant D 2002. Meta-analyse et modelisation de l’ingestionvolontaire chez le porc. Journees de la Recherche Porcine, Paris (FRA) 2002/02/05–07, 129–134.

Mantel N and Haenszel W 1959. Statistical aspects of the analysis of data fromretrospective studies of disease. Journal National of Cancer Institute 22,719–748.

Martin O and Sauvant D 2002. Meta-analysis of input/output kinetics inlactating dairy cows. Journal of Dairy Science 85, 3363–3381.

McCulloch CE and Searle SR 2001. Generalized, linear, and mixed models. JohnWiley & Sons, Inc., New York.

McPhee MJ, Oltjen JW, Famula TR and Sainz R 2006. Meta-analysis of factorsaffecting carcass characteristics of feedlot steers. Journal of Animal Science 84,3143–3154.

Meschy F, Bravo D and Sauvant D 2004. Analyse quantitative des reponses desvaches laitieres a l’apport de substances tampon. INRA Productions Animales17, 11–18.

Milliken GA 1999. Mixed models. Seminar to the Cleveland Chapter of theAmerican Statistical Association, May 10.

Offner A and Sauvant D 2004. Prediction of in vivo starch digestion in cattlefrom in situ data. Animal Feed Science and Technology 111, 41–56.

Offner A, Bach A and Sauvant D 2003. Quantitative review of in situ starchdegradation in the rumen. Animal Feed Science and Technology 106, 81–93.

Rico-Gomez M and Faverdin P 2001. La nutrition proteique modifie l’ingestiondes vaches laitieres: analyse bibliographique. Rencontres RecherchesRuminants 8, 285–288.

Rubin DB 1997. Estimating causal effects from large data sets using propensityscores. Annals of Internal Medicine 127, 757–763.

Sauvant D 1992. La modelisation systemique en nutrition. Reproduction,Nutrition, Development 32, 217–230.

Sauvant D 1994. Modelling homeostatic and homeorhetic regulations inlactating animals. Livestock Production Science 39, 105–113.

Sauvant D and Martin O 2004. Empirical modelling through meta-analysis vsmechanistic modelling. In Nutrient digestion and utilization in farm animals:modelling approaches (ed. E Kebreab). CAB International, London.

St-Pierre NR 2001. Invited review: integrating quantitative findings frommultiple studies using mixed model methodology. Journal of Dairy Science 84,741–755.

St-Pierre NR and Glamocic D 2000. Estimating unit costs of nutrients frommarket prices of feedstuffs. Journal of Dairy Science 83, 1402–1411.

Schmidely P, Glasser F, Doreau M and Sauvant D 2008. Digestion of fatty acidsin ruminants: a meta-analysis of flows and variation factors. 1. Total fattyacids. Animal 2, 677–690.

Tomassone R, Lesquoy E and Millier C 1983. La regression: nouveau regard surune ancienne methode statistique. In Actualites scientifiques et agronomiquesde l’INRA (ed. Masson). INRA, Paris.


1214

Documents

Meta-Analyses of Experimental Data