E D I TO R I A L

What Can Meta-Analyses Tell Us Aboutthe Reliability of Life Cycle Assessmentfor Decision Support?Miguel Brandao, Garvin Heath, and Joyce Cooper

This special supplemental issue makesclear that meta-analysis is very usefulin clarifying an understanding of im-pact magnitude and variability and ofthe underlying technological parametersthat drive the results [of the LCAs].

The body of life cycle assessment (LCA) literature is vastand has grown over the last decade at a dauntingly rapid rate.Many LCAs have been published on the same or very similartechnologies or products, in some cases leading to hundredsof publications. One result is the impression among decisionmakers that LCAs are inconclusive,owing to perceived and real variabil-ity in published estimates of life cy-cle impacts. Despite the extensiveavailable literature and policy needfor more conclusive assessments, onlymodest attempts have been made tosynthesize previous research. A sig-nificant challenge to doing so aredifferences in characteristics of theconsidered technologies and inconsistencies in methodologicalchoices (e.g., system boundaries, coproduct allocation, and im-pact assessment methods) among the studies that hamper easycomparisons and related decision support.

An emerging trend is meta-analysis of a set of results fromLCAs, which has the potential to clarify the impacts of a par-ticular technology, process, product, or material and producemore robust and policy-relevant results. Meta-analysis in thiscontext is defined here as an analysis of a set of publishedLCA results to estimate a single or multiple impacts for a sin-gle technology or a technology category, either in a statisticalsense (e.g., following the practice in the biomedical sciences)or by quantitative adjustment of the underlying studies to makethem more methodologically consistent. One example of thelatter approach was published in Science by Farrell and col-leagues (2006) clarifying the net energy and greenhouse gas(GHG) emissions of ethanol, in which adjustments includedthe addition of coproduct credit, the addition and subtractionof processes within the system boundary, and a reconciliationof differences in the definition of net energy metrics. Such ad-justments therefore provide an even playing field on which all

c© 2012 by Yale UniversityDOI: 10.1111/j.1530-9290.2012.00477.x

Volume 16, Number S1

studies can be considered and at the same time specify the con-ditions of the playing field itself. Understanding the conditionsunder which a meta-analysis was conducted is important forproper interpretation of both the magnitude and variability inresults.

This special supplemental issueof the Journal of Industrial Ecol-ogy includes 12 high-quality meta-analyses and critical reviews of LCAsthat advance understanding of thelife cycle environmental impactsof different technologies, processes,products, and materials. Also pub-lished are three contributions onmethodology and related discussions

of the role of meta-analysis in LCA. The goal of this specialsupplemental issue is to contribute to the state of the science inLCA beyond the core practice of producing independent studieson specific products or technologies by highlighting the abilityof meta-analysis of LCAs to advance understanding in areas ofextensive existing literature. The inspiration for the issue camefrom a series of meta-analyses of life cycle GHG emissions fromelectricity generation technologies based on research from theLCA Harmonization Project1 of the National Renewable En-ergy Laboratory (NREL), a laboratory of the U.S. Departmentof Energy, which also provided financial support for this specialsupplemental issue. (See the editorial from this special supple-mental issue [Lifset 2012], which introduces this supplementalissue and discusses the origins, funding, peer review, and otheraspects.)

The first article on reporting considerations for meta-analyses/critical reviews for LCA is from Heath and Mann(2012), who describe the methods used and experience gainedin NREL’s LCA Harmonization Project, which produced sixof the studies in this special supplemental issue. Their har-monization approach adapts key features of systematic reviewto identify and screen published LCAs followed by a meta-analytical procedure to adjust published estimates to ones basedon a consistent set of methods and assumptions to allow in-terstudy comparisons and conclusions to be made. In a secondstudy on methods, Zumsteg and colleagues (2012) propose a

www.wileyonlinelibrary.com/journal/jie Journal of Industrial Ecology S3

E D I TO R I A L

checklist for a standardized technique to assist in conduct-ing and reporting systematic reviews of LCAs, including meta-analysis, that is based on a framework used in evidence-basedmedicine. Widespread use of such a checklist would facilitateplanning successful reviews, improve the ability to identify sys-tematic reviews in literature searches, ease the ability to up-date content in future reviews, and allow more transparencyof methods to ease peer review and more appropriately gen-eralize findings. Finally, Zamagni and colleagues (2012) pro-pose an approach, inspired by a meta-analysis, for categorizingmain methodological topics, reconciling diverging methodolog-ical developments, and identifying future research directions inLCA. Their procedure involves the carrying out of a literaturereview on articles selected according to predefined criteria. Theanalysis highlights the need for improvement in LCA practica-bility and model fidelity.

The 12 meta-analyses in this special supplemental issue arelisted in Table 1. These studies elucidate the GHG emissionsof alternative electricity generation technologies (coal, pho-tovoltaics [PVs; crystalline silicon and thin-film PVs in twoarticles], concentrating solar power [CSP], wind, and nuclear)

and carbon-capture and storage, as well as LCA applications tobiobased materials and computers. Each study began with theidentification of relevant LCAs and followed with an analysisof a subset selected on the basis of screening criteria describedin each manuscript. As shown in Table 1, this subset rangedfrom 5 to 53 LCAs; however, in most cases the meta-analysescovered a larger number of technology systems, as each indi-vidual LCA in the selected set quite often compared multiplesystems.

Most of the 12 meta-analyses focus on the life cycle GHGemissions of electricity generation by different sources. Coal-fired and nuclear electricity generation systems are harmonizedby Whitaker and colleagues (2012) and Warner and Heath(2012). Harmonization of 53 utility-scale coal-fired electricitygeneration LCAs by Whitaker and colleagues (2012) finds thatapproximately 99% of life cycle GHG emissions are directlyrelated to the coal fuel cycle (including combustion) such thata first-order estimate of life cycle GHG emissions could be basedon knowledge of the technology type, coal mine emissions,thermal efficiency, and the combustion carbon dioxide emissionfactor alone without requiring full LCAs. This is in contrast to

Table 1 Meta analyses in this special supplemental issue

Technology, product or material studied Article Number of LCAs reviewed

Electricity generation Burkhardt, J. et al. Life Cycle Greenhouse Gas Emissions ofTrough and Tower Concentrating Solar Power ElectricityGeneration: Systematic Review and Harmonization

10

Dolan, S. and G. Heath. Life Cycle Greenhouse Gas Emissions ofUtility-Scale Wind Electricity Generation: Systemic Reviewand Harmonization

49

Hsu, D. et al. Life Cycle Greenhouse Gas Emissions of CrystallineSilicon Photovoltaic Electricity Generation: SystematicReview and Harmonization

13

Kim, H. C. et al. Life Cycle Greenhouse Gas Emissions ofThin-Film Photovoltaic Electricity Generation: SystematicReview and Harmonization

5

Padey, P. et al. A Simplified Life Cycle Approach for AssessingGreenhouse Gas Emissions of Wind Electricity

19

Price, L. et al. Wind Power as a Case Study: Improving Life CycleAssessment Reporting to Better Enable Meta-Analyses

18

Schreiber, A. et al. Meta-Analysis of Life Cycle AssessmentStudies on Electricity Generation with Carbon Capture andStorage

15

Warner, E. and G. Heath. Life Cycle Greenhouse Gas Emissionsof Nuclear Electricity Generation: Systematic Review andHarmonization

27

Whitaker, M. et al. Life Cycle Greenhouse Gas Emissions ofCoal-Fired Electricity Generation: Systematic Review andHarmonization

53

Biobased materials Weiss, M. et al. A Review of the Environmental Impacts ofBiobased Malerials.

44

Desktop computers Teehan, P. et al. Sources of Variation in Life Cycle Assessmentsof Desktop Computers

13

Consumer printers Gambeta, E. et al. Life Cycle Assessment in the Print Industry: ACritical Review

12

Notes: LCA = life cycle assessment; GHG = greenhouse gas.

S4 Journal of Industrial Ecology

E D I TO R I A L

the findings of Warner and Heath (2012) for light water nuclearpower. Significant variability remained after harmonization bysystem boundary and performance parameters, which could bequalitatively explained by variations in assumed primary sourceenergy mix, uranium ore grade, and the selected LCA method(i.e., process chain vs. economic input-output LCA methods).

Solar electricity generation systems are explored in threearticles. First, Burkhardt and colleagues (2012) harmonize tenCSP system LCAs and illustrate the use of a two-level har-monization process for parabolic trough and power tower tech-nologies. Utilizing so-called light harmonization, when the solarfraction and several other performance parameters of both tech-nologies are harmonized, a significant reduction in variabilitycompared to the published estimates of life cycle GHG emis-sions is revealed. A more intensive level of harmonization wasthen employed on a smaller pool of studies, which included ap-plication of consistent global warming intensities of materialsin the life cycle inventory and inclusion of auxiliary naturalgas and electricity consumption, revealing an even greater re-duction in the estimated variability but an increased centraltendency compared to the lightly harmonized results (owing tothe inclusion of required auxiliary natural gas and electricityconsumption, which are often incorrectly excluded from CSPLCAs). In a second investigation of solar electricity by Kim andcolleagues (2012), five studies are used to harmonize amorphoussilicon (a-Si), cadmium telluride (CdTe), and copper indiumgallium diselenide (CIGS) photovoltaic systems by adjusting ef-ficiency, irradiation, performance ratio, balance of system, andlifetime. Although the adjustment of all of these parameters isfound to contribute to a reduced estimate of variability, the im-portance of irradiation, efficiency, and lifetime are highlighted.Similarly, Hsu and colleagues (2012) harmonize 13 LCAs oncrystalline silicon photovoltaic electricity generation by adjust-ing efficiency, irradiation, the performance ratio, and lifetimeand also identify irradiation and lifetime as drivers of variabilityin the results.

The final three studies on electricity generation investigatewind power systems. Price and Kendall (2012) provide a system-atic review of LCAs to investigate life cycle GHG emissions formodern wind turbines in a wide study region (studies are fromAustralia, Brazil, Canada, Europe, India, New Zealand, Taiwan,and the United States). Adjustments made in turbine size, geo-graphic location, and end-of-life treatment in addition to thoseintended to produce a consistent system boundary across studiesare critiqued. The results of the critique are combined with arequirement that LCAs chosen for review include only thosewith original LCA data. The 18 LCAs passing the screeningcriteria are then assessed in a scoring rubric designed to as-sist in an understanding of consistency in LCA meta-analyses.Next, Padey and colleagues (2012) provide a meta-analysis oflife cycle GHG emissions of wind electricity on the basis of 19LCAs for systems recently manufactured and operated in Eu-rope. These authors use a screening approach somewhat similarto that of Price and colleagues, use adjustment levels repre-senting Europe, and find manufacturing materials, load factoras a function of wind speed, and product lifetime to be most

influential. Finally, estimates of life cycle GHG emissions fromwind electricity are harmonized by Dolan and Heath (2012)in an analysis of 49 LCAs from a global study region. Theseauthors employ a light harmonization approach with less of afocus on differences in manufacturing and end-of-life manage-ment. Adjustment of the capacity factor (i.e., the load factor),operating lifetime, and system boundaries revealed that harmo-nization by capacity factor resulted in the largest reduction invariability in life cycle GHG emissions. Note also that the windmeta-analyses of, for example, Padey and colleagues and Dolanand Heath must not be compared on the basis of the resultingmeans, as each harmonize/adjust parameters to a different set ofconditions.

Schreiber and colleagues (2012) performed a meta-analysisof 15 LCA studies on electricity generation with three carboncapture and storage technologies (postcombustion, oxyfuel, andprecombustion) with a focus on GHG reduction for differentregions, fuels, and time horizons. They present a condensedoverview of methodological variations, findings, and conclu-sions gathered from the 15 LCAs. Considering all capture tech-nologies, time horizons, or fuels evaluated, the potential climatebenefits of these technologies are counterbalanced by impactson a range of other environmental categories (e.g., acidifica-tion, eutrophication, and photochemical ozone creation). Theresults are significantly sensitive to three parameter sets: powerplant efficiency and energy penalty of the capture process, car-bon dioxide capture efficiency and purity, and fuel origin andcomposition.

For the studies beyond those related to electricity generation,Weiss and colleagues (2012) perform a comprehensive meta-analysis on the environmental benefits and burdens of biobasedmaterials, in which 44 LCAs were reviewed. The authors foundthat biobased materials save both energy and GHG emissionsrelative to their fossil counterparts. Conversely, biobased mate-rials may increase eutrophication and stratospheric ozone deple-tion. Differences in impacts on acidification and photochemicalozone formation are inconclusive. The large uncertainty of indi-vidual LCA studies highlights the difficulties in drawing generalconclusions about the relative environmental merits betweendifferent materials.

Teehan and colleagues (2012) provide a systematic reviewof LCAs on desktop computers aimed at understanding vari-ability and discrepancies among published studies. Specifically,whereas the majority of studies find that the use phase domi-nates GHG emissions, three studies disagree with the majority.Given this, Teehan and colleagues select and decompose 13LCAs to the system component, life cycle phase, and inventoryflow levels. Their decomposition to the component level washampered by a lack of transparency in the published studiesthat did not allow assessment or adjustment of the underlyingparameters. Using published data, they find the manufactur-ing phase at a smaller but substantial level of contribution tothe overall results. Alternatively, Teehan and colleagues foundmuch higher transparency in the use-phase data within thestudies reviewed. They reveal that assumptions concerning thehours of daily use directly correlate with the dominance of the

Brandao et al., What Can Meta-Analyses Tell Us About the Reliability of LCA? S5

E D I TO R I A L

use phase, and they question the general applicability of lowuse estimates (e.g., within the context of plug load measure-ments). As a result, they identify the use phase as dominantfor energy demand and contribution to climate change, withthe only exception being regions with low GHG electricitygeneration.

Finally, Gambeta and colleagues (2012) critically review 12LCAs on consumer imaging equipment using an InternationalOrganization for Standardization (ISO) 14040 framework toidentify common practices, limitations, and opportunities forimprovement and standardization. Their analysis suggests thatcomparisons across studies are significantly hampered by vari-ability in methods and reporting. They conclude that stan-dardization of the functional unit and the assumptions that areinterwoven with it has a high potential to increase quantitativecomparability across studies.

This special supplemental issue makes clear that meta-analysis is very useful in clarifying an understanding of impactmagnitude and variability and of the underlying technologi-cal parameters that drive the results. Meta-analyses of LCAsare becoming more widely recognized in the field for thesevirtues through special sessions at conferences, for instance the2010 International Life Cycle Assessment (InLCA) conference(Heath et al. 2010) and the upcoming 2012 Society of Environ-mental Toxicology and Chemistry (SETAC) World Congress(Heath and Brandao 2012), the call for papers for this spe-cial issue that produced many excellent submissions, includingsome not yet published, and the few publications that precededthe aforementioned (e.g., the multiregression analysis of Lenzenand Munksgaard 2002). However, the results of LCA studies—and the subsequent decisions they support—are dependent ona wide range of factors that make each LCA study unique. Thismay limit the use of LCA for decision support, unless LCAstudies abide by the same methodological guidelines and prin-ciples and are thus consistent and comparable. Data quality isoften cited as the major bottleneck of robust LCAs, but otherfactors play a large role. The variability of LCA results doesnot depend solely on the variability of the data employed, butrather on a range of factors. Methodological choices relatedto scope, system boundaries, allocation, choice of impact as-sessment method, as well as other assumptions, make LCA atool that often generates uncertain outcomes. All these factorsdecrease the impact LCA could have in supporting decisionsin both public policy and business domains. Therefore, moreharmonization needs to take place. More standardization couldinclude the adoption of a clear set of criteria to facilitate anal-ysis of how data quality, scope, assumptions, key findings, andthe like affect the results, and for the complete reporting of allkey assumptions and methods. Therefore the robustness of LCAstudies cannot be assessed without an uncertainty analysis.

On a global level, the ISO 14040–44 series (ISO 2006a,2006b) attempt to provide some level of standardization andharmonization in both methodological and procedural choicesand reporting. Recent developments that complement and gobeyond the ISO standards come from the European Commis-sion’s Joint Research Centre, including the International Ref-

erence Life Cycle Data System (ILCD) handbook (EuropeanCommission 2010) and an LCA directory2 containing severalLCA studies and using clear fields that structure and facilitateanalysis in terms of suitability for consideration in a policy-support context or for meta-analysis purposes. Additional devel-opments include those under the United Nations EnvironmentProgramme (UNEP)-SETAC Life Cycle Initiative.3

Many journals have published numerous LCA studies inrecent years. In order to ensure the quality and relevance ofLCA studies, not only is peer review an important step, butso is conformity to a common set of rules for performing anLCA. Even though there is still no commonly accepted andapplied global standard, not even ISO, the articles in thisspecial supplemental issue show that LCA results are, moreoften than not, pointing in the same direction. This sug-gests that LCA is already relevant for supporting decisions,though it could be strengthened through meta-analysis of pre-vious research and methodological guidelines for the conduct offuture LCAs.

Acknowledgements

Support for this special supplemental issue was provided bythe U.S. Department of Energy through the National Renew-able Energy Laboratory.

Notes

1. Additional data and results of the project are available athttp://openei.org/apps/LCA.

2. http://lct.jrc.ec.europa.eu/assessment/directories3. http://lcinitiative.unep.fr/

References

Burkhardt, J., G. Heath, and E. Cohen. 2012. Life cycle greenhousegas emissions of trough and tower concentrating solar power elec-tricity generation: Systematic review and harmonization. Journalof Industrial Ecology. DOI: 10.1111/j.1530-9290.2012.00474.x

Dolan, S. and G. Heath. 2012. Life cycle greenhouse gas emissionsof utility-scale wind electricity generation: Systemic review andharmonization. Journal of Industrial Ecology. DOI: 10.1111/j.1530-9290.2012.00464.x

European Commission. 2010. General guide for life cycle assessment–Detailed guidance. International reference life cycle data system(ILCD) handbook. Ispra, Italy: Joint Research Centre, Institutefor Environment and Sustainability.

Farrell, A. E., R. J. Plevin, B. T. Turner, A. D. Jones, M. O’Hare,and D. M. Kammen. 2006. Ethanol can contribute to energy andenvironmental goals. Science 311(5760): 506–508.

Gambeta, E., J. Bousquin, M. Esterman, and S. Rothenberg. 2012. Lifecycle assessment in the print industry: A critical review. Journalof Industrial Ecology. DOI: 10.1111/j.1530-9290.2012.00471.x

Heath, G., H.-K. Kim, B. Sovacool, and R. Plevin. 2010. Special sessionon Meta-analysis of energy LCAs. Presented at Life Cycle Assess-ment X: Bridging Science, Policy and the Public, 2-4 November,Portland, OR.

S6 Journal of Industrial Ecology

E D I TO R I A L

Heath, G. and M. Brandao. 2012. Special session on increasing scien-tific and policy understanding through meta-analysis of life cycleassessments. Presented at 6th SETAC World Congress 2012: Se-curing a sustainable future: Integrating science, policy and people,20-24 May, Berlin, Germany.

Heath, G. and M. Mann. 2012. Background and reflections on theLife Cycle Assessment Harmonization Project. Journal of IndustrialEcology. DOI: 10.1111/j.1530-9290.2012.00478.x

Hsu, D., P. O’Donoughue, V. Fthenakis, G. Heath, H. C. Kim, P.Sawyer, J.-K. Choi, and D. Turney. 2012. Life cycle greenhousegas emissions of crystalline silicon photovoltaic electricity gener-ation: Systematic review and harmonization. Journal of IndustrialEcology. DOI: 10.1111/j.1530-9290.2011.00439.x.

ISO (International Organization for Standardization). 2006a. ISO14040. Environmental management – Life cycle assessment – Princi-ples and framework. Geneva, Switzerland: ISO.

ISO (International Organization for Standardization). 2006b. ISO14044. Environmental management – Life cycle assessment – Re-quirements and guidelines. Geneva, Switzerland: ISO.

Kim, H. C., V. Fthenakis, J.-K. Choi, and D. Turney. 2012. Life cy-cle greenhouse gas emissions of thin-film photovoltaic electricitygeneration: Systematic review and harmonization. Journal of In-dustrial Ecology. DOI: 10.1111/j.1530-9290.2011.00423.x.

Lenzen, M. and J. Munksgaard. 2002. Energy and CO2 life-cycle analy-ses of wind turbines—Review and applications. Renewable Energy26(3): 339–362.

Lifset, R. 2012. Toward meta-analysis in life cycle assessment. Journalof Industrial Ecology. DOI: 10.1111/j.1530-9290.2012.00473.x

Padey, P., I. Blanc, D. Le Boulch, and X. Zhao. 2012. A simplified lifecycle approach for assessing greenhouse gas performance of windelectricity. Journal of Industrial Ecology. DOI: 10.1111/j.1530-9290.2012.00466.x.

Price, L. and A. Kendall. 2012. Wind power as a casestudy: Improving life cycle assessment reporting to bet-ter enable meta-analyses. Journal of Industrial Ecology. DOI:10.1111/j.1530-9290.2011.00458.x.

Schreiber, A., P. Zapp, and J. Marx. 2012. Meta-analysis of lifecycle assessment studies on electricity generation with car-bon capture and storage. Journal of Industrial Ecology. DOI:10.1111/j.1530-9290.2011.00435.x.

Teehan, P. and M. Kandlikar. 2012. Sources of variation in life cycleassessments of desktop computers. Journal of Industrial Ecology.DOI: 10.1111/j.1530-9290.2011.00431.x.

Warner, E. and G. Heath. 2012. Harmonization of nuclear lifecycle GHG emissions. Journal of Industrial Ecology. DOI:10.1111/j.1530-9290.2012.00472.x.

Weiss, M., J. Haufe, M. Carus, M. Brandao, S. Bringezu, B. Her-mann, and M. K. Patel. 2012. A review of the environmentalimpacts of biobased materials. Journal of Industrial Ecology. DOI:10.1111/j.1530-9290.2012.00468.x.

Whitaker, M., G. Heath, P. O’Donoughue, and M. Vorum. 2012. Lifecycle greenhouse gas emissions of coal-fired electricity generation:Systematic review and harmonization. Journal of Industrial Ecology.DOI: 10.1111/j.1530-9290.2012.00465.x.

Zamagni, A., P. Masoni, P. Buttol, A. Raggi, and R. Buonamici. 2012.Finding life cycle assessment research direction with the aid ofmeta-analysis. Journal of Industrial Ecology. DOI: 10.1111/j.1530-9290.2012.00467.x

Zumsteg, J., J. Cooper, and M. Noon. 2012. Systematic review check-list: A standardized technique for assessing and reporting reviewsof life cycle assessment data. Journal of Industrial Ecology. DOI:10.1111/j.1530-9290.2012.00476.x

About the Authors

Miguel Brandao was a scientific officer at the Joint ResearchCentre of the European Commission in Ispra, Italy, when thisissue was prepared. He is currently working at the Interna-tional Life Cycle Academy, Barcelona, Spain. He is also anassociate editor for LCA of the Journal of Industrial Ecology, anda member of the Steering Committee of SETAC Europe LCA.Garvin Heath is a senior scientist and member of the Technol-ogy Systems and Sustainability Analysis Group in the StrategicEnergy Analysis Center of the U.S. Department of Energy’sNational Renewable Energy Laboratory (NREL), Golden, Col-orado, USA. He led the team that conducted NREL’s LCAHarmonization Project. Joyce Cooper is an associate professorof mechanical engineering at the University of Washington,Seattle, Washington, USA.

Address correspondence to:Miguel BrandaoInternational Life Cycle Academy2.-0 LCA ConsultantsBarcelona-Copenhagenmb@lca-net.comwww.lca-net.com

Brandao et al., What Can Meta-Analyses Tell Us About the Reliability of LCA? S7