View
25
Download
0
Category
Tags:
Preview:
DESCRIPTION
The use of information from experts for agricultural official statistics. Ballin Marco, Carbini Riccardo, Loporcaro Maria Francesca, Lori Massimo, Moro Roberto, Olivieri Valeria, Scanu Mauro Istituto Nazionale di Statistica. Roma, 10 July 2008. Aim of the work and summary of the talk. - PowerPoint PPT Presentation
Citation preview
Ballin Marco, Carbini Riccardo,
Loporcaro Maria Francesca, Lori Massimo,
Moro Roberto, Olivieri Valeria, Scanu Mauro
Istituto Nazionale di Statistica
The use of information from experts for agricultural official
statistics
Roma, 10 July 2008
Aim of the work and summary of the talkQ 2008
Roma, 10 June 2008
• The main aim of this work is to investigate if we can assess the quality of statistics based on experts opinion and if these statistics can be used as official statistics
• The talk– The Italian experience – Some reminders on elicitation and a proposal for an
elicitation scheme – Some results belonging to an experimental
elicitation– Quality report and next steps in the research
IntroductionQ 2008
Roma, 10 June 2008
Up to now, expert opinion has been widely used to produce short term statistics on crops.
Main users:• Eurostat• System of National Accounts• Economic Operators
Summarizing, local authorities supply an evaluation of area and yield on the different crops (Jannuary: estimates on areas for winter cereals, May: estimates on area for maize, June estimates on yields for winter crops ,….).
ExampleQ 2008
Roma, 10 June 2008
Example: common wheat area (national level) according to the Farm Structure Survey (sample survey) and aggregated evaluations from local authorities
Common wheat area (ha)
0
100.000
200.000
300.000
400.000
500.000
600.000
700.000
800.000
1997 1998 1999 2000 2003 2005
FSS
Evaluations from local authorities
ExampleQ 2008
Roma, 10 June 2008
Example 2: maize area (national level) according to the Farm Structure Survey (sample survey) and aggregated evaluations from local authorities
Maize area (ha)
850000
900000
950000
1000000
1050000
1100000
1150000
1200000
1997 1998 1999 2000 2003 2005
FSS
Evaliations from local authorities
Why a survey on expert opinions?Q 2008
Roma, 10 June 2008
…a first judgement• Up to now in Italy these “statistics” seem to reproduce quite well estimates produced by sample surveys for many crops
…some other important strengths:• Timeliness (results available before traditional survey estimates)• Analytic data (estimates with a high geographical detail)• Inexpensive
Why a survey on expert opinions?Q 2008
Roma, 10 June 2008
… but also some weaknesses:
• Heterogeneity of the process at local level• Lacking of accountability• Problems in assessing data quality
In other words
• difficulties in defining and filling up a quality report
ProposalQ 2008
Roma, 10 June 2008
From evaluation to structured elicitation
To overcome these difficulties and to transform experts opinion in an additional tool for official statisticians useful to investigate phenomena difficult to observe with traditional surveys we propose to adopt a formal expert elicitation
“Expert elicitation in the context of uncertainty quantification aims at a credible and traceable account of specifying probabilistic information regarding uncertainty in a structured and documented way.” (Hora, 1992)
ProposalQ 2008
Roma, 10 June 2008
From evaluation to structured elicitation
Elicitation is currently and successfully applied in many experimental situations, as weather forecasting, biomedical applications, nuclear risks assessment, attributing foodborne pathogen illness to food consumption …
Protocols on the elicitation process have already been introduced in many research fields. These allow to evaluate the quality of the elicitation process, making the users aware about the use of the statistical results
…….a proposal of elicitation in the official statistics context
• 1. Organization
– Definition of the problem (and the questionnaire)– Finding out at least one expert and one facilitator– To train the experts and the facilitators
• 2. Elicitation
– Carry out the interviews– Transform expert opinions into probability distributions– Combine opinions of different experts (if available) – Feedback– Production of final results
ProposalQ 2008
Roma, 10 June 2008
• 3. Evaluation of the elicitation results (quality report)
–Timeliness, Coherence, Relevance, Comparability, Accessibility (as usual statistics)
–Accuracy
• Variability of elicited distributions (if more then one distribution have been elicited; this indicator should replace the usual variance)
• Effect of feedback (this should replace the indicators on effect of re-interview)
• Fiducial interval (this should replace the confidence interval)• Assessment of expert knowledge by means of control/seed
variables (this indicator should replace indicators on bias)
–Furthermore • Self assessment of expert knowledge and of his/her sources of
information (by expert)• Assessment of how the questions in the questionnaire are phrased
(by expert)• meta-information on the interview (by facilitator)
ProposalQ 2008
Roma, 10 June 2008
An experience: the saffron caseQ 2008
Roma, 10 June 2008
Step 1. OrganizationProblem definition: structure of saffron sector in Italy (Saffron is a rare crop, not observable by sample surveys)
Questions on the following phenomena: production and total area of saffron in Abruzzo, in Italy, in the world; export and import quantities; number of planted bulbs in the last year; forecast on production, number of operators,…
Selection of experts: president of the consortium of saffron producers in one of the two most important areas for the production of saffron in Italy (Navelli county)
Selection of facilitators: Istat personnel with strong agricultural background belonging to regional office
Training: document on how the elicitation process is conducted and on the main biases that can affect the elicitation process. Mail and phone contacts
An experience: the saffron caseQ 2008
Roma, 10 June 2008
Step 2. Elicitation
face to face Interview (about one hour) on
• Minimum, Maximum, Mode (most probable value) for each phenomenon of interest
• Distribution shape using fixed interval method (Ten “X” to be put in five intervals of equal length)
Check of data and first results• Check of data (compatibility, units measure, etc.,…)• Fitted distribution (Rectangular or Beta)• Point estimates• Fiducial interval
Feedback from expert and facilitator • Update of elicitation by expert and facilitator• Final results
An experience: the saffron caseQ 2008
Roma, 10 June 2008
Final result (concerning production of saffron in 2007)120 Kg is the point estimation (mode)(105 kg -130 kg) is the range within the production of saffron in 2007 lies with probability one for the expert111-126 Kg is the 95% fiducial interval according to the expert distribution
Expert info
Fitted distribution
Fiducial interval
Point estimate
Q 2008
Roma, 10 June 2008
Step 3. Evaluation of elicitation results Some indicators among those proposed before,
•Effect of feedback (expert confirmed his point of view after the production of the first report)•Fiducial intervals (e.g. 111-126 Kg for the production of saffron in Italy)•Self assessment of expert knowledge and of his/her sources of information
An experience: the saffron case
0
1
2
3
4
Confidence in thereference sources
degree of agreementwith other
experts/producers
Contacts with local andnational producers
statistical culture
Index (sum of points/max) 81,2%
Q 2008
Roma, 10 June 2008
Step 3. Evaluation of elicitation results
• Assessment of how the questions are phrased (by expert)
•Results on number of planted bulbs could produce misunderstanding if they are not integrated with information about type and quality of bulbs
• Meta-information on the interview (by facilitator):
•Expert declared and seemed to be honoured to collaborate with Istat•He appeared well trained and showed a true mastery of the topics •He collaborated during the feedback too•He has years of experience and has written many publications on the topic
An experience: the saffron case
Final summary and Future developmentsQ 2008
Roma, 10 June 2008
Summary1) The elicitation process formalizes the uncertainty of one or more
experts on particular phenomena2) Statistics based on experts have some good properties 3) In some cases expert opinion is the only available sources of
information4) It is possible to define protocols and quality reports to upgrade
“numerical evaluations” to statistics 5) some indicators have been proposed for accuracy6) A small experience has been (partially) illustrated
Future developments 1) Combination of elicited distribution belonging to two or more experts
(by feedback, mathematical aggregation, …)2) Combination of elicited distributions and data belonging to surveys
(linear combination, bayesian framework, ….., used as early estimates to be replaced when data are available).
3) Development of protocols for fields different from agriculture4) …..
Roma, 10 June 2008
Bibliography
Anthony O'Hagan, Caitlin E. Buck, Alireza Daneshkhah, J. Richard Eiser, Paul H. Garthwaite, David J. Jenkinson, Jeremy E. Oakley, Tim Rakow (2006) Uncertain Judgements: Eliciting Experts' Probabilities. Wiley
Di Bacco (1990). l’aggregazione di valutazioni diprobabilità: una rassegna…non imparziale , 8, ed. Pitagora. Bologna
Garthwaite P. H., Kadane J. B., and O'Hagan A. (2005): Statistical methods for eliciting probability distributions, Journal of the American Statistical Association, 100, 680-701.
ISTAT (1993): Manuale delle statistiche agricole rilevate con le tecniche estimative. Note e relazioni, 1993, 1. Roma: ISTAT.
Mortera J.(1990). “Aggregazione delle opinioni: una panoramica” Rassegna di metodi statistici ed applicazioni, 8, ed. Pitagora. Bologna
SHELF (Sheffield Elicitation Framework): http://www.tonyohagan.co.uk/shelf
J.P. van der Sluijs, P.H.M. Janssen, A.C. Petersen, P. Kloprogge, J.S. Risbey, W. Tuinstra, J.R. Ravetz (2004):Tool Catalogue of the RIVM/MNP Guidance for Uncertainty Assessment and Communication
Q 2008
Roma, 10 June 2008
Saffron: a flower – many recipies
Saffron is obtained from the yellow “core” of the flower Crocus. It is mainly produced in Iran and India, with minor productions in Spain, Greece, and Italy.Italy produces about 120 kg of saffron per year.
It is used in many regional recipes, as the Risotto alla Milanese
Q 2008
Recommended