Ballin Marco, Carbini Riccardo,

Loporcaro Maria Francesca, Lori Massimo,

Moro Roberto, Olivieri Valeria, Scanu Mauro

Istituto Nazionale di Statistica

The use of information from experts for agricultural official

statistics

Roma, 10 July 2008

Aim of the work and summary of the talkQ 2008

Roma, 10 June 2008

• The main aim of this work is to investigate if we can assess the quality of statistics based on experts opinion and if these statistics can be used as official statistics

• The talk– The Italian experience – Some reminders on elicitation and a proposal for an

elicitation scheme – Some results belonging to an experimental

elicitation– Quality report and next steps in the research

IntroductionQ 2008

Roma, 10 June 2008

Up to now, expert opinion has been widely used to produce short term statistics on crops.

Main users:• Eurostat• System of National Accounts• Economic Operators

Summarizing, local authorities supply an evaluation of area and yield on the different crops (Jannuary: estimates on areas for winter cereals, May: estimates on area for maize, June estimates on yields for winter crops ,….).

ExampleQ 2008

Roma, 10 June 2008

Example: common wheat area (national level) according to the Farm Structure Survey (sample survey) and aggregated evaluations from local authorities

Common wheat area (ha)

0

100.000

200.000

300.000

400.000

500.000

600.000

700.000

800.000

1997 1998 1999 2000 2003 2005

FSS

Evaluations from local authorities

ExampleQ 2008

Roma, 10 June 2008

Example 2: maize area (national level) according to the Farm Structure Survey (sample survey) and aggregated evaluations from local authorities

Maize area (ha)

850000

900000

950000

1000000

1050000

1100000

1150000

1200000

1997 1998 1999 2000 2003 2005

FSS

Evaliations from local authorities

Why a survey on expert opinions?Q 2008

Roma, 10 June 2008

…a first judgement• Up to now in Italy these “statistics” seem to reproduce quite well estimates produced by sample surveys for many crops

…some other important strengths:• Timeliness (results available before traditional survey estimates)• Analytic data (estimates with a high geographical detail)• Inexpensive

Why a survey on expert opinions?Q 2008

Roma, 10 June 2008

… but also some weaknesses:

• Heterogeneity of the process at local level• Lacking of accountability• Problems in assessing data quality

In other words

• difficulties in defining and filling up a quality report

ProposalQ 2008

Roma, 10 June 2008

From evaluation to structured elicitation

To overcome these difficulties and to transform experts opinion in an additional tool for official statisticians useful to investigate phenomena difficult to observe with traditional surveys we propose to adopt a formal expert elicitation

“Expert elicitation in the context of uncertainty quantification aims at a credible and traceable account of specifying probabilistic information regarding uncertainty in a structured and documented way.” (Hora, 1992)

ProposalQ 2008

Roma, 10 June 2008

From evaluation to structured elicitation

Elicitation is currently and successfully applied in many experimental situations, as weather forecasting, biomedical applications, nuclear risks assessment, attributing foodborne pathogen illness to food consumption …

Protocols on the elicitation process have already been introduced in many research fields. These allow to evaluate the quality of the elicitation process, making the users aware about the use of the statistical results

…….a proposal of elicitation in the official statistics context

• 1. Organization

– Definition of the problem (and the questionnaire)– Finding out at least one expert and one facilitator– To train the experts and the facilitators

• 2. Elicitation

– Carry out the interviews– Transform expert opinions into probability distributions– Combine opinions of different experts (if available) – Feedback– Production of final results

ProposalQ 2008

Roma, 10 June 2008

• 3. Evaluation of the elicitation results (quality report)

–Timeliness, Coherence, Relevance, Comparability, Accessibility (as usual statistics)

–Accuracy

• Variability of elicited distributions (if more then one distribution have been elicited; this indicator should replace the usual variance)

• Effect of feedback (this should replace the indicators on effect of re-interview)

• Fiducial interval (this should replace the confidence interval)• Assessment of expert knowledge by means of control/seed

variables (this indicator should replace indicators on bias)

–Furthermore • Self assessment of expert knowledge and of his/her sources of

information (by expert)• Assessment of how the questions in the questionnaire are phrased

(by expert)• meta-information on the interview (by facilitator)

ProposalQ 2008

Roma, 10 June 2008

An experience: the saffron caseQ 2008

Roma, 10 June 2008

Step 1. OrganizationProblem definition: structure of saffron sector in Italy (Saffron is a rare crop, not observable by sample surveys)

Questions on the following phenomena: production and total area of saffron in Abruzzo, in Italy, in the world; export and import quantities; number of planted bulbs in the last year; forecast on production, number of operators,…

Selection of experts: president of the consortium of saffron producers in one of the two most important areas for the production of saffron in Italy (Navelli county)

Selection of facilitators: Istat personnel with strong agricultural background belonging to regional office

Training: document on how the elicitation process is conducted and on the main biases that can affect the elicitation process. Mail and phone contacts

An experience: the saffron caseQ 2008

Roma, 10 June 2008

Step 2. Elicitation

face to face Interview (about one hour) on

• Minimum, Maximum, Mode (most probable value) for each phenomenon of interest

• Distribution shape using fixed interval method (Ten “X” to be put in five intervals of equal length)

Check of data and first results• Check of data (compatibility, units measure, etc.,…)• Fitted distribution (Rectangular or Beta)• Point estimates• Fiducial interval

Feedback from expert and facilitator • Update of elicitation by expert and facilitator• Final results

An experience: the saffron caseQ 2008

Roma, 10 June 2008

Final result (concerning production of saffron in 2007)120 Kg is the point estimation (mode)(105 kg -130 kg) is the range within the production of saffron in 2007 lies with probability one for the expert111-126 Kg is the 95% fiducial interval according to the expert distribution

Expert info

Fitted distribution

Fiducial interval

Point estimate

Q 2008

Roma, 10 June 2008

Step 3. Evaluation of elicitation results Some indicators among those proposed before,

•Effect of feedback (expert confirmed his point of view after the production of the first report)•Fiducial intervals (e.g. 111-126 Kg for the production of saffron in Italy)•Self assessment of expert knowledge and of his/her sources of information

An experience: the saffron case

0

1

2

3

4

Confidence in thereference sources

degree of agreementwith other

experts/producers

Contacts with local andnational producers

statistical culture

Index (sum of points/max) 81,2%

Q 2008

Roma, 10 June 2008

Step 3. Evaluation of elicitation results

• Assessment of how the questions are phrased (by expert)

•Results on number of planted bulbs could produce misunderstanding if they are not integrated with information about type and quality of bulbs

• Meta-information on the interview (by facilitator):

•Expert declared and seemed to be honoured to collaborate with Istat•He appeared well trained and showed a true mastery of the topics •He collaborated during the feedback too•He has years of experience and has written many publications on the topic

An experience: the saffron case

Final summary and Future developmentsQ 2008

Roma, 10 June 2008

Summary1) The elicitation process formalizes the uncertainty of one or more

experts on particular phenomena2) Statistics based on experts have some good properties 3) In some cases expert opinion is the only available sources of

information4) It is possible to define protocols and quality reports to upgrade

“numerical evaluations” to statistics 5) some indicators have been proposed for accuracy6) A small experience has been (partially) illustrated

Future developments 1) Combination of elicited distribution belonging to two or more experts

(by feedback, mathematical aggregation, …)2) Combination of elicited distributions and data belonging to surveys

(linear combination, bayesian framework, ….., used as early estimates to be replaced when data are available).

3) Development of protocols for fields different from agriculture4) …..

Roma, 10 June 2008

Bibliography

Anthony O'Hagan, Caitlin E. Buck, Alireza Daneshkhah, J. Richard Eiser, Paul H. Garthwaite, David J. Jenkinson, Jeremy E. Oakley, Tim Rakow (2006) Uncertain Judgements: Eliciting Experts' Probabilities. Wiley

Di Bacco (1990). l’aggregazione di valutazioni diprobabilità: una rassegna…non imparziale , 8, ed. Pitagora. Bologna

Garthwaite P. H., Kadane J. B., and O'Hagan A. (2005): Statistical methods for eliciting probability distributions, Journal of the American Statistical Association, 100, 680-701.

ISTAT (1993): Manuale delle statistiche agricole rilevate con le tecniche estimative. Note e relazioni, 1993, 1. Roma: ISTAT.

Mortera J.(1990). “Aggregazione delle opinioni: una panoramica” Rassegna di metodi statistici ed applicazioni, 8, ed. Pitagora. Bologna

SHELF (Sheffield Elicitation Framework): http://www.tonyohagan.co.uk/shelf

J.P. van der Sluijs, P.H.M. Janssen, A.C. Petersen, P. Kloprogge, J.S. Risbey, W. Tuinstra, J.R. Ravetz (2004):Tool Catalogue of the RIVM/MNP Guidance for Uncertainty Assessment and Communication

Q 2008

Roma, 10 June 2008

Saffron: a flower – many recipies

Saffron is obtained from the yellow “core” of the flower Crocus. It is mainly produced in Iran and India, with minor productions in Spain, Greece, and Italy.Italy produces about 120 kg of saffron per year.

It is used in many regional recipes, as the Risotto alla Milanese

Q 2008