CIHR Team in CIHR Team in MicrosimulationMicrosimulationSSimulation imulation TTechnology for echnology for AApplied pplied RResearch (STAR)esearch (STAR)

Montreal, November 24-25, 2009Montreal, November 24-25, 2009

Goal of the MeetingGoal of the Meeting

• To develop a detailed plan for achieving the objectives of the project, specifically:– To identify the investigators responsible for

each objective/subproject – To specify the final products (software,

publications, course syllabus, etc.)– To specify milestones and deadlines for each

product

NET Vision NET Vision

• The end-product of this 5-year research program will be – a set of integrated, validated, transparent, and

user-friendly disease simulation models – widely known and accessible to policy-makers

and researchers within Canada and internationally

– supported by extensive documentation and novel substantive results published in highly respected scientific journals

NET subprojects NET subprojects

• Model-specific subprojects

• General subprojects

• KTE-related subprojects

• Training-related subprojects

• The book subproject

Model-specific subprojects Model-specific subprojects

• Breast cancer model: Validation? • Colon cancer model: Validation? • Diabetes model: Development/validation • CHD model: Modification/Validation • OA model: Further development/validation

sensitivity analysis, applicationsFor each subproject:

– Final product from the NET – Person responsible– Milestones and deadlines– KTE aspects

General subprojects General subprojects

• Validation framework and methods review• Macrosimulation ontology• Software development • Integration of POHEM• Application of integrated model: obesity • Application of integrated model: health

inequalitiesFor each subproject:

– Final product from the NET – Person responsible– Milestones and deadlines– KTE aspects

KTE-related subprojects KTE-related subprojects

• NET website • NET repository• POHEM model documentation • Advisory committee (policy makers) For each subproject:

– Final product from the NET – Person responsible– Milestones and deadlines

Training Training

• Trainees and trainee awards – New investigators– Post-doc fellows– PhD students – Master’s students– Summer students– Other

• Course on microsimulation – Final product from the NET – Person responsible– Milestones and deadlines– KTE aspects

The Book The Book

• Introduction to epidemiological simulation models• Types of models, examples of models • Microsimulation ontology• Model development and validation framework • Sensitivity analysis • Statistical issues • General description of POHEM • Disease-specific POHEM models • Multi-disease POHEM model • Examples of applications • The future of modeling

POHEM-OAPOHEM-OA

Age & Sex(CCHS 01)

BMI(CCHS)

OA Diagnosis

HRQOL(CCHS 2001)

RegionIncome

Education(CCHS)

Crude ratesBCLHD

Regression modelNPHS 1994-2004

Incidence modelNPHS 2000-02

Tobit modelCCHS 2001

OA Surgery

OA Drugs

Tobit modelVGH 2007

RegressionNPHS

Crude ratesBCLHD

Crude ratesBCLHD

OA stageBCLHD

Effect of drugsLiterature

DirectCosts

Cost model

Cost model

Cost model

Side-effects

Co-morbidity

Death

Other riskfactors

Indirectcosts

No model at this time

No model at this time

Literature

No model at this time

No model at this time

No model at this time

POHEM-OAPOHEM-OA

Literature

POHEM-OA – update POHEM-OA – update

• Admin database linked to CCHS (for parameter validation): data for 1991-2004 received, preliminary analyses initiated, waiting for data update (to 2008)

• Parameter validation in Ontario and Quebec: preliminary discussions completed, SAS codes will be ready to send out in December/January

• Sensitivity analysis: methodology partially developed, will continue next year

• Cost module: work will start in January• Obesity application: detailed plan developed, actual

simulations to start in December• OA treatment application: preliminary plan developed,

simulations to start next year (January/February)

POHEM-OA publication updatePOHEM-OA publication update

Journal publications• Kopec JA, Sayre EC, Flanagan W, Fines P, et al. Cibere J, Rahman M, Bansback

N, Anis AH, Jordan JM, Sobolev B, Aghajanian J, Kang W, Greidanus NV, Garbuz DS, Hawker GA, Badley EM. Development of a population-based microsimulation model of osteoarthritis in Canada. Osteoarthritis Cartilage (in press)

Presentations and abstracts • Sayre EC, Finès R, Flanagan WM, Rahman MM, Kang W, Cibere J, Anis AH,

Badley EM, Kopec JA. A Tobit model for predicting Health Utilities Index Mark 3 from osteoarthritis disease duration: a population-based study. To be presented at the Annual Scientific Meeting of the American College of Rheumatology, Philadelphia, October 16-21, 2009

• Kopec JA, Finès P, Flanagan WM, Sayre EC, Rahman M, Bansback N, Cibere J, Anis H, Jordan JM, Badley EM. Projecting the burden of osteoarthritis in Canada using microsimulation. Presented at the Annual Meeting of the European League Against Rheumatism, Copenhagen, June 10-13, 2009.

• Finès P, Kopec JA, Flanagan WM, Sayre EC, Rahman M. Microsimulation of osteoarthritis in Canada – Case study of a chronic disease in Canada. Presented at the Meeting of the International Microsimulation Association, Ottawa, June 8-10, 2009.

Model validation: Model validation: Conceptual issues Conceptual issues

AgendaAgenda

• Validation framework• Validation principles • Question re: validity evidence from examining

model development • Questions re: validity evidence from examining

model output • Rating of validity evidence • Validation-related subprojects

Validation framework:Validation framework:Sources of validity evidenceSources of validity evidence

• Evidence from examining model development Conceptual model validity (theories, definitions, content, structure); Parameter validity (parameters based on expert opinion, literature, data analysis, databases, calibration); Computer program validity (type of simulation, software, code, internal organization)

• Evidence from examining model outputPlausibility (face validity); Internal consistency; Parameter sensitivity; Between-model comparisons; Comparisons with external data;

• Evidence from examining the consequences of model-based decisions

Validation principlesValidation principles

• Models gain credibility through thorough development, extensive validation, and use

• Full and complete model validation is never possible, validation never ends

• A model can be valid (and validated) for one application and not valid (or validated) for another; for example, a model may be valid as an aid to decision making, but not as a forecasting tool, and vice versa

• Epidemiological microsimulation models such as POHEM models are developed for multiple purposes and should be validated accordingly

Validation principles – cont.Validation principles – cont.

• Model validation studies are of relatively high interest and should not be too difficult to publish

• Published validation studies tend to increase model uptake by researchers

• Validity evidence based on examining model development process can/should be part of model description

• Probably the most powerful validation studies (but also the most difficult to do) are sensitivity analysis, between-model comparisons, and validations against external data

• Given the time and other constrains, we need to strike the right balance between model validation and applications

Evidence from examining model Evidence from examining model development development

• Question re: Evidence of conceptual model validity, parameter validity and computer program validity – Should we try to include this type of validity evidence in

all papers describing POHEM models? – How extensive should this evidence be? Should we

follow our own guidelines/framework? – What other documentation should we develop to include

those results that are not published or publishable?

Evidence from examining model Evidence from examining model outputoutput

Questions re: parameter sensitivity, between-modelcomparisons, and comparisons with external data

– Should this type of validation be part of the NET? – Is this type of validation equally important for all

models? – Are all these sources of validity evidence equally

important and feasible? – Do we know exactly how to do it? – If we do it, should we aim to publish all the results and if

not, what other documentation should we develop to include those results that are not published?

Rating of validity evidence Rating of validity evidence

• How many aspects have been validated? • How detailed and transparent is the description of

the validation of each aspect? • How extensive is the validation of each aspect

(many different approaches)? • How quantitative is the validation of each

quantitative aspect?

Validation-related subprojectsValidation-related subprojects

• Model validation framework • Sensitivity analysis – a review• Disease-specific POHEM models – further

development/validation (breast, colon, CHD, diabetes, OA)

• Multi-disease POHEM model – description and validation

Example:Example:Model validation frameworkModel validation framework

• Final product: paper/chapter

• Person responsible: Jacek

• Deadline: February 2010 (submission)

• KTE: publication

Training Training

Agenda Agenda

• Trainee awards

• Course development

Trainee awards Trainee awards

• How many• What type• How much • When• For how long • For what subprojects• Review of proposals

Course developmentCourse development

• Audience • Content • Level • Delivery • Availability• Persons responsible • Milestones and deadline • KTE aspect