1. Value Proposition - The Society for Risk Analysis (SRA ...sra.org/sites/default/files/pdf/ARMSG_Principles... · Web viewThis document specifies a set of principles, guidelines

August 15, 2016 Page 1 of 14

Principles, Guidelines and Core Knowledge for Analytic Support of Risk Management

1. Value Proposition The field of risk management is in need of a set of clearly written principles that apply to all domains of application. That set of principles should provide the basis for a general climate of practice that all risk management analyses should comply with that set of principles, and provide a basis for Third Party Reviews. Funding agencies / clients should insist that those principles be followed, and provide adequate funding and scheduling such that the risk management analysts can comply with those principles. All Requests For Proposals and equivalent documents should specify that those principles be followed.

2. Overview This document specifies a set of principles, guidelines and core knowledge for analytic support of risk management. As such, this document is a quite extensive compendium of many different aspects of the area. In fact the nature of this document is to characterize the core knowledge in several different ways. As an example of that, we don’t get to our sections on principles and guidelines until Sections 8 and 10. That is because those principles and guidelines are embedded in the logic and content of the preceding sections, including: Value Proposition, Goals, Distinguishing Features, Modus Operandi, Core Values, and Challenges Addressed. The logical flow of those sections will become apparent as they are read. Then we go on to include sections on Mappings and Paths Forward. Finally, our approach to specifying other core knowledge, i.e., other than what is covered in the body of the text, is simply to include eleven appendices, each covering a different aspect. All in all, this document includes many lists at different levels. One could argue that the result is an untidy and unattractively large compendium. We argue that there is a need to gather all of these principles, guidelines and core values in one place, to establish consistency and aid in reference.

Taking that one step further, one could ask: “This sounds like many ways to argue over the number of angels that can dance on the head of a pin. Where does the rubber meet the road?” The rubber meets the road four places:1.) Where a risk management analysis practitioner refers to this document to make sure his or her

analysis is consistent with all of the principles, guidelines and core knowledge specified here.2.) Where a commissioner/client of any risk management or related analysis insists that any risk

management analysis he or she commissions is consistent with the principles, guidelines and core knowledge specified here, ideally as verified by a third party review.

3.) Where an oversight panel refers to this document to determine the degree to which its investigated risk management or related analysis is consistent with the principles, guidelines and core knowledge specified here.

4.) Where all of us risk management analysis practitioners, as a community, confer with each other to reach consensus on the principles, guidelines and core knowledge specified here.

Finally, a reasonable reaction to this document is that it seems far too ambitious in what it intends to cover. Our response: This document can be developed over as many years as it takes. In fact, much of the document is presented in a “fill in later” format. Though that said, we will design the process so that early steps, in fact all steps, will produce value along the way.

document.docx

8/15/16 Principles and Guidelines for Analytic Support of Risk Analysis & Management Page 2 of 14

Other Notes:- At times some text is followed by “[XX]” where XX are the initials of a contributor who suggested that

text.- Sections 12, 13 and 14, Paths Forward, Initial Pass at Writing Assignments, and Issues to be Addressed

in the August Webinar and the December Roundtable, are operational – to manage the generation of future versions of this document.

- <<Sidehat>> comments are inserted as comments to the panel of writers of this document.

The next page presents a Table of Contents.

document.docx


Table of ContentsContents1. Value Proposition......................................................................................................................................1

2. Overview...................................................................................................................................................1

3. Definition and Scope.................................................................................................................................4

4. Goals of This Document and the Project Supporting It............................................................................4

5. Distinguishing Features............................................................................................................................5

6. MO – Modus Operandi.............................................................................................................................5

7. Core Values...............................................................................................................................................6

8. Ten Principles...........................................................................................................................................6

9. The Challenges Addressed........................................................................................................................8

10. Guidelines...............................................................................................................................................9

11. Mappings, Graphics, Roadmaps...........................................................................................................10

12. Paths Forward.......................................................................................................................................10

13. Initial Pass at Writing Assignments......................................................................................................10

14. Issues to be Addressed in the August Webinar and the December Roundtable...................................11

15. Other Core Knowledge: Eleven Appendices.......................................................................................11

Appendix 1. Features of a “Good” Risk Analysis vs. a “Poor” Risk Analysis...........................................11

Appendix 2. A List of Problems and Pitfalls to be Addressed, Based on Experience................................12

Appendix 3. Misconceptions of Risk..........................................................................................................12

Appendix 4. Survey of Failures of Risk Assessment..................................................................................12

Appendix 5. Illustrative Examples Conveying the Importance of our Core Values, Principles & Guidelines....................................................................................................................................................12

Appendix 6. Examples of Issues of Trust and Acceptance.........................................................................12

Appendix 7. Evolution of Risk Assessment Models, And Future Directions............................................12

Appendix 8. Other Bold New Approaches to Risk Analysis and Management.........................................13

Appendix 9. Leveraging Work Of Other Groups, Tying Our Concepts Into A Larger Framework..........13

Appendix 10. A Catalog of Risk Management Standards..........................................................................13

Appendix 11. Defining Verification and Validation...................................................................................14

document.docx


3. Definition and Scope We title this document with the words “Analytic Support of Risk Management.” That is to orient the contents to the management of risk, as opposed to simply its analysis, etc. In fact the scope of this document extends to all analytic aspects related to, in support of, risk management: risk identification, research, analysis, assessment, evaluation, communication, decision making, and management. All of those analytic steps generate information to be used to inform risk management decisions. Rather than bog us down with long lists each time we discuss the area covered, we will summarize that list as risk analysis, and refer to it in terms adjusted to context in different parts of this document. The scope of this document also extends to all domains of application. Those domains are listed, in one suggested taxonomy, in Section 10, but we also present that suggested taxonomy here: health, environment, terrorism, infrastructure, engineered systems, cyber security, cyber-physical systems (e.g. drones, driverless cars), natural hazards, asset management, finance, governance, foreign policy, military, crime, insurance (life, health, property, etc.), project risk, and any other domains contributors suggest. As the two previous lists make clear, even how best to define and title those aspects and domains is subject to argument.

4. Goals of This Document and the Project Supporting It <<Is this too many goals? Are many of these goals too wildly optimistic and/or call for actions/authority

above our pay grade? If you think so, suggest remedies.>>These goals are ordered generally from the top down, from the most strategic to successively more

tactical support for the goals above. That is, each goal is in support of the goals above it.While that pattern is not fully linear, it generally follows.

G1. To achieve the most effective risk management possible,through applying the most effective analytic support of that risk management possible,in all areas of analytic support of risk management.

G2. To support, encourage and enforce, as much as possible, analytic support of risk managementthat makes the best use of available data, SME judgments, assumptions and analysesto most effectively advise risk management decisions.

G3. To establish a system of principles and guidelines for the practice of analytic support of risk management, those principles and guidelines recognized as a single, universally applicable set of principles and guidelines.

G4. To enable the shared understanding and communication of that system of principles and guidelines, including the technical language necessary for meaningful and universally consistent application of those principles and guidelines, and for meaningful and universally consistent testing for compliance with those principles and guidelines.

G5. To create an environment and culture of “Analysis Quality,” where risk management analysts design and conduct their analyses explicitly to be in compliance with those principles and guidelines.

G6. To create an environment and culture of “Analysis Quality,” where risk managers using those analyses insist that those analyses be in compliance with those principles and guidelines.The ideal, here, is a world in which any analysis in support of significant risk management decisions is subjected to tests, ideally third party tests, of the compliance of that analysis with the principles and guidelines presented here.

document.docx


G7. To establish a procedural framework where funders and commissioners of risk analyses expect those analyses to comply with the principles specified in this document, as determined by some form of peer review.

G8. To establish a procedural framework where funders and commissioners of risk analyses allocate adequate budget and schedule both for the risk analysis itself and the associated peer review.

G9. To enable some system(s) by which analyses in support of risk management decisions can be tested against those principles. This goal includes necessary sub-goals of transparency and documentation.

G10. That procedural framework shall be designed to specifically counter the pattern, often observed, where risk assessment is a “check the box” activity. [SP]

G11. Establish a knowledge base among the funders/commissioners/users, perhaps as a very basic 1 or 2 page widely disseminated document that establishes:- a proper understanding of what risk analysis is about;- a proper understanding of how risk analysis is supposed to assist in decision making. [FP]

5. Distinguishing Features The two distinguishing features of the work developed here both follow from the orientation of the specialty group authoring this work, the Applied Risk Management Specialty Group, first operative word “Applied,” second operative words “Risk Management:”Focus on decision support: The fundamental test behind each principle and guideline presented here is

“What is necessary for the most effective decision support?” That is as opposed to obscure academic issues “deep inside” sophisticated calculations and modeling processes. Those issues are certainly necessary to be considered, but the ultimate standard is what best supports the risk management decisions that must be made.

Focus on “in the trenches” experience: We have found that often the key issues in support of risk management decisions concern practical considerations such as coping with lack of data, lack of validated data, communication into a less-than-ideal decision process, and communication with non-specialists. While more academic concerns are also important, we have found that the more practical considerations are just as important if not more so. As a consequence of this focus, we will pay particular attention to “Reports From the Trenches” from actual analysis practitioners concerning their real world experiences of attempting to support the best possible risk management in what is often markedly less-than-ideal decision support situations.

6. MO – Modus Operandi Iterative Learning: The project of establishing the principles and guidelines satisfying the above goal

structure is a massive undertaking. It can best be accomplished in an iterative process, where successive drafts are generated then circulated for review and comment, and where those drafts become progressively more specific in terms of principles and guidelines applicable to more specific domains of risk management support.

Learning as a Community Process: No one group of analysts has the expertise and field experience called for to specify the principles and guidelines called for, in particular as those principles and guidelines become more specific to be applied to more specific domains of application, i.e. domains of risk

document.docx


management support. Therefore the iterative process described above should include the broadest possible participation by practitioners and users of analyses in support of risk management and, as the principles and guidelines become more specific to specific domains of application, it should include successive recruitment of specialists in those more specific domains.

7. Core Values <<A key issue in standards/etc. is which words to use among the set: “Should, Shall, Must, May.” We

have used “should” here, perhaps OK for “Core Values”. Or not. We should settle that issue.>>

CV1. Analysis in support of risk management should bring the power of analysis to bear to apply what is known, including all uncertainties, and consider what is not known, to generate the most effective guidance for risk management possible.

CV2. Analysis in support of risk management should provide honest, transparent risk management advice independent of vested interests.

CV3. Analysis in support of risk management should be sufficiently comprehensive for its purpose.CV4. Analysis in support of risk management and any associated models should be adequately verified

and validated.CV5. Analysis in support of risk management and any associated models should be effectively peer

reviewed.CV6. Analysis in support of risk management and any associated models should include a consideration

of the risk of improper analysis. [This list started by RW then refined]

8. Ten Principles <<These principles cover some of the same ground as do the “Goals” listed in Section 5. Is that bad? Should we try to fix that?>>We here present a first draft of principles, based on our own experience. We also propose to embark on a parallel effort to identify the seminal references, compile all principles listed in those references, and subject that list to peer review. The principles that follow overlap in some ways. We found that the clearest way to present these principles is in their current form, including overlaps. Those overlaps can be intelligently considered in applying the principles. The first two principles are expanded by all of the principles below them.P1. The single overriding principle:

Make the best use of all available data and analyses to advise risk management decisions.Analyses in support of risk management decisions should be conducted such that they bring the available data, including SME judgments, assumptions and analysis tools most effectively to bear to advise those risk management decisions. That includes the clarity of that guidance for non-specialist risk managers, and appropriately couching that guidance with the limitations of data, scope and analyses involved. Principles P2 – P10 spell out particular aspects of this first principle:

P2. Analyses must be effective. That is, they must be engaged effectively in the risk management decision process, as opposed to be simply analyses conducted in isolation from that process.Principles P3 – P10 spell out particular aspects of this second principle.

document.docx


P3. An essential element of that effectiveness is trust, that is, all associated decision makers (i.e. users) and stakeholders must trust that all activities of the risk assessment have been conducted with full transparency and explanation, and have complied with all of the principles listed here. “Externalities” (e.g. process) of the assessment, which may focus on models, data, expert judgments, etc. can be very important for people to actually want to use the assessment in decision making, and so affect its effectiveness. [FG]

P4. Results should be formatted and presented such that they are meaningful to risk managers who may not be specialists in risk assessment/analysis/evaluation. “Meaningful” means such that those managers can apply those results validly to advise their risk management decision(s), including balancing all of the risk considerations within the analysis with all other decision factors.

P4a. One aspect of that: Results should be formatted in units that non-specialists can validly trade off against other aspects of their decision alternatives. For example, “Confidence Factor” may be a scale meaningful to a risk analysis specialist, but may not be meaningfully to a non-specialist risk manager, i.e. meaningful enough to validly trade off that attribute against, e.g., cost.

P4b. Another aspect of that: Results should be formatted in units that reflect the actual level of precision and accuracy involved. A common pitfall is to present results such as relative risks in bar charts, which are read as ratio-scale data, which is only valid if in fact those risks are evaluated on a ratio scale. Another pitfall is to present results to, e.g. three significant digits when in fact they may only be valid to one significant digit. A pitfall related to that is to present results without error bars when error bars are called for, or with error bars that are too narrow, or without specifying the confidence intervals spanned by those error bars.

P5. Timeliness: The analysis and results should be conducted and presented along a timeline such that they support the risk management decisions to be made. The pitfall addressed here: Often risk management analyses are conducted too late to genuinely advise the decisions they are purported to advise. In those cases one danger is that the analysis is conducted simply to defend a decision already made. That leads to the next principle:

P6. Adjust the analysis to the actual, as opposed to the announced, role it is to play, and practice full disclosure of the role it is to play. For example, as just noted, if the analysis is in fact being conducted only to defend a decision already made, that can be a valid role for the analysis, but it must then be conducted in such a way that it is a genuine test of the defensibility of the decision, as opposed to an analysis slanted in the direction of supporting the decision, and the reader should be fully aware of that role. Related to that:

P7. The analysis should include language, clearly understandable to the non-specialist and prominently displayed in the first text, that explicitly presents the actual role of the analysis. For example, if it is to advise a decision, then the alternatives considered outlined and the criteria for the evaluation should be outlined, both for later more complete presentation. If it is to defend a decision already made, then it is to be announced as such, and the criteria for that defense outlined, for later more complete presentation.

P8. Other roles for the analysis should be announced clearly. P6 and P7 are framed to highlight a central question of the role of the analysis, decision advice vs. defense of a decision already made. In addition, there are other roles to be considered and that also should be announced in the opening pages of the report. Among those roles:- Visualization, i.e. to make the benefits and/or costs of alternatives more clear and salient to the risk

managers and in some cases the stakeholders and impactees.

document.docx


- Facilitate communication among all parties, including the analysis team, between SMEs, analysts and risk-managers, and with the public, stakeholders and impactees.

- Guide further research and analysis.P9. Reports must clearly state assumptions and caveats,

and the implications of those assumptions and caveats for using the results to advise decisions,in terms understandable to the actual decision makers / risk managers. In addition, those statements of assumptions and caveats must be placed immediately next to results, including results text, numbers, graphics and tables such that the risk managers understand:- the limitations of the analysis and associated caveats, and in addition, in particular:- the scope, and the implications of that scope for interpreting and applying the results,- the assumptions, and the implications of those assumptions for interpreting and applying the results,- the data limitations, and the implications of those limitations for interpreting and applying the

results.We have found cases where some of those scope, assumptions and data limitations are described in some places in the report, but analysis users often do not carefully integrate all parts of the report, and tend to focus solely on the results text/pages/graphics, and so do not incorporate information on the scope/assumption/data limitations appropriately in their interpretation and application of the results to advise their decision(s).

P10. Aggregating over P6 – P9: Full Disclosure. That is, in the real world, with real world challenges and budget, schedule and data limitations, analyses to support risk management must often be less than ideal. We recognize that. The point is not that all analyses should be conducted at an ideal level, but rather that the degree to which, and ways in which, the analysis is less than ideal, and the implications of those shortfalls, should be fully disclosed to the risk managers, and the implications of those shortfalls clearly spelled out such that risk managers understand them and can validly apply that information to their interpretation and use of the results in advising their risk management decisions.

9. The Challenges Addressed As mentioned, the nature of this document is to characterize the principles, guidelines and core knowledge in several different ways. So here we characterize that in terms of challenges addressed. In particular, any risk management or related analysis should specifically address all of these challenges that apply, and to specify how each of those challenges is addressed. As with most of this document, this list is only meant to begin the discussion. In later versions we will interleave these challenges with the principles needed to meet each challenge. We will also incorporate key existing references, e.g. the IRGC’s “Top Deficits in Risk Management.”

C1. Capturing the risk generation process and its complexity. Identifying what aspects of that process are known, what aspects are knowable, and what aspects are unknowable. Then developing risk management advice that validly accounts for that process, its complexity, and what aspects are known, knowable and unknowable.

C2. A related challenge, or in many cases the same challenge expressed differently: Characterizing the risk event space, the scenario space. That includes characterizing the completeness with which that space is known, and the unknowable parts of that space. Then developing risk management

document.docx


advice that validly accounts for that process, its complexity, and what aspects are known, knowable and unknowable.

C3. Reducing large amounts of data down to effective decision guidance in a way that is valid and reviewable by a third party.

C4. Assessing the uncertainties.

C5. Taking those uncertainties into account in risk management. Note that Challenges 6 - 8 below address this challenge more specifically.

C6. Preparedness for scenarios “not on the list.” That is, recognizing cases where scenarios may occur that are not on the list of scenarios assumed by the analysis, and considering preparedness for those scenarios.

C7. Developing robust risk management strategies.

C8. Developing resilient risk management strategies.

C9. Setting an adequate budget to achieve consistency with the principles, guidelines and core knowledge specified here.

C10. Appropriately choosing among and applying the most appropriate analysis, among the many analyses that could be applied.

C11. If the risk involves an adversary, e.g. terrorist, criminal, government, competitor, then modeling that adversary in a way that captures intelligent and adaptive behavior.

C12. Data availability, collection.

C13. Data validation.

C14. Data management.

C15. Data Quality Assurance and Quality Control.

C16. Model validation.

C17. Model documentation.

C18. Model communication.

C19. Decision process validation.

C20. Decision process documentation.

C21. Decision process communication.

10. Guidelines We propose that guidelines are to be developed later specific to each field of application, and in fact developed by practitioners in each field. For now, then, we will simply list our initial list of fields of application, as subsection titles:

10.1 Health

10.2 Environment

document.docx


10.3 Terrorism

10.4 Infrastructure

10.5 Engineered Systems

10.6 Cyber Security

10.7 Cyber-Physical systems, e.g. drones and driverless cars

10.8 Natural Hazards

10.9 Asset Management

10.10 Finance

10.11 Governance

10.12 Foreign Policy

10.13 Military

10.14 Crime

10.15 Insurance: Life, Health, Property, etc.

10.16 Project Risk

10.XX Any other domains contributors suggest

11. Mappings, Graphics, Roadmaps Carl Dister and Rob Waller have suggested mapping and graphics methodologies to help visualize relationships between key concepts. One example: Concept Mapping. [RW]They may be providing those.

12. Paths Forward Respond to the current draft, including scathing critiques and extensive editing as called for.Prepare issues to be addressed in the August Webinar and December Roundtable (started in Section 14).Suggest how best to proceed.Specify writing assignments (started in Section 13).Volunteer to perform some of those writing assignments.Set up some mechanism for enlisting support for writing Guidelines for each domain of application.

13. Initial Pass at Writing Assignments Review Risk Analysis for related articles, and develop text for us such that

we are not inconsistent with those articles, and cite them and use them as appropriate.Review the INCOSE 14 standards (and the other standards cited by Carl Dister) (See Appendix 10),

develop text such that we are not inconsistent with those standards,and cite them and use them as appropriate.

document.docx


Consider reviewing and tabulating those standards in some formsuch that we can draw inferences from those several standards for our work.

First drafts of any Guidelines for any area of application.Carl Dister and Rob Waller, could you please develop a draft of Section 12?

14. Issues to be Addressed in the August Webinar and the December Roundtable Are we OK with the scope presented in the first paragraph of the doc? If not, what would be better?The document has evolved into a very byzantine structure. Is that OK? If not, how best solve?Are we happy with the structure of Principles and Goals applying generally,

then separate Guidelines for each area of application?One variant: Some principles apply more to some domains of application more than others.

We can structure that.How should we organize the structure of Guidelines for each area of application vs. Challenges?Note that we hope that Guidelines can be written by specialists within each area of application.

Is that just a hopeless idea, with too high a failure rate?If we are going to do that, how do we engage other organizations/Do we do the hard work of integrating all existing standards? For example the “INCOSE 14”?Do we attempt to push for some framework with which to enforce these Principles/Guidelines?One idea: Do we attempt to push for some organizational framework for review of analyses?

For example an “ARMSG Seal of Approval”?Do we include Post Mortems, Case Studies?

15. Other Core Knowledge: Eleven Appendices We have many other topics that we want to include as core knowledge that we, frankly, could not fit into any framework such as Sections 1 - 14. For those, we simply include them as appendices. As with Section 10 (Guidelines), we would expect these appendices to be written by several different people over a period of time. One reaction to this list of appendices is that it has a certain hodgepodge character.To which we respond:1.) There’s nothing terribly wrong with that, and there is something to be said for gathering all of these

concepts in one place.2.) As we develop each of these appendices, patterns and overlaps may become clear, so that we can

organize the concepts of these appendices more effectively than we can as of now.

Appendix 1. Features of a “Good” Risk Analysis vs. a “Poor” Risk Analysis [FG] Mapped into our principles, etc.

One aspect of this could be a user friendly (client friendly) framework, table, etc. that non-specialist clients could use, keyed to his or her domain of application, to easily:

- check to see if a given risk analysis method is “State of Art,” “accepted” or “under consideration,”including concepts of maturity models, for that domain [SP, CD]

- compare one risk assessment with another for that domain,including the pros and cons of each one [SP]

document.docx


- relate the above two issues to the complexity of the system of risks involved [CD]- become aware of unmet challenges and future directions. [SP]

Appendix 2. A List of Problems and Pitfalls to be Addressed, Based on Experience Perhaps cross reference with the Challenges of Section 10.

Appendix 3. Misconceptions of Risk Here we could include a summary of Terje Aven’s excellent book of that title, where he lists 19 thoughtful misconceptions . Then we can map that into our principles, etc. [CD]

Appendix 4. Survey of Failures of Risk Assessment - These should be actual illustrative examples conveying the importance of our core values, principles and

guidelines.- These should include how those failures might have been avoided by adherence to the principles etc. of

this document. In fact those failures could be mapped into the principles that apply. [RW]- The International Risk Group has defined common mistakes in risk governance that should be

incorporated here by reference. [SA]

Appendix 5. Illustrative Examples Conveying the Importance of our Core Values, Principles & GuidelinesThis is a separate appendix covering the same concepts as the first bullet of Appendix X,

but with examples not limited to failures. [RW]

Appendix 6. Examples of Issues of Trust and Acceptance The examples should make clear how important those issues are, and how they relate to process and the transparency surrounding any risk assessment. This appendix can explore on what grounds people trust statements by others. This may be a bit contentious because trusting that something is good does not guarantee that the work actually is good. Nonetheless, the issue of trust may be quite important to consider, in the sense that “externalities” (e.g. process) of the assessment, which may focus on models, data, expert judgments, etc. can be very important for people to actually want to use the assessment in decision making, and so affect its effectiveness. [FG]

Appendix 7. Evolution of Risk Assessment Models, And Future Directions For each domain of application listed in Section 10, how risk assessment has evolved in that domain, and how far it still has to go, i.e. discussing current shortfalls but then taking an optimistic view of the possibilities for improvement. We could use this format to describe the concept of “maturity models” of risk assessment. We would see if there are common basic principles guiding that evolution in all domains. From that work, we can try to develop basic principles, and to form a universal/grand theory of the evolution of risk assessment that applies to all the domains. [SP]

An important part of this appendix:A roadmap of where risk analysis and management has evolved, to where it is going in the various domains of application, including perhaps an approximate timeline. This could be in part modeled on the

document.docx


example of the EPA Next Gen program. [CD] Though throughout this appendix, the discussion should be quite directly and explicitly focused on our principles and guidelines.

Appendix 8. Other Bold New Approaches to Risk Analysis and Management

Appendix 9. Leveraging Work Of Other Groups, Tying Our Concepts Into A Larger Framework Examples we could consider:- INCOSE (www.incose.org) has developed principles of addressing complexity in systems,

which Risk Analysis/Risk Management is an example of. [CD]- Formal “philosophy of systems” work going on which we could possibly leverage, as applied to risk.

See http://systemology.org/manifesto.html . [CD, RW]- “A Framework for the Next Generation of Risk Science,” Kreski et al,

Environmental Health Perspectives, Volume 122, number 8, August 2014 [PL]- The EPA Next Gen program. [CD]- The IRGC Risk Management Escalator, mapped to risk management practices

in each domain of application. Then we can map that into our principles, etc. [CD]- Connecting risk analysis and management with systems engineering and systems thinking. [CD]- Alternative procedures for auditing for compliance with our principles.

We could consider the role auditors play in risk assessment as they perform various parts of validation and perhaps verification in their work, and how the different domains of application audit differently (i.e. Risk Based Auditing). [CD]

Appendix 10. A Catalog of Risk Management Standards A good short term goal will be to simply compile a list of seminal papers and guidelines. As an example we copy in a table from INCOSE 2006 [CD]. We would also include ISO 31000 and ISO/IEC 15288 (referenced in NIST-800-53) [CD], as well as risk governance documents by the International Risk Governance Council and other documents from EPA and DOT.

document.docx

http://systemology.org/manifesto.html

http://www.incose.org/


Appendix 11. Defining Verification and Validation Much has been written about Verification and Validation. Here we briefly summarize:Verification is: Is the model/analysis calculationally correct in mapping from

scope/data/assumptions/SME judgments to the results, including appropriately caveating those results?Validation is: (Definitions vary, but in ways that can be broken down into two subsets:)

- Is the model consistent with the “real world,” observed events, etc., sometimes simply assessed by correlation between past model results and what actually happened. The iconic case: Predicting rain the next day, so you just correlate past records of predictions and rain.

- Does the model fulfill its mission to best advise the decisions it is to advise. This is what you use when you don’t have observed events. Barry Ezell and I have two papers/chapters on that topic, but I don’t want to pollute your thinking with our thinking.

In further work we could include a citation for a more extensive treatment.

document.docx

Documents

1. Value Proposition - The Society for Risk Analysis (SRA ...sra.org/sites/default/files/pdf/ARMSG_Principles... · Web viewThis document specifies a set of principles, guidelines