ISI 2009 Short Course #12Professional Ethics

Part II. Professional Ethics and the work of a National Statistical Office

William Seltzer

(Fordham University)

seltzer@fordham.edu

© 2009 William Seltzer

All rights reserved

2

Outline of Part II

• A. Introduction

• B. General ethical issues not covered in any detail in part II (Assumed they are covered in some detail in part I)

• C. Ethical issues of special relevance to the work of a national statistical office

3

Introduction: Ethical issues to be covered in some detail

• Law, policy, and ethics: overlaps, differences, and sources of authority

• Integrity of the statistical agency and its outputs

• Use of very out-of-date methods

• Statistical confidentiality

• Administration of a statistical agency

• Prevention and coping strategies and tools

4

Introduction:Mechanics and sources

• Duration: About 90-120 minutes

• Breaks: To be decided

• Questions: After each sub-section

• One group problem-solving exercise

• A general bibliography is given beginning at slide # 55 of this PowerPoint presentation.

5

Introduction: Scope of Part II

• Although the title of part II refers to ethics and the work of a National Statistical Office (NSO), virtually all the principles and views expressed in my presentation refer equally to any governmental statistical agency

• Accordingly, I will use the terms NSO and government statistical agency interchangeably

6

Introduction: Introductions

• Please introduce yourselves, saying a few words about your experience, if any, with a national statistical office or government statistical agency.

• Also say a few words about what you hope to get out of this part of the course.

• Any questions so far?

7

B. General ethical issues NOT covered in any detail in part II

• Need to balance conflicting ethical principles

• Different approaches to ethical norms

• Need for perspective: not every difference of opinion about methods, analysis, or conclusions is really an ethical dispute

• Plagiarism

• Responsibility to cite sources and describe methods and quality

8

1. Need to balance conflicting ethical principles

• Different ethical principles may conflict with one another

For example, in the political field there is often the apparent difficulty of

reconciling calls for peace with those for justice

• There is often no single right way to reconcile conflicting ethical principles

9

2. Different approaches to ethical norms

There are several different ways of understanding ethical norms. Three widely recognized approaches are:

• Deontological (Absolute rules. For example, the 10 Commandments)

• Utilitarian (For example, the greatest good for the greatest number)

• Rawlsian (High priority given to social justice)

10

3. The importance of perspective

• Of course, ethics are important

• However, not every difference of opinion about methods, analysis, or conclusions is really an ethical dispute

• Importance of being aware of consequences for many different stakeholders, including the powerless and vulnerable

11

4. Plagiarism

• One doesn’t take the work of others and pretend you are the originator or author

• This holds true for the staff and leadership of a national statistical office and for the organization itself

12

5. Responsibility to cite sources and describe methods and quality

• Good science requires that sources and methods are clearly and fully described and this requirement is reflected in the ethics documents of most professional societies.

• Of course, NSO outputs are released in a variety ways. The key is to balance need to make information on methods, sources, and quality available to data users in ways that different kinds of users can access.

13

B. General ethical issues not covered in any detail in part II

• Any questions?

14

C. Ethical issues addressed in some detail in this course

1. Law, policy, and ethics: overlaps, differences, and sources of authority

2. Integrity of the statistical agency and its outputs

3. Use of very out-of-date methods

4. Statistical confidentiality

5. Administration of a statistical agency

6. Prevention and coping strategies and tools

15

1. Law, policy, and ethics: overlaps, differences, and sources

of authority

• When working in a government statistical agency, we have to serve many masters. They include the following persons:

i. Your immediate supervisors

ii. Their supervisors

iii. The agency head (or if you are the head of the agency, the person or

committee to which you report)

16

1. Law, policy, and ethics -- continued

• We must also serve several different kinds of impersonal authorities. They include:

i. The law (those explicitly pertaining to statistics and other laws)

ii. Government and agency policies

iii. Scientific standards (statistical science and relevant subject-matter science)

iv. The needs and interests of data users

v. Professional ethics (statistical and relevant subject-matter professions)

17

1. Law, policy, and ethics -- continued

Sources of authority1. Statistical policy

UN Statistical CommissionEurostat

2. Statistical ethicsFundamental Principles of Official

StatisticsISI Code of Professional EthicsEthical standards of various national

statistical professional societies

18

1. Law, policy, and ethics -- continued

• Usually, the dictates of the law, sound scientific standards, agency policies, the instructions of supervisors, and ethics all point in the same direction

• At times, however, these personal and impersonal masters may be in conflict and pull us in different directions

• Note that simply following the law or following orders is no excuse for all sorts of crimes, including human rights crimes, such as genocide

19

1. Law, policy, and ethics -- concluded

• Can any of you cite an example in your career where you or the agency in which you worked had to deal with conflicts among two or more of these authorities?

• How were the issues addressed or

resolved?

• Any questions?

20

2. Integrity of agency and outputs

(a) Policy

(b) Staffing

(c) Fixed release dates

(d) Suppression of results

21

2. Integrity of agency and outputs:(a) Agency policies

• Agency policies should be based largely on statistical and substantive grounds taking into account political priorities (broadly defined)

• Be guided by the interests of the full range of data users and the responding public

• Resist efforts to let short-term partisan political priorities drive policy decisions on structure, staffing, methods, and outputs

22

2. Integrity of agency and outputs: (b) Staffing

• Qualifications and experience

• Professional outlook and sense of responsibility

• Broadly reflecting the diversity of the country

• Identification with the agency and the statistical profession

• Nevertheless, the hiring of otherwise qualified political referrals is ok

23

2. Integrity of agency and outputs: (c) Fixed release dates

• Release dates for politically-sensitive statistical series (for example, release of the consumer price index or the unemployment rate) should be scheduled well in advance

• This schedule should be adhered to and the release dates for such series should not be determined by short-term political considerations

24

2. Integrity of agency and outputs: (d) Suppression of results

• Planned statistical outputs should always be released, unless there are strong professional reasons for not doing so

• If a planned output is not released, a detailed methodological output should be issued in its place, documenting the shortcomings in the planned output and presenting at least summary results

• Any questions?

25

3. Use of very out-of-date methods

• The failure to up-date methodologies to take into account evolving scientific understandings or the changed circumstances of the country and its population if delayed too long can become a serious ethical issue

• We are not talking here about 5 or 10 year delays but those extending to 50 or 100 years

• Any questions?

26

4. Statistical confidentiality

• The confidentiality of personal information provided to government statistical authorities during population censuses and similar statistical operations is a well-established principle, at least among statisticians.

27

4. Statistical confidentiality - continued

• Most sets of ethical norms in the field of statistics base the responsibility for maintaining statistical confidentiality, either explicitly or implicitly, on the grounds that such disclosures may lead to respondent harm.

28

4. Statistical confidentiality - continued

• For example, the ISI’s Ethics Declaration states,

“the information provided shall not be used to harm data providers.”

29

4. Statistical confidentiality - continued

• And the United Nation’s Fundamental Principles of Official Statistics, in principle 6, defines the concept in these words,

“Individual data collected by statistical agencies for statistical compilation, whether they refer to natural or legal persons, are to be strictly confidential and used exclusively for statistical purposes.”

30

4. Statistical confidentiality - continued

• This same ethical norm has been reflected, at least to some degree, in the confidentiality provisions of census and statistical laws adopted in most countries.

31

4. Statistical confidentiality - continued

• These laws are of two broad types:

(1) those that prohibit disclosures for all types of non-statistical uses, and

(2) those that permit disclosures for certain important non-statistical state purposes.

32

4. Statistical confidentiality - continued

• Note that laws of the first type may permit some disclosures considered as harmless (for example, sharing data among statistical agencies to foster enhanced analytical possibilities).

• Such strong statistical confidentiality laws adhere closely to the ethical principle that the personal information provided will not be used to harm or target individuals.

33

4. Statistical confidentiality - continued

• Unfortunately other countries, whether by tradition or in response to perceived national threats, have weaker statistical confidentiality laws that allow for a distinction between authorized and unauthorized disclosures without reference to the issue of individual harm, and only bar the latter.

34

4. Statistical confidentiality - continued

• To understand the implications of this distinction it is useful to consider how disclosures of personal information provided to census and statistical agencies arise.

35

4. Statistical confidentiality - continued

• Such disclosures arise from three main sources:

1) inadvertent disclosures (for example, lost laptops or flash drives, enumeration records erroneously put in the general trash, misdirected mail),

2) ad hoc disclosures (for example, those attributable to individual hackers, disgruntled or over-zealous census staff, or gossipy interviewers), and

36

4. Statistical confidentiality - continued

3) targeted disclosures arising from the efforts of government or private sector entities external to the government statistical system to obtain personal information for some non-statistical purpose.

• The first two sources primarily use what might be termed “back-door” methods of obtaining this information, that is, by means of the unauthorized disclosures prohibited by the confidentiality provisions of the census and statistical acts.

37

4. Statistical confidentiality - continued

• The third type of disclosure, those arising from the efforts of other governmental agencies to gain access to personal information obtained in the census for non-statistical purposes, generally derive from the use of so-called “front-door” methods, that is, by means of the authorized disclosures permitted under these laws.

38

4. Statistical confidentiality - continued

• This distinction between “front-” and “back-door” disclosures is important, both from policy and ethical perspectives. Many discussions of statistical confidentiality either ignore this distinction or focus only on methods used to deter “back-door” disclosures.

39

4. Statistical confidentiality - concluded

• However, it is the “front-door” disclosures that pose the most serious ethical threat to an NSO, having been associated in a number of countries with substantial harm to vulnerable individuals or population subgroups.

• Such misuse of information obtained by an NSO has negatively impacted census and survey response rates for decades.

• However, even inadvertent disclosures can have a very negative impact on the reputation of an NSO.

40

5. Administration of a statistical agency

• Managing the staff and financial resources allocated to a statistical agency imposes a number of ethical obligations on those with managerial responsibilities, from the agency head on down.

• These ethical obligations are not normally covered in professional ethics statements of statistical societies.

41

5. Administration of a statistical agency -- concluded

These ethical responsibilities include:• Treating staff fairly and impartially• Managing the financial affairs of the agency

honestly and in a transparent manner, both with respect to resources provided from the government budget and resources earned by the agency for services or outputs provided

• Avoiding real or apparent conflicts of interest with respect to both personnel and financial management

• Any questions?

42

Group Problem-Solving Exercise

• You are each members of a committee of senior staff of your NSO. The committee has been convened by the Director of the NSO to develop a response plan to an emerging threat to the integrity of your office.

• A senior deputy to a politically-well connected and powerful government minister has telephoned your Director with a request.

43

Group Problem-Solving Exercise - continued

• The request:

Given the expected tightness of the coming elections, the Minister has asked his deputy to request that the release of the latest consumer price statistics, due to be released by your office in two weeks, be postponed for two months, that is, until after the elections. The deputy also indicated that both his boss and the Prime Minister would be very unhappy if this action is not taken.

44

Group Problem-Solving Exercise - continued

• An alternative request from the Minister’s deputy, if there is a second small group:

Given the hostility and threats of possible attack from your neighbor, the country of Freedonia, the NSO is requested to secretly provide to the Minister of Interior a list of the names of all persons born in Freedonia who are living in your capital district and aged from 10 to 40, together with their addresses and other particulars as obtained in the recent population census.

45

Group Problem-Solving Exercise - concluded

• The goal of your committee is to come up with a set of recommendations to your Director to respond to the request.

• In developing these recommendations draw on your own experience and readings as well as on anything that seems relevant from materials we have discussed today.

• You have 15 minutes and one of you from the group should be asked to summarize your recommendations.

46

6. Prevention and coping strategies and tools

• (a) Prevention strategies and safeguards

It is helpful to think of prevention strategies under several broad headings. They include:

(i) strong laws

(ii) close links among government statistical agencies

(iii) close links to major user groups and the news media

47

6. Prevention and coping strategies and tools – (a) Prevention - continued

(iv) Active professional statistical society with membership drawn from academia, business and industry, the non-profit sector, and government

(v) Maintain a spirit of openness and transparency with members of all political parties or groups, including both the government and the opposition

(vi) Encourage the study and documentation of past threats to the integrity of your NSO and facilitate the dissemination of the results of such studies

48

6. Prevention and coping strategies and tools – (a) Prevention - concluded

• Training sessions within the NSO to help educate staff and leadership about the ethical norms of official statistics

• With respect to statistical confidentiality there are a broad range of strategies available. They have been documented in several of the papers posted on the website: https://pantherfile.uwm.edu/margo/www/govstat/integrity.htm

• Add further ideas from small group discussion

• Any questions?

49

6. Prevention and coping strategies and tools – (b) Ways of coping

• When an actual threat to the integrity of an NSO emerges the key defense will be in the contacts previously established among the various groups of data users and other stake holders in a sound, well functioning, and non-political NSO

• A diverse group of such stake-holders may have political contacts beyond those directly available to the NSO

50

6. Prevention and coping strategies and tools – (b) Coping - continued

• In dealing with the current crisis, document what happened so that in future years the NSO and other statisticians can benefit from both what you did right and what you did wrong

• In thinking about strategies and responses be mindful not only of the present situations and the challenges presented, but also of the longer-term implications for your NSO and its reputation

51

6. Prevention and coping strategies and tools – (b) Coping - continued

• More serious coping tools:

(1) Approaches to the press and other media (open and discrete)

(2) Resignation (threats or actual) – by the agency head, senior staff, or mass resignations

(3) Deliberate refusal to carry out ethically irresponsible actions

(4) Pro-active refusals

52

6. Prevention and coping strategies and tools – (b) Coping - concluded

• I would stress that the goal of a robust set of prevention strategies is to minimize the need to ever have to consider using these more serious coping tools. I should say that they all have been used over the years, including a census director in the 1890s having census schedules destroyed rather than turning them over to the police.

• Any questions?

53

Conclusion

• As I hope has been clear from this presentation, those of us who work in official statistics have a number of important ethical obligations to balance along with our legal and technical responsibilities.

• There is often no one right answer. That is, we are faced with choices.

• My goal today has been to help you make these choices as knowledgably and wisely as possible.

54

Thank you

• A copy of my PowerPoint presentation will shortly be posted on the website:

https://pantherfile.uwm.edu/margo/www/govstat/integrity.htm

This website also contains a number of other papers dealing with statistical ethics, particularly statistical confidentiality

Also, please feel free to contact me at seltzer@fordham.edu

55

Bibliography for Professional Ethics and the Work of a National Statistical Office

• AAAS/USORI [American Association for the Advancement of Science/U.S. Office of Research Integrity]. 2000. “The Role and Activities of Scientific Societies in Promoting Research Integrity. A Report of Conference, Washington, DC, April 20, 2000.” September 2000. Available at <www.aaas.org/spp/dspp/sfrl/projects/integrity.htm>.

• Anderson, Margo and William Seltzer. 2006. “Discussion of Habermann, ‘Ethics, Confidentiality and Data Dissemination.’” Journal of Official Statistics. 22(4): 641-49.*

• _______ . 2007. "Challenges to the Confidentiality of U.S. Federal Statistics, 1910-1965."  Journal of Official Statistics.  23(1): 1-34.*

• _______ . 2009. "Federal Statistical Confidentiality and Business Data: Twentieth Century Challenges and Continuing Issues,“ Journal of Privacy and Confidentiality 1 (Spring), pp. 7-52; Comment on Article by Anderson and Seltzer, by C. L. Kincannon, pp. 53-54; Rejoinder, by M. Anderson and W. Seltzer, pp. 55-58.*

56

• Annas, George J., and Michael Grodin, Eds. 1992. The Nazi Doctors and the Nuremberg Code: Human Rights in Human Experimentation. New York: Oxford University Press.

• Association des Administrateurs de l’INSEE. 1985. Code de déontologie statistique. Paris.

• ASA [American Statistical Association]. 2001a. “Committee on Professional Ethics. Charge.” http://www.amstat.org/comm/index.cfm?fuseaction=commdetails&txtComm=CS04

• _______ . 2001b. “By-Laws of The American Statistical Association, Article I, Membership, 4. Termination.” http://www.amstat.org/about/bylaws.htm

• _______ . 1999 “Ethical Guidelines for Statistical Practice.” Also available at <http://www.amstat.org/profession/ethicalstatistics.html>.

• Bailar, Barbara A. 2009. “Who Controls the Release of Statistical Data?” Paper prepared for presentation at the Joint Statistical Meetings, Washington, DC, August 2009.

57

• Barabba, Vincent. 1975. “The Right of Privacy and the Need to Know.” In U.S. Census Bureau, The Census Bureau: A Numerator and Denominator for Measuring Change, Technical Paper 37. Washington, D.C.: US Government Printing Office.

• Berumen, Edmundo, and Victor Beker: 2009. “Recent developments in price and related statistics in Argentina.” Paper prepared for presentation at the 57th session of the International Statistical Institute, Durban, South Africa, August 2009.

• Bodin, Jean-Louis, 2009. “How to react when the independence of statisticians and the integrity of statistics are endangered?” Paper prepared for presentation at the 57th session of the International Statistical Institute, Durban, South Africa, August 2009.

• Caplan, Arthur L. 1992. “The Doctor’s Trial and Analogies to the Holocaust in Contemporary Bioethical Debates.” In Annas, George J., and Michael Grodin, Eds. The Nazi Doctors and the Nuremberg Code: Human Rights in Human Experimentation. New York: Oxford University Press, pp. 258-275.

• DuMez, Elizabeth. 2000. “The Role and Activities of Scientific Societies in Promoting Research Integrity.” Professional Ethics Report Vol 13, No. 3 (Summer): 1-5.

58

• Duncan, George T., Thomas B. Jabine, and Virginia A. de Wolf (Eds.). 1993. Private Lives and Public Policies: Confidentiality and Access of Government Statistics. Panel on Confidentiality and Data Access, Committee on National Statistics, National Research Council and the Social Science Research Council. Washington, DC: National Academy Press.

• FCSM [Federal Committee on Statistical Methodology]. 1999. “Checklist on Disclosure Potential of Proposed Data Releases.” ICDAG1, Interagency Confidentiality and Data Access Group, July 1999. Also available at www.fcsm.gov/docs/_799.doc

• Forbes, Sharleen, Vince Galvin, Andrew Hunter, Paul Maxwell, and Whetu Wereta. 2009. “Recent challenges to the ethics underlying official statistics in New Zealand.” Paper prepared for presentation at the 57th session of the International Statistical Institute, Durban, South Africa, August 2009.

• Gardenier, John. 1996. “What and Where Are Statistical Ethics?” Proceedings of the Section on Statistical Education, 1996. Alexandria, VA: American Statistical Association, pp. 256-260.

• _______. 2001. “Reconciling the Ethical Demands of Official Statistics

and Non-governmental Statistics.” Paper presented at the Joint Statistical Meetings, Atlanta, GA, August 2001.

59

• Glantz, Leonard, H. 1992. “The Influence of the Nuremberg Code on U.S. Statues and Regulations.” In Annas, George J., and Michael Grodin, Eds. The Nazi Doctors and the Nuremberg Code: Human Rights in Human Experimentation. New York: Oxford University Press, pp. 183-200.

• Gorlin, Rena (Ed.). 1999. Codes of Professional Responsibility: Ethics Standards in Business, Health, and Law. Washington, DC: The Bureau of National Affairs. Fourth edition.

• Habermann, Hermann. 2006. “Ethics, Confidentiality, and Data Dissemination.” Journal of Official Statistics, Vol.22, No.4, pp. 599-614.

• International Statistical Institute. 1986. “Declaration of professional ethics for statisticians.” International Statistical Review 227-247. Also available at <http://www.cbs.nl/isi/ethics.htm>.

• Jowell, Roger. 1981. “A Professional Code for Statisticians? Some Ethical and Technical Conflicts.” Bulletin of the International Statistical Institute - Proceedings of the 43rd session, I: 165-205.

• Kultgen, John. 1988. Ethics and Professionalism. Philadelphia: University of Pennsylvania Press.

60

• Lifton, Robert Jay. 1986 The Nazi Doctors: Medical Killing and the Psychology of Genocide. New York: Basic Books.

• Martín-Guzmán, Pilar, and Villán Ildefonso. 2009. “Regulations and practices for the protection of confidentiality of statistical data: some experiences from Spain.” Paper prepared for presentation at the 57th session of the International Statistical Institute, Durban, South Africa, August 2009.

• Murphy, Thomas F. and Marc A. Lapp, eds. 1994. Justice and the Human Genome Project. Berkely (CA): University of California Press.

• National Research Council. 2009. Principles and Practices for a Federal Statistical Agency. Fourth Edition. Constance F. Citro, Margaret E. Martin, and Miron L. Straf, eds. Committee on National Statistics. Washington, DC: National Academy Press.

• Powers, Mary. 2002. “Concepts, Definitions, and Classifications: Some Ethical Issues in Demographic Research and Policy.” Paper presented at the Annual Meeting of the Population Association of America, May 2002.

61

• Prewitt, Kenneth. 2000. “Additional Comments by Census Bureau Director Kenneth Prewitt on the Role of the Census Bureau and the Use of Census Tabulations in the Internment of Japanese Americans in World War II.” Press Release, U.S. Census Bureau, March 24, 2000.

• Rémond, René, et al. 1996. Le ‘Fichier Juif’ : Rapport de la Commision présidée par René Rémond au Premier Ministre. Paris: Plon éditeur.

• Reynolds, Paul Davidson. 1979. Ethical Dilemmas in Social Research: An Analysis of Moral Issues Confronting Investigators in Research Using Human Subjects. San Francisco: Jossey-Bass.

• Royal Statistical Society. 1993. “Royal Statistical Society Code of Conduct.” Attachment to RSS News 20(June): 7.

• Seltzer, William. 1994. “Politics and Statistics: Independence, Dependence or Interaction? Working Paper No. 6, United Nations Department of Economic and Social Information and Policy Analysis: New York.

• _____ . 1998. “Population statistics, the Holocaust, and the Nuremberg Trials.”

Population and Development Review 24(3): 511-552.*

62

• _____ . 1999. “Excluding Indians Not Taxed: Federal Censuses and Native-American in the 19th Century.” Paper presented at the 1999 Joint Statistical Meetings, Baltimore, MD. A shorter version appears in 1999 Proceedings of the Government and Social Statistics Section. Alexandria, VA: American Statistical Association, 161-166.*

• _____ . 2001. "U.S. Federal Statistics and Statistical Ethics: The Role of the American Statistical Association’s Ethical Guidelines for Statistical Practice.” Paper based on a presentation at the Washington Statistical Society, February 2001.*

• _____ . 2005. "Official Statistics and Statistical Ethics: Selected Issues." Paper prepared for the International Statistical Institute, 55th Session, 2005.*

• _____ . 2009. “Statistical Confidentiality and UK Population Censuses: A Summary of Recent Findings and Ethical Issues.” Paper prepared for presentation at the 57th session of the International Statistical Institute, Durban, South Africa, August 2009.

• Seltzer, William and Margo Anderson. 2000. “After Pearl Harbor: The Proper Role of Population Statistics in Time of War.” Paper prepared for presentation at the Annual Meeting of the Population Association of America, Los Angeles, CA, March 2000.*

63

• _____ . 2001. “The Dark Side of Numbers: The Role of Population Data Systems in Human Rights Abuses.” Social Research 68: 2 (Summer): 339-371.

• _____ . 2002. "NCES and the Patriot Act.“ Paper presented at the Joint Statistical Meetings, New York, August 2002.* Abridged version also available in 2002 ASA Proceedings, pp. 3153-56.

• _____ . 2007. "Census Confidentiality under the Second War Powers Act (1942-1947)," Paper prepared for the Annual Meeting of the Population Association of America, New York, NY, March 30, 2007.*

• _____ . 2008. "Using Population Data Systems to Target Vulnerable Population Subgroups and Individuals: Issues and Incidents," in Statistical Methods for Human Rights, Jana Asher, David Banks, and Fritz S. Scheuren, eds. (New York: Springer, 2008), pp. 273-328.

• Slote, Michael. 1995. “Ethics: I. Task of Ethics.” Encyclopedia of Bioethics. Rev. ed. New York: Macmillan-Simon.

• Søbye, Espen. 1998. “Et moerkt kapittel i statistikkens historie?” [A dark chapter in the history of statistics?]. Samfunsspeilet 8, no.4 (10 September). Oslo: Statistics Norway.

.

64

• United Nations Economic and Social Council. 1994. Report of the Special Session of the Statistical Commission, New York, April 11-15. E/1994/29. Also available at <http://www.un.org/depts/unsd/statcom/1994docs/e1994.htm> and reproduced in National Research Council, 2009.

• USGAO [U. S. General Accounting Office]. 2001. Record Linkage and Privacy: Issues in Creating New Federal Research and Statistical Information. GAO-01-126SP. Washington, DC: US GPO. Also available at <http://www.gao.gov/cgi-bin/getrpt?gao-01-126sp>.

• Vanderpool, Harold Y. (Ed.) 1996. The Ethics of Research Involving Human Subjects: Facing the 21st Century. Frederick (MD): University Publishing Group.

• * Available at https://pantherfile.uwm.edu/margo/www/govstat/integrity.htm