Welcome to the 7Welcome to the 7thth IPUMS-International workshop IPUMS-International workshop

Accomplishments, plans and challenges Accomplishments, plans and challenges https://international.ipums.org/international

* * ** * *Robert McCaa, Professor of population historyRobert McCaa, Professor of population history

University of MinnesotaUniversity of Minnesotarmccaa@umn.edu

for additional details, please seefor additional details, please see::www.hist.umn.edu/~rmccaa/ipumslawww.hist.umn.edu/~rmccaa/ipumsla

1. Introductions, organization and program1. Introductions, organization and program» IntroductionsIntroductions

» Doña Carmen Miró, director and founder of CELADEDoña Carmen Miró, director and founder of CELADEthanks to your vision: the world’s largest microdata archive thanks to your vision: the world’s largest microdata archive

» Don Evelio Fabbroni, technical secretary of IASIDon Evelio Fabbroni, technical secretary of IASI» Don Dimas Quiel, Director General of DEC-Panamá, and your staff Don Dimas Quiel, Director General of DEC-Panamá, and your staff » Delegates of the National Statistical Institutes of Latin AmericaDelegates of the National Statistical Institutes of Latin America» Minnesota Population Center: 9 members of the teamMinnesota Population Center: 9 members of the team

» OrganizaciónOrganización» Dinner per-diem for Tuesday Dinner per-diem for Tuesday » All activities are in the Hotel Crowne PlazaAll activities are in the Hotel Crowne Plaza

» Program (please see the workshop folder): two intensive daysProgram (please see the workshop folder): two intensive days» Invited speakers: INEs, CELADE, CED (Barcelona)Invited speakers: INEs, CELADE, CED (Barcelona)» MPC: MPC:

» Listen and learnListen and learn» Demonstrate what has been accomplisted: recovery, confidentiality, integration, Demonstrate what has been accomplisted: recovery, confidentiality, integration,

disseminationdissemination» Show the website and how to make good use of itShow the website and how to make good use of it» Discuss plans and innovations for the 2Discuss plans and innovations for the 2ndnd five-year plan of IPUMS-AL: 2009-2013 five-year plan of IPUMS-AL: 2009-2013

Outline of the presentationOutline of the presentation

no. of slidesno. of slides

1. Introduction1. Introduction 552. Accomplishments—celebrate a success far beyond…2. Accomplishments—celebrate a success far beyond…

a.a. Censuses and documentation recovered Censuses and documentation recovered 99b.b. Censuses integrated (see map and inventory)Censuses integrated (see map and inventory) 33c.c. Confidenciality and securityConfidenciality and security 55d.d. Methods and proceduresMethods and procedures 33

3. Plan and challenges3. Plan and challenges 22» Censuses to integrate, 2010 round, tabulator Censuses to integrate, 2010 round, tabulator

(REDATAM?), GIS, laboratories of high security(REDATAM?), GIS, laboratories of high security

» Past: Thank you! Past: Thank you! » All the statistical institutes of Latin American are All the statistical institutes of Latin American are

cooperating with the IPUMS project. cooperating with the IPUMS project. » Present: Samples of 9 countries (43 censuses) of the Present: Samples of 9 countries (43 censuses) of the

American continents integrated in the IPUMS system American continents integrated in the IPUMS system https://international.ipums.org/internationalhttps://international.ipums.org/international » 2002-8: Argentina, Brazil, Chile, Colombia, Costa Rica, 2002-8: Argentina, Brazil, Chile, Colombia, Costa Rica,

Ecuador, Mexico, Panamá, y VenezuelaEcuador, Mexico, Panamá, y Venezuela» Future: 2009-13: Future: 2009-13:

» The remaining Latin American countriesThe remaining Latin American countries» Censuses of the 2010 round, entrusted before 2013Censuses of the 2010 round, entrusted before 2013» Tabulator, GIS, Laboratories of high securityTabulator, GIS, Laboratories of high security

IPUMS – Latin America: past, present and futureIPUMS – Latin America: past, present and future

Some members of the IPUMS team (2008) Some members of the IPUMS team (2008)

(Not present: computer gurus, some researchers, and others who were too busy (Not present: computer gurus, some researchers, and others who were too busy to make time for taking a photo!)to make time for taking a photo!)

Steven Ruggles, inventor of IPUMS, Professor of History, and Director of the Minnesota Population Center

The objectives of IPUMSThe objectives of IPUMS

1.1. Preserve census microdata and documentation for all the Preserve census microdata and documentation for all the countries in the worldcountries in the world

2.2. Integrate microdata and metadata Integrate microdata and metadata 3.3. Disseminate--without cost--extracts of samples with the Disseminate--without cost--extracts of samples with the

corresponding documentation to researcherscorresponding documentation to researchers

IPUMS MilestonesIPUMS Milestones

» 1995: IPUMS-USA first release of integrated microdata1995: IPUMS-USA first release of integrated microdata IPUMS-USA continues: 1850-2000 + ACS IPUMS-USA continues: 1850-2000 + ACS

samplessamples» 1999: IPUMS-International funded1999: IPUMS-International funded» 2002 - 12002 - 1stst International release: 7 countries, including International release: 7 countries, including

Colombia and MexicoColombia and Mexico» 2006 release: 20 countries, 63 censuses, 2006 release: 20 countries, 63 censuses, » 2008 release: 35 countries, 111 censuses2008 release: 35 countries, 111 censuses

» ~263 million person records~263 million person records» Two thousand usersTwo thousand users

» 2013 release: ~60 countries, ~200 censuses 2013 release: ~60 countries, ~200 censuses Note: microdata are already entrusted to MPC Note: microdata are already entrusted to MPC

Workshop goals Workshop goals • Evalute the integrations to date (9 countries, 43 Evalute the integrations to date (9 countries, 43

censuses)censuses)• Discuss plans to complete the remaining integrations Discuss plans to complete the remaining integrations

(12 countries, 43 censuses) (12 countries, 43 censuses) • Consider the incoporation of electronic boundary files Consider the incoporation of electronic boundary files

corresponding to the microdata corresponding to the microdata • Study the obstacles and challenges in the Study the obstacles and challenges in the

harmonization of census samples for the 2010 census harmonization of census samples for the 2010 census round.round.

• Discuss methods and procedures of the IPUMS project Discuss methods and procedures of the IPUMS project to improve themto improve them

• Respond to doubts, questions, or concerns regarding Respond to doubts, questions, or concerns regarding any aspect of the projectany aspect of the project

6 presentations on IPUMS 6 presentations on IPUMS

1.1. Introduction: past, present, and future – Bob McCaaIntroduction: past, present, and future – Bob McCaa2.2. Metadata: The IPUMS dynamic system – Toni LópezMetadata: The IPUMS dynamic system – Toni López3.3. How to make an extract (to obtain microdata) – Miguel How to make an extract (to obtain microdata) – Miguel

RicaurteRicaurte4.4. How Integration is accomplished – Matt SobekHow Integration is accomplished – Matt Sobek5.5. Complementing microdata with GIS: the example of NHGIS Complementing microdata with GIS: the example of NHGIS

– Petra Noble– Petra Noble6.6. The REDATAM tabulator of integrated microdata The REDATAM tabulator of integrated microdata

implemented by IECM (CED, Barcelona) – Toni Lópezimplemented by IECM (CED, Barcelona) – Toni López

» 1959: CELADE began the grand project OMUECE 1959: CELADE began the grand project OMUECE (Operation of Census Samples). (Operation of Census Samples).

» Only CELADE, of all the UN demographic centers, Only CELADE, of all the UN demographic centers, » Began a project to archive microdataBegan a project to archive microdata» Stimulated an archival program for both data and Stimulated an archival program for both data and

documentationdocumentation» Already, in 1977, CELADE was entrusted with 61 sets Already, in 1977, CELADE was entrusted with 61 sets

of microdata encompassing 20 countries. of microdata encompassing 20 countries. » Principal goal: special and comparative tabulationsPrincipal goal: special and comparative tabulations» Comparative demographic research of many countries Comparative demographic research of many countries » Standardization of basic codes to attain a minimal level of Standardization of basic codes to attain a minimal level of

comparability.comparability.

2a. The Americas: 2a. The Americas: The global vanguard in preserving microdataThe global vanguard in preserving microdata

Census Microdata: 1950sCensus Microdata: 1950sfew countries archived microdatafew countries archived microdata

(a country in green indicates microdata exist for the decade) (a country in green indicates microdata exist for the decade)see: www.hist.umn.edu/~rmccaa/IUMSI/country6.htmsee: www.hist.umn.edu/~rmccaa/IUMSI/country6.htm

Mollweide projection

Census Microdata: 1960sCensus Microdata: 1960sThe Americas: The Americas:

in the vanguard for encouraging the preservation of microdatain the vanguard for encouraging the preservation of microdata

Mollweide projection

Census Microdata: 1970sCensus Microdata: 1970salready in the Americas, the preservation of microdata is almost already in the Americas, the preservation of microdata is almost

universal and is becoming widespread in Europe, Africa and Asiauniversal and is becoming widespread in Europe, Africa and Asia

Mollweide projection

Census Microdata: 1980sCensus Microdata: 1980sThe preservation of microdata became generalizedThe preservation of microdata became generalized

Mollweide projection

Perú: Perú: Can Can the tapes for the tapes for the census the census of 1981 be of 1981 be recoveredrecovered??

Census Microdata: 1990sCensus Microdata: 1990smany countries preserved microdatamany countries preserved microdata

(or are disposed to recover them) (or are disposed to recover them)

Mollweide projection

Rep. Dom.: Rep. Dom.: ¿can the ¿can the data for the data for the census of census of 1993 be 1993 be recovered?recovered?

Inventory of census microdata archived by region Inventory of census microdata archived by region and decade (% of censuses conducted)and decade (% of censuses conducted)

•Note: cases confirmed by the corresponding official statistical institute. Some Note: cases confirmed by the corresponding official statistical institute. Some datasets remain to be certified. Some countries have not responded to the invitation to datasets remain to be certified. Some countries have not responded to the invitation to inventory their stocks of data. inventory their stocks of data. Source: http://www.hist.umn.edu/~rmccaa/IPUMS/country6.htmSource: http://www.hist.umn.edu/~rmccaa/IPUMS/country6.htm

Region/continent Countries 2000s 1990s 1980s 1970s   1960s

Latin America 21 100% 100% 89% 81% 72%

North America 27 91% 72% 64% 24% 8%

Africa 58 15% 22%  25%  15%  2% 

Asia 44 ?% 54% 31% 30% 13%

Europe 46 ?% 67% 55% 41% 13%Pacific(pob>.5m) 7 100% 100% 100% 43% 29%

The CELADE archivesThe CELADE archives~3000 microdata tapes preserved with the ~3000 microdata tapes preserved with the

corresponding documentation corresponding documentation

Los Archivos de CELADELos Archivos de CELADE~3000 ~3000 cintas de microdatos preservadas con su cintas de microdatos preservadas con su

documentación documentación correspondientecorrespondiente

For the entire region, manuals are lacking for only 10 censuses:For the entire region, manuals are lacking for only 10 censuses:

1. El Salvador: 1. El Salvador: 19921992 20072007

2.2. Guatemala: Guatemala: 19731973 19941994 20022002

3.3. Honduras: Honduras: 19741974

4.4. Nicaragua: Nicaragua: 19951995 20052005

5.5. Rep. Dom.: Rep. Dom.: 20022002

6.6. Perú: Perú: 20072007

2b. Integration: IPUMS-Latin America in global context 2b. Integration: IPUMS-Latin America in global context dark greendark green = already integrated = already integrated

(35 countries, 111 censuses, 263 millon person records)(35 countries, 111 censuses, 263 millon person records)green = to be integrated (39 countries, 103 censuses, 150 mill.)green = to be integrated (39 countries, 103 censuses, 150 mill.)

Mollweide projection

Photos from the first integration project: Photos from the first integration project: Colombian microdata, Colombian microdata,

February-MarchFebruary-March, 2000, 2000::4 experts from DANE 4 experts from DANE

+7 ac+7 acadeademics (3 univermics (3 universitiessities))

Standard: UNSD Standard: UNSD Principals and Principals and recommendationsrecommendations......

Census Census documentation documentation assembled for the assembled for the microdata of microdata of ColombiaColombia

IPUMS-Latin AmericaIPUMS-Latin America» Samples currently available in IPUMSSamples currently available in IPUMS» 9 Latin American countries, 43 censuses: 9 Latin American countries, 43 censuses:

average = 4.8 censuses per country average = 4.8 censuses per country » 26 countries for other regions, 68 censuses: 26 countries for other regions, 68 censuses:

average = 2.6 censuses per countryaverage = 2.6 censuses per country» In 2013, is all goes well:In 2013, is all goes well:» 21 Latin American countries, 86 censuses21 Latin American countries, 86 censuses» 60 countries for other regions, 120 censuses.60 countries for other regions, 120 censuses.

2c. Statistical Confidentiality 2c. Statistical Confidentiality and securityand security

»Cited by UN-ECE as “good practice”Cited by UN-ECE as “good practice”»On-site inspection: the Dennis Trewin ReportOn-site inspection: the Dennis Trewin Report

Why was IPUMS cited as Why was IPUMS cited as “good practice” by “good practice” by the UN-ECE the UN-ECE

(2007, Annex 23, pp. 98-103)?(2007, Annex 23, pp. 98-103)?http://www.unece.org/stats/documents/tfcm.htm

Good practices (see annex 23):Good practices (see annex 23):

» High level of confidence and transparency between the High level of confidence and transparency between the researchers (users) and the national statistical institutesresearchers (users) and the national statistical institutes

» The conditiions of use are well definedThe conditiions of use are well defined» Sanctions for mis-use are clearly spelled outSanctions for mis-use are clearly spelled out» Good use is assured by both juridical and administration Good use is assured by both juridical and administration

mechanisms to prevent violationsmechanisms to prevent violations» Sanctions are imposed no only against those who misuse the Sanctions are imposed no only against those who misuse the

data but also against their institutions.data but also against their institutions.» The data are anonymized by highly efficient technical meansThe data are anonymized by highly efficient technical means

The standard agreement between National Statistical The standard agreement between National Statistical Institutes and the University of MinnesotaInstitutes and the University of Minnesota

Statistical confidentiality and security:Statistical confidentiality and security:see the Trewin Reportsee the Trewin Report

» ““The best practice for an international The best practice for an international repository of microdata”repository of microdata”

» ““The security of IPUMS is first class…the The security of IPUMS is first class…the standard of the best national statistical offices”standard of the best national statistical offices”

» ““in full compliance with the principles and in full compliance with the principles and recommendations of the ECE”recommendations of the ECE”

2d. IPUMS methods and procedures2d. IPUMS methods and procedures» Dissemination by internetDissemination by internet» Comprehensive documentation, including Comprehensive documentation, including

» Data dictionaries and codebooksData dictionaries and codebooks» Complete original source documentation in the official Complete original source documentation in the official

language:language: questionnaires, manuals, etc. questionnaires, manuals, etc.

» All translated to English and converted into metadatabase All translated to English and converted into metadatabase for each censusfor each census

» Integration Integration ≠ standardization≠ standardization» Composite codes (11, 12, 21, 22…) ≠ serial codes (1, 2, 3, …) Composite codes (11, 12, 21, 22…) ≠ serial codes (1, 2, 3, …)

(see next slide)(see next slide)

Chile Chile MéxicoMéxico

CodeCode LabelLabel 19921992 20022002 19901990 2000200000 NIUNIU X X X X X X X X

ACTIVE (In Labor Force)ACTIVE (In Labor Force)100100 EMPLOYED, not specifiedEMPLOYED, not specified · · · · · · · · 110110 At workAt work X X X X X X X X 111111 At work, and 'student'At work, and 'student' · · · · · · X X 112112 At work, and 'housework'At work, and 'housework' · · · · · · X X 113113 At work, and 'seeking work'At work, and 'seeking work' · · · · · · X X 114114 At work, and 'retired'At work, and 'retired' · · · · · · X X 115115 At work, and 'no work'At work, and 'no work' · · · · · · X X 116116 At work, and 'other'At work, and 'other' · · · · · · X X 117117 At work, family holding, not specifiedAt work, family holding, not specified · · · · · · · · 118118 At work, family holding, not agriculturalAt work, family holding, not agricultural · · · · · · · · 119119 At work, family holding, agriculturalAt work, family holding, agricultural · · · · · · · · 120120 Have job, not at work last weekHave job, not at work last week X X X X X X X X

IPUMS—Integration method: IPUMS—Integration method: composite codes (multiple digits)composite codes (multiple digits)

retains not only significant distinctions retains not only significant distinctions but also integrates comparable conceptsbut also integrates comparable concepts

In addition…In addition…

»Microdata: new high precision samples not Microdata: new high precision samples not only for contemporary censuses but also for only for contemporary censuses but also for historical ones (before the 90s)historical ones (before the 90s)

» Systematic metadata for all variablesSystematic metadata for all variables»UniversesUniverses»DefinitionsDefinitions»Comparability Comparability »Dynamic System—facilitates comparing the Dynamic System—facilitates comparing the

wording of questionnaires and instructions for any wording of questionnaires and instructions for any combination of countries and censusescombination of countries and censuses

3. IPUMS-Latin America II, 2009-13: objectives3. IPUMS-Latin America II, 2009-13: objectivesObjectivesObjectives

1.1. Conclude the integration of the censuses of the remaining Conclude the integration of the censuses of the remaining countries in the region countries in the region

2.2. Incorporate samples for the 2010 roundIncorporate samples for the 2010 round3.3. Add digital boundary files at the second administrative Add digital boundary files at the second administrative

levellevel4.4. Facilitate pre-analysis with an on-line tabulatorFacilitate pre-analysis with an on-line tabulator5.5. Construct a laboratory of high security Construct a laboratory of high security

Invitation:Invitation: 1.1. Confirm participation in IPUMS-AL IIConfirm participation in IPUMS-AL II2.2. Facilitate copies of digital boundary filesFacilitate copies of digital boundary files3.3. In time, make available census microdata and In time, make available census microdata and

documentation for the 2010 rounddocumentation for the 2010 round4.4. Discuss participation in the high security laboratoryDiscuss participation in the high security laboratory

» Appreciation:Appreciation:» To the founders (and current members) of CELADE for having had the To the founders (and current members) of CELADE for having had the

vision and ability to assemble and preserve microdatavision and ability to assemble and preserve microdata» To the official statistical institutes for, first, cooperating with CELADE, To the official statistical institutes for, first, cooperating with CELADE,

and second, for participating in the IPUMS projectand second, for participating in the IPUMS project

» ReflectionsReflections» Latin America: a model for statistical cooperationLatin America: a model for statistical cooperation» IPUMS: already the world’s largest microdatabaseIPUMS: already the world’s largest microdatabase

» Invitation:Invitation:» Participate in IPUMS-AL IIParticipate in IPUMS-AL II» Entrust microdata and documentation Entrust microdata and documentation » Consider: Tabulator, GIS, laboratory of high securityConsider: Tabulator, GIS, laboratory of high security

Appreciation, reflections and invitation Appreciation, reflections and invitation

Thank you!!Thank you!!

rmccaa@umn.edurmccaa@umn.edu