• Home

  • Documents

  • Data Commons · Data Commons vs collection of datasets Collections of datasets (ala NIH Data...
29
Data Commons

Data Commons · Data Commons vs collection of datasets Collections of datasets (ala NIH Data Commons). Solves the problem of finding the dataset. But the remaining problems --- cleaning,

  • Upload
    others

  • View
    4

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Data Commons · Data Commons vs collection of datasets Collections of datasets (ala NIH Data Commons). Solves the problem of finding the dataset. But the remaining problems --- cleaning,

Data Commons

Page 2: Data Commons · Data Commons vs collection of datasets Collections of datasets (ala NIH Data Commons). Solves the problem of finding the dataset. But the remaining problems --- cleaning,

Context

Data powers everything

- policy, - journalism,- health,- science

How do we make it easier?

Page 3: Data Commons · Data Commons vs collection of datasets Collections of datasets (ala NIH Data Commons). Solves the problem of finding the dataset. But the remaining problems --- cleaning,

Problem is not a shortage of data

Demographics: ACS, Housing Survey, Community Survey,

Economics: BLS, BEA, World Bank, OECD, …Health: CDC Wonder, CDC Diabetes, County Health, …Climate: NOAA, Hurricanes, Flooding, …Genomics: NCBI, ENCODE, ...

Too many formats, schemas, ...

Page 4: Data Commons · Data Commons vs collection of datasets Collections of datasets (ala NIH Data Commons). Solves the problem of finding the dataset. But the remaining problems --- cleaning,

Using Data: current model

Forage for data sources, track down assumptions/provenance

Clean it up, compile data sources, figure out storage, …

High upfront costs, sparse ecosystem, few tools, …

Page 5: Data Commons · Data Commons vs collection of datasets Collections of datasets (ala NIH Data Commons). Solves the problem of finding the dataset. But the remaining problems --- cleaning,

Google maps made satellite imagery part of everyone’s life

Page 6: Data Commons · Data Commons vs collection of datasets Collections of datasets (ala NIH Data Commons). Solves the problem of finding the dataset. But the remaining problems --- cleaning,

We want to do that for data

From search for dataset, download, clean, normalize,

join …

to

Just ask Google

Page 7: Data Commons · Data Commons vs collection of datasets Collections of datasets (ala NIH Data Commons). Solves the problem of finding the dataset. But the remaining problems --- cleaning,

Census(ACS)BLSFBI

NOAA

CDCSequence data

Wikidata Landsat

Aggregated Knowledge Graph

Grid

Medicare

EPA

APIS

JournalistsStudents ResearchersSearch, News Cloud

Page 8: Data Commons · Data Commons vs collection of datasets Collections of datasets (ala NIH Data Commons). Solves the problem of finding the dataset. But the remaining problems --- cleaning,

Data Commons vs collection of datasets

Collections of datasets (ala NIH Data Commons). Solves the problem of finding the dataset.

But the remaining problems --- cleaning, joining … remain

Datacommons --- a single KG built by cleaning, normalizing, joining all these datasets

Page 9: Data Commons · Data Commons vs collection of datasets Collections of datasets (ala NIH Data Commons). Solves the problem of finding the dataset. But the remaining problems --- cleaning,

Data Commons v 0.9

- People, places, …: Integrated view of Census (ACS), CDC, BLS, BEA, FEC, NOAA, DEA, DOL,… on average 5k variables for every state, county, city, zip, school district, congressional district, …

- Education: College Scoreboard, NCES, ...- Disasters: earthquakes, hurricanes, floods, fires

- Scientific Collections: Bronx Botanical Gardens, ENCODE, parts of NCBI, GTEx

Page 10: Data Commons · Data Commons vs collection of datasets Collections of datasets (ala NIH Data Commons). Solves the problem of finding the dataset. But the remaining problems --- cleaning,

Data Commons v 0.9

Applications for 4 categories

- Researchers- Students- Journalists- Google Users

Page 11: Data Commons · Data Commons vs collection of datasets Collections of datasets (ala NIH Data Commons). Solves the problem of finding the dataset. But the remaining problems --- cleaning,

APIs

APIs in

- REST- Python / Python Notebooks- SPARQL- SQL against Big Query- Google Sheets

Page 12: Data Commons · Data Commons vs collection of datasets Collections of datasets (ala NIH Data Commons). Solves the problem of finding the dataset. But the remaining problems --- cleaning,

Python Notebooksfor students &researchers

Page 13: Data Commons · Data Commons vs collection of datasets Collections of datasets (ala NIH Data Commons). Solves the problem of finding the dataset. But the remaining problems --- cleaning,

Simple Charting Tool

Page 14: Data Commons · Data Commons vs collection of datasets Collections of datasets (ala NIH Data Commons). Solves the problem of finding the dataset. But the remaining problems --- cleaning,

Simple CorrelationTool

Page 15: Data Commons · Data Commons vs collection of datasets Collections of datasets (ala NIH Data Commons). Solves the problem of finding the dataset. But the remaining problems --- cleaning,

Google Sheets API

Page 16: Data Commons · Data Commons vs collection of datasets Collections of datasets (ala NIH Data Commons). Solves the problem of finding the dataset. But the remaining problems --- cleaning,

Biomedical Data Commons

Page 17: Data Commons · Data Commons vs collection of datasets Collections of datasets (ala NIH Data Commons). Solves the problem of finding the dataset. But the remaining problems --- cleaning,

Census(ACS)UniProtCheMBL

dbSNP

CDCSequence data

Entrez Gene GTEx

Aggregated Knowledge Graph

MeSH

Medicare

WHO

APIS

CitizenScientists

Students ResearchersDoctors Cloud

Page 18: Data Commons · Data Commons vs collection of datasets Collections of datasets (ala NIH Data Commons). Solves the problem of finding the dataset. But the remaining problems --- cleaning,

Biomedical Data Commons Datasets

- CDC - 500 Cities- CDC - Diabetes

Atlas- CDC - Wonder- ChEMBL*- ClinVar- dbSNP- Dartmouth

Medicare Atlas

- Disease Ontology*- FDA -

Pharmacologic Class*

- GTEx- ENCODE- Entrez Gene- MeSH*- NY Times -

COVID-19 Cases + Deaths

- SIDER*- SPOKE*- UCSC Genome

Browser- US Census - SAHIE- UniProt*- WHO - COVID-19

Cases + Deaths- WHO - ICD-10

Codes*Source: UCSF SPOKE

Page 19: Data Commons · Data Commons vs collection of datasets Collections of datasets (ala NIH Data Commons). Solves the problem of finding the dataset. But the remaining problems --- cleaning,

Eg: Extract 3 data points on given variants

Given a list of genetic variants

Find:

1. Clinical Significance2. Functional Category3. Significant Gene Associations in Whole Blood

Page 20: Data Commons · Data Commons vs collection of datasets Collections of datasets (ala NIH Data Commons). Solves the problem of finding the dataset. But the remaining problems --- cleaning,

Current Methods to find information

Option 1:Site Search

Use needed database browser and search all variants individually

Option 2:Download and Analyze Data

Download data, read it into memory, and parse it for the needed info programmatically

Option 3:Data Commons

Run 1 query on Data Commons

Page 21: Data Commons · Data Commons vs collection of datasets Collections of datasets (ala NIH Data Commons). Solves the problem of finding the dataset. But the remaining problems --- cleaning,

More on Option 2

- I want to find out more about a list of genetic variants- Download ClinVar and analyze clinical significance of the

variants - a few hours- Download dbSNP and identify the type of genomic region

these SNPs are located - a couple of days- Download GTEx and identify which genes a genetic variant is

significantly associated in a given tissue - a week

Page 22: Data Commons · Data Commons vs collection of datasets Collections of datasets (ala NIH Data Commons). Solves the problem of finding the dataset. But the remaining problems --- cleaning,

Example Data Commons Query

Issue Query to Data Commons

Time to run query: 21 seconds

Page 23: Data Commons · Data Commons vs collection of datasets Collections of datasets (ala NIH Data Commons). Solves the problem of finding the dataset. But the remaining problems --- cleaning,

Many Other Commons ...

- Economics- Covid- Energy

….

Page 24: Data Commons · Data Commons vs collection of datasets Collections of datasets (ala NIH Data Commons). Solves the problem of finding the dataset. But the remaining problems --- cleaning,

Technical Challenges

- Representation- Infrastructure- Inference

- General thoughts on KGs ...

Page 25: Data Commons · Data Commons vs collection of datasets Collections of datasets (ala NIH Data Commons). Solves the problem of finding the dataset. But the remaining problems --- cleaning,

Representational Issues

Representing time, geo, measures, provenance, ...

Stat. aggregates exacerbate the problem of long predicates: maleLatinoPopulationUnder...

Triples are easy to understand, relatively widely understood

But real systems need more expressiveness

Page 26: Data Commons · Data Commons vs collection of datasets Collections of datasets (ala NIH Data Commons). Solves the problem of finding the dataset. But the remaining problems --- cleaning,

Infrastructure

Data Commons has about 50B triples

Bio part is another 100B triples

Wide range of queries and latency requirements

- simple queries that require millisecond responses- complex sparql queries- everything in between

Page 27: Data Commons · Data Commons vs collection of datasets Collections of datasets (ala NIH Data Commons). Solves the problem of finding the dataset. But the remaining problems --- cleaning,

Data Commons approach

Multiple ‘backing’ stores, that support different classes of queries with different latencies

- Big Query … in memory hash tables- KG is a view on top of more native encodings- DLG + function terms + contexts + simple kinds of inference

is the real ‘language’ --- serves as the ‘epistemological/knowledge level’

Page 28: Data Commons · Data Commons vs collection of datasets Collections of datasets (ala NIH Data Commons). Solves the problem of finding the dataset. But the remaining problems --- cleaning,

PopulationsObservationsPlacesTriples

Translator

KG Schema <-> Relational Schema maps

Datalog (Sparql, Graphql, …)

SQL

Big Query

Cache Builder

Cache

Graph Nav APIs, Time Series, etc.

Weather

Mixer

Serving System

https://pantheon.corp.google.com/bigquery?project=google.com:datcom-store-dev&p=google.com:datcom-store-dev&d=dc_v3_clustered&t=Triple&page=table&pli=1
Page 29: Data Commons · Data Commons vs collection of datasets Collections of datasets (ala NIH Data Commons). Solves the problem of finding the dataset. But the remaining problems --- cleaning,

Ending thoughts

- Vocabulary creep: Google KG, Cyc, Wikidata all have many tens of thousands of ‘schema’ terms. Human language manages with few thousand terms. How do we bring the compositionality of NL to KR?

- The problems of old AI haven’t been solved. They just can’t be expressed clearly in today’s ML formalisms. e.g., World’s tallest mountain yesterday?


From Data to Knowledge - Profiling & Interlinking Web Datasets

From Data to Knowledge - Profiling & Interlinking Web Datasets

Technology
Creative + Data: Increase the Impact of Your Datasets

Creative + Data: Increase the Impact of Your Datasets

Internet
ArcGIS Data Reviewer: Quality Assessment for Elevation Raster Datasets Roslyn Dunn ArcGIS Data Reviewer: Quality Assessment for Elevation Raster Datasets

ArcGIS Data Reviewer: Quality Assessment for Elevation Raster Datasets Roslyn Dunn ArcGIS Data Reviewer: Quality Assessment for Elevation Raster Datasets

Documents
Data Commons Garvan - 2016

Data Commons Garvan - 2016

Science
Understanding Complex Datasets: Data Mining with Matrix ... › eb53 › 62c08a8bf013... · Understanding Complex Datasets: Data Mining with Matrix Decompositions D.B. Skillicorn

Understanding Complex Datasets: Data Mining with Matrix ... › eb53 › 62c08a8bf013... · Understanding Complex Datasets: Data Mining with Matrix Decompositions D.B. Skillicorn

Documents
Data Commons & Data Science Workshop

Data Commons & Data Science Workshop

Health & Medicine
Derestricting Datasets - How to License Research Data · Why license research data? Licensing options Creative Commons Open Data Commons Open Government Licence GILF/AusGOAL Licences

Derestricting Datasets - How to License Research Data · Why license research data? Licensing options Creative Commons Open Data Commons Open Government Licence GILF/AusGOAL Licences

Documents
Uploading datasets to data site

Uploading datasets to data site

Documents
Data Model and Required Datasets

Data Model and Required Datasets

Documents
Today’s Topics Data Commons’ approach Data Commons’ Services Use of MedBiquitious Professional Profile standard

Today’s Topics Data Commons’ approach Data Commons’ Services Use of MedBiquitious Professional Profile standard

Documents
Creative Commons & Open Data

Creative Commons & Open Data

Technology
Using FME to Create Open Data Datasets

Using FME to Create Open Data Datasets

Technology
The NCI Cancer Research Data Commons NCI Cancer... · NaBonal Cancer Data Ecosystem NCI Cancer Research Data Commons NIH Data Commons Pilot Data Commons co-locate data, storage and

The NCI Cancer Research Data Commons NCI Cancer... · NaBonal Cancer Data Ecosystem NCI Cancer Research Data Commons NIH Data Commons Pilot Data Commons co-locate data, storage and

Documents
NCI Cancer Research Data Commons · NCI Cancer Research Data Commons (CRDC) - Concept . Data commons co-locate data, storage and computing infrastructure with commonly used services,

NCI Cancer Research Data Commons · NCI Cancer Research Data Commons (CRDC) - Concept . Data commons co-locate data, storage and computing infrastructure with commonly used services,

Documents
Towards a Data Commons

Towards a Data Commons

Health & Medicine
NHS Hospital Data and Datasets: A Consultation · PDF fileNHS Hospital Data and Datasets: A Consultation IMPROVE ... NHS HOSPITAL DATA AND DATASETS: ... outcome and patient satisfaction,

NHS Hospital Data and Datasets: A Consultation · PDF fileNHS Hospital Data and Datasets: A Consultation IMPROVE ... NHS HOSPITAL DATA AND DATASETS: ... outcome and patient satisfaction,

Documents
Data integration with diverse datasets

Data integration with diverse datasets

Documents
Data harmonization in diverse datasets - BIG DATA SCIENCE

Data harmonization in diverse datasets - BIG DATA SCIENCE

Documents
The Global Commons of Data

The Global Commons of Data

Documents
AnaCredit Reporting Manual - Part II – Datasets and data ... · Part II – Datasets and data attributes ... AnaCredit Reporting Manual – Part II – Datasets and data attributes

AnaCredit Reporting Manual - Part II – Datasets and data ... · Part II – Datasets and data attributes ... AnaCredit Reporting Manual – Part II – Datasets and data attributes

Documents
BRC 2011 Session #4 – “Omics” Data. Session #4 - Outline Challenges and Opportunities pathogen datasets; host datasets; integrating pathogen-host datasets

BRC 2011 Session #4 – “Omics” Data. Session #4 - Outline Challenges and Opportunities pathogen datasets; host datasets; integrating pathogen-host datasets

Documents
Semantic Interoperability in Archaeological Datasets: Data Mapping

Semantic Interoperability in Archaeological Datasets: Data Mapping

Documents
Toward a Mobile Data Commons

Toward a Mobile Data Commons

Data & Analytics
Chapter 18 - Data sources and datasets

Chapter 18 - Data sources and datasets

Documents
NIH Data Summit - The NIH Data Commons

NIH Data Summit - The NIH Data Commons

Science
Disconnected Data Applications: using Datasets, XML and Transactions

Disconnected Data Applications: using Datasets, XML and Transactions

Documents
Building the data commons

Building the data commons

Technology
Data Preparation and Use of WHI datasets

Data Preparation and Use of WHI datasets

Documents
The UK Environmental Change Network datasets – integrated ... · the Creative Commons Attribution 4.0 License. The UK Environmental Change Network datasets – integrated and co-located

The UK Environmental Change Network datasets – integrated ... · the Creative Commons Attribution 4.0 License. The UK Environmental Change Network datasets – integrated and co-located

Documents
Querying Heterogeneous Datasets on the Linked Data Web

Querying Heterogeneous Datasets on the Linked Data Web

Technology