Embed Size (px)
Citation preview
Big Data, Big Headache?
Statement
“GIMA students are not well equipped to
deal with Big Data in their future careers”
Who am I?
Tom Broersen
Who am I?
Tom Broersen
Who am I?
Tom Broersen
Big Data, Big Headache?
Ask the co-workers
Ask Google
MANY different definitions
“Too big for one computer”
“When the size becomes a problem”
“Distributed systems and non-
conventional database technologies”
MANY different definitions
“Too big for one computer”
“When the size becomes a problem”
“Distributed systems and non-
conventional database technologies”
All is relative… AHN3 and NDW
AHN3: 1.5 billion points
> 1TB in size
NDW traffic data: 27.000
measurements per minute
All is relative… LHC @CERN
The Large Hadron Collider (LHC) at CERN Data Centre:
• 600 million collisions per second
• Filtered to 1GB/s (roughly 80.000 GB/day)
• >> 10.000 processing cores
All is relative… SKA TELESCOPE
The SKA project (2024):
• One exabyte of data every day
(1000 * 1000 * 1000 GB)
• Requires millions of
traditional servers
All is relative…
AHN3: 1TBNDW:
27.000 per minute
LHC: 80TB/day
SKA: exabyte/day
Who cares?
“The definition of Big Data? Who
cares? It’s what you’re doing with it.”
It’s not about the definition
BIG DATA
< What we know…
< The rest…
How will Big Data change our lives?
How will Big Data change our lives?
Comprehensive picture of the patient as an individual
How will Big Data change our lives?
De-centralised power grid
How will Big Data change our lives?
Finance
How will Big Data change our lives?
Shift planning and avoiding “out of stock conditions”
High potential in GEO-data
Geo (Big) Data apps
But: minor role of GIS
Big Data, Big Headache?
But: minor role of GIS
Are you well equipped?
Data Engineer
“Hacker mindset”
“Builds the product”
System implementation
Scripting (Python, JavaScript)
Databases (SQL, NoSQL)
Big Data (Hadoop, Spark)
Data Scientist
“Curious about data”
“Asks the right questions”
Statistics (R)
Story-telling
Visualisation and cartography
Analysis (ESRI ArcGIS)
Are you well equipped?
Data Scientist
“Curious about data”
“Asks the right questions”
Statistics (R)
Story-telling
Visualisation and cartography
Analysis (ESRI ArcGIS)
Are you well equipped?
Data Engineer
“Hacker mindset”
“Builds the product”
System implementation
Scripting (Python, JavaScript)
Databases (SQL, NoSQL)
Big Data (Hadoop, Spark)
Are you well equipped?
Data Scientist
“Curious about data”
“Asks the right questions”
Statistics (R)
Story-telling
Visualisation and cartography
Analysis (ESRI ArcGIS)
Are you well equipped?
Data Engineer
“Hacker mindset”
“Builds the product”
System implementation
Scripting (Python, JavaScript)
Databases (SQL, NoSQL)
Big Data (Hadoop, Spark)
Are you well equipped?
Data Engineer
“Hacker mindset”
“Builds the product”
System implementation
Scripting (Python, JavaScript)
Databases (SQL, NoSQL)
Big Data (Hadoop, Spark)
Data Scientist
“Curious about data”
“Asks the right questions”
Statistics (R)
Story-telling
Visualisation and cartography
Analysis (ESRI ArcGIS)
Statement
“GIMA students are not well equipped to
deal with Big Data in their future careers”
Are you well equipped?
Data Scientist Data Engineer
“Curious about data”
“Asks the right questions”
“Hacker mindset”
“Builds the product”
System implementation
Scripting (Python, JavaScript)
Databases (SQL, NoSQL)
Big Data (Hadoop, Spark)Statistics (R)
Story-telling
Visualisation and cartography
Analysis (ESRI ArcGIS)
Are you well equipped?
Data Scientist Data Engineer
“Curious about data”
“Asks the right questions”
“Hacker mindset”
“Builds the product”
System implementation
Scripting (Python, JavaScript)
Databases (SQL, NoSQL)
Big Data (Hadoop, Spark)Statistics (R)
Story-telling
Visualisation and cartography
Analysis (ESRI ArcGIS)
How to build a great resume?
(Modern)
Who has the future?
