Upload
andertech
View
113
Download
1
Embed Size (px)
DESCRIPTION
Baron Schwartz' slides from his Making big data small at Strata Conf 2013
Citation preview
MAKING BIG DATA SMALLBARON SCHWARTZ
STRATA CONFERENCE NYC 2013
ME@XAPRB ON TWITTER
LINKEDIN.COM/IN/XAPRB
Optimization, Backups, Replication, and more
Baron Schwartz, Peter Zaitsev &
Vadim Tkachenko
High PerformanceMySQL
3rd Edition
Covers Version 5.5
STORAGETRANSFER / BANDWIDTHCOMPUTE (CPU, RAM, I/O)TIME
DATA LIFECYCLE
COMPUTERETRIEVESTORETRANSFERGATHER
THE MEGABYTE HOUR WHAT DOES 350 MILLISECONDS COST?
BIG-O NOTATION
GUIDING PRINCIPLES
DO MORE WITH LESS
ADDRESS THE AUDIENCE
GENERATE KNOWLEDGE
DATA
INFORMATION
KNOWLEDGE
SUCH KNOWLEDGE IS TOO WONDERFUL FOR ME;IT IS HIGH, I CANNOT ATTAIN IT.
- PSALM 139
ANSWER THE QUESTION
DOING MORE WITH LESS
LOSSLESS COMPRESSION
LOSSY REPRESENTATION
DATA LIFECYCLE
COMPUTERETRIEVESTORETRANSFERGATHER
THE RISK?
LESS COMPUTELESS DATA
LOSSINESS FEATURES
TECHNIQUES
LESS COMPUTELESS DATA
LOSSINESS FEATURES
TECHNIQUES
DECREASE REDUNDANCY
LESS D
ATA
DELTA COMPRESSION
LESS D
ATA
NULLS AND ZEROES
LESS D
ATA
SAMPLING
LESS D
ATA
AGGREGATION
LESS D
ATA
DISCARDING DUPLICATE SIGNALS
LESS D
ATA
COMPRESSION
LESS D
ATA
LESS COMPUTELESS DATA
LOSSINESS FEATURES
TECHNIQUES
BLOOM FILTERS
LESS C
OM
PUT
E
EXPONENTIALLY WEIGHTED MOVING AVERAGES
LESS C
OM
PUT
E
0
100
200
300
400
1 2 3 4 5
0
100
200
300
400
1 2 3 4 5
APPROXIMATION
LESS C
OM
PUT
E
LESS COMPUTELESS DATA
LOSSINESS FEATURES
TECHNIQUES
MODELS
LO
SSINESS
LO
SSINESS
LO
SSINESS
LESS COMPUTELESS DATA
LOSSINESS FEATURES
TECHNIQUES
HISTOGRAMS
FEAT
UR
ES
HISTOGRAMS
FEAT
UR
ES
PERCENTILES
FEAT
UR
ES
SPARSE OUTLIER STORAGE
FEAT
UR
ES
LOST HAYSTACKS
WHAT ABOUT THE PIONEER ANOMALY?
SMALL DATA METHOD
APPLYDISCARDISOLATEEXPLOREGATHER
SAMPLES
VISUALIZE
NEEDLES!
HAYSTACK
REVIEW
SMART-FRIENDS METHOD
CONTACT / QUESTIONS@XAPRB ON TWITTER
LINKEDIN.COM/IN/XAPRBPLEASE
RATE THIS TALK :-)
IMAGE CREDITSHTTP://WWW.FLICKR.COM/PHOTOS/PIPER/458847242/
HTTP://WWW.FLICKR.COM/PHOTOS/MIONG/4222013704/
HTTP://WWW.FLICKR.COM/PHOTOS/BENSTEVINSON/4908627827/
HTTP://EN.WIKIPEDIA.ORG/WIKI/FILE:BLOOM_FILTER.SVG
HTTP://WWW.FLICKR.COM/PHOTOS/JINGLESLENOBEL/4915172257/
HTTP://WWW.FLICKR.COM/PHOTOS/MAUROPPP/429231880/
HTTP://WWW.FLICKR.COM/PHOTOS/ECSTATICIST/3979554514/
HTTP://WWW.FLICKR.COM/PHOTOS/TAMBAKO/9158311021/
HTTP://WWW.FLICKR.COM/PHOTOS/FUZZYLITTLEMANPEACH/4689679368/
HTTP://WWW.FLICKR.COM/PHOTOS/FWP/49514206/
HTTP://WWW.FLICKR.COM/PHOTOS/QTHOMASBOWER/3511960429/
HTTP://WWW.FLICKR.COM/PHOTOS/JIMNISTA/7239478006/
HTTP://WWW.FLICKR.COM/PHOTOS/FLOWERS-OF-THE-SEA/6971586987/
HTTP://WWW.FLICKR.COM/PHOTOS/OLDPATTERNS/5817264314/
HTTP://WWW.FLICKR.COM/PHOTOS/BREWBOOKS/184343329/
HTTP://WWW.FLICKR.COM/PHOTOS/PAREEERICA/4235666733/
HTTP://WWW.FLICKR.COM/PHOTOS/JDHANCOCK/5177358991/
HTTP://WWW.FLICKR.COM/PHOTOS/ROSEMARY/47396922/
HTTP://WWW.FLICKR.COM/PHOTOS/ECSTATICIST/1337749333/
HTTP://WWW.FLICKR.COM/PHOTOS/PINKSHERBET/233228813/
HTTP://BOST.OCKS.ORG/MIKE/SIMPLIFY/
HTTP://WWW.FLICKR.COM/PHOTOS/PSOUP216/7540279198/
HTTPS://GITHUB.COM/BRENDANGREGG/HEATMAP/WIKI
HTTP://WWW.FLICKR.COM/PHOTOS/WALKINGSF/6221430560/
HTTP://WWW.FLICKR.COM/PHOTOS/WALKINGSF/7835295762/
HTTP://WWW.FLICKR.COM/PHOTOS/NASAMARSHALL/8615696355/
HTTP://WWW.FLICKR.COM/PHOTOS/BOOLEANSPLIT/8482641188/
HTTP://WWW.FLICKR.COM/PHOTOS/HTAKASHI/6861028238/
HTTP://WWW.FLICKR.COM/PHOTOS/ZDETO/4383899069/
JPL VIA HTTP://WWW.PLANETARY.ORG/EXPLORE/PROJECTS/PIONEER-ANOMALY-1/PIONEER_ANOMALY_STAIRS_DATA_TAPES_3.HTML
HTTP://EN.WIKIPEDIA.ORG/WIKI/FILE:PIONEER_10-11_SPACECRAFT.JPG