ELASTICSEARCH @SYNTHESIO
FRED DE VILLAMIL, DIRECTOR OF INFRASTRUCTURE @FDEVILLAMIL FEBRUARY 2016
BACKGROUND
• FRED DE VILLAMIL, 37 ANS, DIRECTOR OF INFRASTRUCTURE @SYNTHESIO
• LINUX / (FREE)BSD SINCE 1996 • OPEN SOURCE CONTRIBUTOR SINCE 1998 • RUNS ELASTICSEARCH IN PRODUCTION SINCE 0.17.6
ABOUT SYNTHESIO
• Synthesio is the leading social intelligence tool for social media monitoring & social analytics.
• Synthesio crawls the Web for relevant data, enriches it with sentiment analysis and demographics to build social analytics dashboards.
ELASTICSEARCH @SYNTHESIO, JANUARY 2016
• 3 clusters, 116 physical servers, 271TB storage, 7,25TB RAM
• 60B indexed documents, 160TB data • 1.5B new documents each month: mix of Web
pages, forums and social media posts
DEC. 2014: THE MYSQL NIGHTMARE
• Cross clusters queries on 3 massive Galera clusters • Up to 50M rows fetched from a massive 4B rows
reference table • Then a cross cluster joint on a 20TB, 35B records
monolithic MySQL database • Poor performances, frequent timeouts
JAN. 2015: CLIPPING REVOLUTION
JAN. 2015: CLIPPING REVOLUTION
• 1 global index, 512 shards, 5B documents • 1000 new documents / second • 47 servers running ElasticSearch 1.3.2 then 1.3.9 • Capacity : 37TB, 24TB data, 2.62TB RAM
CLUSTER TYPOLOGY
• 2 HTTP NODES: VIRTUAL MACHINES, 4 CORE, 8GB RAM EACH
• 3 MASTER NODES : VIRTUAL MACHINES, 4 CORE, 8GB RAM • 42 DATA NODES : PHYSICAL SERVERS, 6 CORE XEON
E5-1650 V2, 3*900GB SSD IN RAID 0, 64GB RAM
CLIPPING REVOLUTION DATA MODEL
• ROUTING ON A MONTHLY BASIS
• EACH CRAWLED DOCUMENT IS INDEXED WITH NESTED DASHBOARD IDS.
• QUERIES ON TIME PERIOD + DASHBOARD ID
{ "document": { "dashboards": { "dashboard_id": 1, "dashboard_id": 2 } } }
PROBLEMS
• TOO MANY SHARDS (WAS MEANT TO BE A WEEKLY ROUTING)
• 500GB TO 900GB SHARDS (!!!) GROWING AFTER THE MONTH IS OVER. 3 HOURS FOR A REALLOCATION
• GARBAGE COLLECTOR NIGHTMARE, CONSTANTLY FLAPPING CLUSTER
• A ROLLING RESTART TAKES 3 FULL DAYS (IF WE’RE LUCKY)
MORE PROBLEMS
• IMMUTABLE, MONOLITHIC MAPPING: NEW FEATURE BLOCKED UNTIL WE FIX IT
• IMPOSSIBLE TO DELETE A DASHBOARD WITHOUT REINDEXING A WHOLE MONTH
• 20% DELETED DOCUMENTS WASTING 3TB
IMMUTABLE MAPPING AND DELETED DATA
• SEGMENTS : IMMUTABLE FILES USED BY LUCENE TO WRITE ITS DATA. UP TO 2500 / SHARD (!!!)
• NO REAL DELETE: DELETED DOCUMENTS GET THE DELETED FLAG
• ELASTICSEARCH _OPTIMIZE: MERGE A SHARD SEGMENTS IN 1 AND PURGE DELETED DOCS
• BUT: REQUIRES 150% OF THE SHARD SIZE ON DISK
MERGE AND DELETE The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.
SOLUTIONS
• STORAGE MOVE FROM MMAPFS TO NIOFS • GARBAGE COLLECTOR SWITCH FROM CMS TO G1GC • FORCED EXPIRE ON THE FIELDDATA CACHE
MMAPFS VS NIOFS
• MMAPFS : MAPS LUCENE FILES ON THE VIRTUAL MEMORY USING MMAP. NEEDS AS MUCH MEMORY AS THE FILE BEING MAPPED
• NIOFS : APPLIES A SHARED LOCK ON LUCENE FILES AND RELIES ON THE FILE SYSTEM CACHE
CMS VS G1GC
• CMS: SHARED CPU TIME WITH THE APPLICATION. “STOPS THE WORLD” WHEN TOO MANY MEMORY TO CLEAN UNTIL IT SENDS AN OUTOFMEMORYERROR
• G1GC: SHORT, MORE FREQUENT, PAUSES. WON’T STOP A NODE UNTIL IT LEAVES THE CLUSTER
G1GC OPTIONS
JAVA_OPTS="$JAVA_OPTS -XX:+USEG1GC "
JAVA_OPTS="$JAVA_OPTS -XX:MAXGCPAUSEMILLIS=200"
JAVA_OPTS="$JAVA_OPTS -XX:GCPAUSEINTERVALMILLIS=1000"
JAVA_OPTS="$JAVA_OPTS -XX:INITIATINGHEAPOCCUPANCYPERCENT=35"
FIELD DATA CACHE EXPIRE
• FORCES ELASTICSEARCH TO PERIODICALLY EMPTY ITS INTERNAL FIELDDATA CACHE
• OVERLAPS THE GARBAGE COLLECTOR JOB • PERFORMANCES ISSUES WITH FREQUENTLY ACCESSED DATA • ELASTIC SAYS NEVER DO THIS!!! BUT IT FIXED OUR
BIGGER PROBLEM
FIELDDATA OPTIONS
FIELDDATA: BREAKER:
LIMIT: 80% CACHE:
SIZE: 30%
EXPIRE: 1M
DEC. 2015: BLACKHOLE, THE WELL NAMED
BLACKHOLE
• 36 INDEXES, 50B DOCUMENTS • 1,5B NEW DOCUMENTS EACH MONTH • 72 SERVERS RUNNING ELASTICSEARCH 1.7.5 • CAPACITY: 209TB, 120TB DATA, 4,5TB RAM • QUERIES ON THE WHOLE DATASET
CLUSTER TYPOLOGY
72 4 CORE XEON D-1520, 64GB RAM SERVERS • 2 HTTP NODES • 3 MASTER NODES • 67 DATA NODE : 4*800GB SSD IN RAID 0
INDEXING
• A GO PROGRAM MERGES 3 GALERA CLUSTERS AND 1 ES CLUSTER INTO A KAFKA QUEUE: 30.000 DOCUMENTS / SECOND
• 8 GO INDEXERS MAP THE INDEX / DATA NODE DISTRIBUTION AND PUSH THE DATA DIRECTLY ON THE RIGHT DATA NODE: 60.000 DOCUMENTS / SECOND DURING 3 WEEKS WITH 120.000 / SECOND PEAKS
INDEXING PROCESS
PROBLEMS
• THE XEON D CPU CAN’T COPE WITH OUR INDEXING PACE • THE 50TB KAFKA QUEUE IS TOO SLOW
PROBLEM: WE’RE CPU BOUND
POST INDEXING PROBLEMS
• QUERIES ON THE WHOLE DATA WORKS, FILTERED QUERIES CRASHES HALF OF THE CLUSTER
• BIGGEST QUERIES ARE SLOW AS HELL
SOLUTIONS
• ELASTICSEARCH CACHES THE RESULT OF FILTERED QUERIES: SET _CACHE TO FALSE.
• UPGRADE TO 1.7.5: FILTERED QUERIES CACHE HAVE A MEMORY LEAK IN 1.7.4
• DIVIDE THE GLOBAL QUERIES PER INDEX AND RUN IN PARALLEL
JAN. 2016: BLINK
JAN. 2016: BLINK
• 8200 indexes, 10B documents • 25000 new document / second • 70 physical servers running ElasticSearch 1.7.4,
then 1.7.5 • Capacity: 60TB, 50TB data 2.62TB RAM
CLUSTERS TYPOLOGY
2 CLUSTERS RUNNING 4 CORE XEON D-1520, 64GB RAM SERVERS • 2 HTTP NODES • 3 MASTER NODES • 30 DATA NODE : 2*480GB SSD IN RAID 0
NEW PRODUCT DESIGN
• 1 INDEX / DASHBOARD, 1 SHARD / 5 MILLIONS DE DOCS • VERSIONED MAPPING: MAPPING_ID__DASHBOARD_ID • MULTIPLE MAPPING VERSIONS OF A DASHBOARD IN // • MAPPING UPGRADE AND REINDEX WITHOUT INTERRUPTION • BALDUR FOR DASHBOARDS ROUTING
BALDUR
BALDUR IN A NUTSHELL
1. THE API SERVER SENDS AN ELASTIC SEARCH QUERY 2. BALDUR INTERCEPTS THE QUERY AND GETS THE
DASHBOARD CLUSTER ID AND ACTIVE MAPPING VERSION 3. BALDUR ROUTES THE QUERY TO THE CLUSTER HOSTING
THE DASHBOARD DATA
ADDING A MAPPING VERSION
• THE INDEXER CREATES NEW NEW_MAPPING_ID__DASHBOARD_ID INDEX
• THE INDEXER ADDS A LINE IN BALDUR’S DATABASE WITH THE DASHBOARD AND MAPPING IDS
• THE INDEXERS INDEXES BOTH MAPPING_ID__DASHBOARD_ID AND NEW_MAPPING_ID__DASHBOARD_ID
• WHEN NEW_MAPPING_ID__DASHBOARD_ID HAS CAUGHT UP, BALDER SWITCHES THE ACTIVE MAPPING
PROBLEMS
• EACH DATA NODE HOSTS 1000S LUCENE SEGMENTS • 75% OF THE HEAP IS USED FOR SEGMENTS MANAGEMENT • WE CREATE MORE SEGMENTS THAN WE’RE ABLE TO OPTIMIZE • 5000 DOCS, RANDOM READS BASED, BULK INDEXING PUTS
MYSQL ON THEIR KNEES • FREQUENT CONFIGURATION UPDATES DURING THE RAMP UP
ARE COMPLICATED
TOO MANY LUCENE SEGMENTS
• CONTINUOUS OPTIMISATION SCRIPTS • CONTINUOUS OLD INDEXES CLEANUP
CUT MYSQL A LITTLE SLACK
• FETCH THE DOCUMENTS IN BLACKHOLE BY 5000 • IF SOME DOCUMENTS ARE MISSING, FETCH IN MYSQL • RESULT: 99.9% DOCUMENTS EXTRACTED FROM
BLACKHOLE, THROUGHPUT * 5
REPLACE MYSQL WITH BLACKHOLE
30 MINUTES FULL RESTART $FQDN: CURL -XPUT 'HTTP://ESMASTER01.ESCLUSTER02.PRODUCTION.INT:9200/_CLUSTER/SETTINGS' -D '{ "TRANSIENT" : { "CLUSTER.ROUTING.ALLOCATION.ENABLE": "NONE" } }' $FQDN: ANSIBLE-PLAYBOOK -I PRODUCTION -E RADCK_ID=PATRACK -L DATANODES ROLLING.YML $FQDN: CURL -XPUT 'HTTP://ESMASTER01.ESCLUSTER02.PRODUCTION.INT:9200/_CLUSTER/SETTINGS' -D '{ "TRANSIENT" : { "CLUSTER.ROUTING.ALLOCATION.ENABLE": "ALL" } }' WAIT FOR THE GREEN LIGHT, PROCESS THE OTHER RACK ID.
RACK AWARENESS
RACK AWARENESS IN A NUTSHELL
1. DEFINE 2 VIRTUAL RACK IDS 2. ASSIGN DATA NODES A RACK 3. ENABLE RACK AWARENESS 4. PRIMARY SHARDS PICK UP A SIDE, REPLICA PICK UP THE
OTHER ONE
ADDING NEW DOCUMENTS TO A DASHBOARD
WHERE DO YOU GO, MY LOVELY?
• PROBLEM: HOW DO WE KNOW IN WHICH DASHBOARD FITS A NEW DOCUMENT
• STOP RELYING ON MYSQL AND SPHINX • 50M NEW DOCUMENTS TO PROCESS A DAY
SOLUTION : PERCOLATION
• REVERSE DIRECTORY SYSTEM • WE STORE QUERIES, NOT DOCUMENTS • FOR EACH NEW DOCUMENT, WE MATCH THE DOCUMENT
AGAINST OUR QUERIES
PERCOLATION ISSUES
• IT TRIES TO MATCH EVERY STORED QUERY • SO FAR, WE HAVE 35000 STORE QUERIES • RAW USE: 1,750,000,000,000 MATCHES A DAY • CPU GREEDY
SOLUTIONS
• ROUTING WITH THE DASHBOARD AND DOCUMENT LANGUAGES
• FILTER AGAINST THE QUERY SECOND • RESULT: UP TO 100,000 QUERIES / SECOND
QUESTIONS ?
@FDEVILLAMIL @SYNTHESIO
Recommended
