Embed Size (px)
Citation preview
Scaling Dubsmash's backendfrom 0 to 100+ million users
PYCON.DE Munich - Daniel Taschik – 10/29/2016
We hit a nerve.
>100M Users
192 Countries
1.5BVideos
Dubsmash
Connect Create Communicate
The Start
The Start
Backend• Django-powered BE for content
management• web-based Dubloader to add
sounds• deployed on Heroku
Content Delivery• sound files in S3• meta information in JSON file in S3• files served via Cloudfront CDN
Metrics• Dubloader with < 100 req/min• >500 TB! of traffic in January 2015
Dubsmash Service Landscape
Backend
Router
S3 sound storage
Cloudfront CDN
New Features: Registration & Search
User registration • API based on REST framework• Django user model• store user’s most like sounds• push notifications for new content
Server-side Sound search• new Django-based service• search via ElasticSearch using Haystack• Celery-based indexing on RQ
Metrics• 100.000 registrations within first 24h• >20.000 requests per minute on search service
caching
Cloudfront CDN
S3 sound storage
Dubsmash Service Landscape
Searc
h
Router
main API
DubTalk
Social Graph Service• friend relations on platform• Django • TitanDB on Cassandra• later DynamoDB
DubTalk Service• group & video management• Django
Service Communication• Async via Celery on RabbitMQ• Sync via internal HTTPS API
Metrics• > 50.000 requests per min on both• > 150.000.000 videos stored
NoSQL
Cloudfront CDN
caching
S3 sound storage
Dubsmash Service Landscape
GraphDubTalk
Router
Monolith
relational DB
Large Scale Problems
favorited sounds outgrew our PostgreSQL
• > 1.000.000.000 favorited sounds• simple data model & access pattern• Premium-7 120GB RAM, 1TB disk instance
dtaschik@unic0rn:~/dubsmash$ heroku pg:table-size -a dubsmash
name | size-----------------------------------+------------ users_favs | 158 GB
dtaschik@unic0rn:~/dubsmash $ heroku pg:index-size -a dubsmashname | size-----------------------------------+------------ users_favs_username_key | 132 GB
ID username
sound_id
1 daniel3 Dzdcjc
2 sarah 3jGYzH
Let’s make it a new service!
Cloudfront CDNS3 sound storage
Dubsmash Service Landscape
Auth
Graph DubTalk
Favs
Router
Monolith
relational DB
caching
NoSQL
many more
Our Goal
Interested? Come and join! 😜
Building the largest mobile video communication platform!
