View
420
Download
5
Category
Preview:
Citation preview
A Database story by DevOpsexploring SQL Database in Production.
Anton Martynenko
Dev & QA & OPS = DevOpsHead of Technical Operations at Youscan
.Net, Windows Azure, Sql Server, NoSql…
Twitter: @aamartynenkoam@youscan.biz
YouScan is a leader of the emerging social media monitoring & analytics market in Russia and Ukraine.
Our products:http://youscan.ruhttp://leadscanner.ru
The Reality:
Time is limited
Talk covers a lot of stuff
No time to go into details
If you have a question – make notes and ask later
The Production
Main DB
WWWAndroid
appService iOs app
The Production with more databases
Main DBClientsTools
WWWAndroid
appService iOs app
The Production
Main DB
WWWAndroid
appService iOs app
Database as a Black Box
WWWAndroid
appService iOs app
Database as a source of problems
Main DB
WWWAndroid
appService iOs app
Main DB
Firewall for production DB
WWWAndroid
appService iOs app
DevOps – Developer with access to Prod
Main DB
WWWAndroid
appService iOs app
Typical Database headaches
1.Performance
2.Whatever-SQL mess in DB
3.Maintenance
Main DB
Typical Database headaches
1.Performance
2.Whatever-SQL mess in DB
3.Maintenance
Main DB
Performance issues
Memory
Disk I/O bottlenecks
High CPU usage
Know your Production hardware!
- Memory
- Disk I/O
- CPU usage
Understand your load
Latencies are important
Understand Database “Math”
Tables
Indexes
Queries
Understand Transaction Isolation
Serializable
Repeatable reads
Read committed
Read uncommitted
Performance: User Experience
WWW
“Place order”
“Success”
Main DB
User’s Transaction is complex
WWW
“Place order”
“Success”
Main DB
User’s Transaction is complex
WWW
“Place order”
“Success”
Main DB
So Slow!
Performance: TOP I/O Queries
Performance: TOP CPU Queries
Google to find TOP IO/CPU Queries
Catch the bottleneck
CPU Score Query
13543541 SELECT * FROM Products WHERE …
7545314 SELECT * FROM Users where …
567541 INSERT INTO Reviews …
IO Score Query
68743513 SELECT * FROM Authors INNER JOIN...
1475214 INSERT INTO OrderQueue …
1246876 INSERT INTO SystemLogs …
Analyze Top IO Queries
Analyze Top CPU Queries
Understand your query
Understand indexes used by query
Understand execution plan
Slow “SELECT”
Missing Index
Suboptimal Query
Low Memory (or Big Index)
Low disk IOPS (or too many queries)
Slow “INSERT”, “UPDATE”
Too many Indexes
Suboptimal Query
Low disk IOPS (or too many inserts)
Understand the query
Catch!
Where is the problem?
Missing Index
Suboptimal Query
Low Memory (or Big Index)
Low disk IOPS (or too many queries)
Too many Indexes
Root cause: missing index
Missing Index
Suboptimal Query
Low Memory (or Big Index)
Low disk IOPS (or too many queries)
Too many Indexes
Review existing indexes Add new index Or change existing
index
Root cause: query
Missing Index
Suboptimal Query
Low Memory (or Big Index)
Low disk IOPS (or too many queries)
Too many Indexes
Understand execution plan Use Sql query parameters Do select/update minimal
field set
Root cause: low memory
Missing Index
Suboptimal Query
Low Memory (or Big Index)
Low disk IOPS (or too many queries)
Too many Indexes
Review fields in index Archive old data Check DB Engine config Add more memory
Root cause: low IOPS
Missing Index
Suboptimal Query
Low Memory (or Big Index)
Low disk IOPS (or too many queries)
Too many Indexes
Use partitioning Go to SSD Get rid of queries
Root cause: too many indexes
Missing Index
Suboptimal Query
Low Memory (or Big Index)
Low disk IOPS (or too many queries)
Too many Indexes Remove unused Remove features
Many indexes = many features
Remove uselessfeatures
Analyze user behavior
Big Table
Table Partitioning
Jan 2014
Feb 2014
Mar 2014
Apr 2014
May 2014
June 2014
Table Orders, All Data
Partitioned table Orders, Partition = Month
Database Sharding Multiple instances,Shard per country
Main DB
GB Main DB
DE Main DB
FR Main DB
Single instance,All Data
Multitenancy Multiple instances,Shard per client
Main DB
Coca Cola Main DB
BMWMain DB
ShellMain DB
Single instance,All Data
NestleMain DB
Macdonald’sMain DB
SiemensMain DB
DB License very expensive
Coca Cola Main DB
BMWMain DB
ShellMain DB
NestleMain DB
Macdonald’sMain DB
SiemensMain DB
Typical Database headaches
1.Performance
2.Whatever-SQL mess in DB
3.Maintenance
Main DB
Databases do a few things really well:
• They store and retrieve data
• They enforce relationships between data entities
• They provide the means to query the data for answers
T-SQL, PL-SQL, Whatever-SQL are poor as programming languages
SQL programming tools sux
Debugging and testing is not easy…
Typical T-SQL or PL-SQL mess in Database
What about Refucktoring your SQL code?
Typical T-SQL or PL-SQL mess in Database
Java/C#/etc. with modern tools
Typical T-SQL or PL-SQL mess in Database
WTF your code is doing in Database?
There are no reasons to put your Business Logic
in Database
Data -> Database
Data access -> DAL
Transaction -> (not sql)Code
Deadlocks
Complex code --> Deadlocks
ORM
ORM
Main DB
Application
In Legacy systems more often than notORM adds complexity to the system
ORM
ORM is magic
?
Main DB
Application
Main DB
Lightweight ORM
Application
Typical Database headaches
1.Performance
2.Whatever-SQL mess in DB
3.Maintenance
Main DB
Manage schema changes
Change = Script
Two-staged schema changes
Antipattern: Configuration in DB
WWWAndroid
appService iOs app
Antipattern: Lookup tables in DB
Antipattern: Queuing in Database
Keep Schema clean and simple
Get rid of configuration in DB
Get rid of lookup tables in DB
Get rid of write contention
Long term goals:
Database size
Restore/backup
10 minutes
1 Hour
12 Hours
X 3 days
Do Archive old data
Trust is foundation for everything
Thanks! Questions?
Me: https://www.linkedin.com/pub/anton-martynenko/10/289/9b5
YouScan - http://youscan.ru/LeadScanner - http://leadscanner.ru/
Monitoring & Metrics:Zabbix - http://www.zabbix.com/KissMetrics - https://www.kissmetrics.com/Kibana - http://www.elasticsearch.org/overview/kibana/
Recommended