Upload
jesse-young
View
16.801
Download
1
Embed Size (px)
DESCRIPTION
These are the slides for the HA+DRBD+Postgres talk I gave at PostgresWest '08.
Citation preview
Linux‐HA+DRBD+PostgreSQL
JesseYoung
TargetAudience
• SystemAdministrators• DBAslookingintoreplicaBon
Goals
• InspiremorereplicaBondiscussions• InformSystemAdmins/DBAs
• Describecomponents
• SimpleSetup
• DemonstraBon
• PerformanceTips
Intro
• ZonarSystems,Inc.– 45,000vehiclessendingdatain24/7andgrowing– 15databaseservers– 5TBdata,growing~25GBdaily– PostgreSQL+PostGIS
WhyuseLinux‐HA+DRBD?
• SimplereplicaBon• GPSdatacomingintothesystem24/7
• MiBgateproblemsfromserverhardware
• UsersandIntegraBonsoYwareaccess
Linux‐HA
• “Provide a high availability (clustering) solu6on for Linux which promotes reliability, availability, and serviceability (RAS) through a community development effort.”
• SystemAdmins:YoucanconBnueto(half)sleep,evenifaserverdies.
DRBD• DistributedReplicatedBlockDevice
• Mirroredstorageacrossservers
Imagefromh_p://www.drbd.org
PostgreSQL
• Databaseengineweallknowandlove
Setup
• Dual‐procQuad‐[email protected]• 16GBRAM
• 2x74GBSCSI@15kRPM(RAID1‐OS)
• 6x300GBSCSI@10kRPM(RAID10–DRBD)
• LSIMegaraid–2Channel,512MBCachew/BBU
QuickInstall
• Versions– heartbeat‐2.0.8– drbd‐8.0.5– PostgreSQL–Anyversion
InstallDRBD
• Compilefromsourceor
• yuminstalldrbdkmod‐drbd
ConfigureDRBDresourcepostgres{onnode‐001{device/dev/drbd0;disk/dev/sdb1;address10.40.0.1:7788;meta‐diskinternal;}
onnode‐002{device/dev/drbd0;disk/dev/sdb1;address10.40.0.2:7788;meta‐diskinternal;}}
BeginDRBD
• Eachnode:– modprobedrbd
– drbdadmcreate‐mdpostgres– drbadmupall
CreateParBBononDRBDDevice
• Primarynode:– drbdadm‐‐‐‐overwrite‐data‐of‐peerprimaryall
– mke2fs‐j/dev/drbd0– mountdevice– Edit/etc/init.d/postgresql• PGDATAtoDRBDdevice
– initdborsync/copyexisBngdatabase
DRBDstatus
• cat/proc/drbd
version:8.0.5(api:86/proto:86)
SVNRevision:3011buildbyjesse@node‐001.zonarsystems.net,2007‐09‐0223:16:51
0:cs:Connectedst:Primary/Secondaryds:UpToDate/UpToDateCr‐‐‐
ns:191681144nr:108784dw:191798660dr:1228217841al:962834152bm:9054lo:0pe:0ua:0ap:0
resync:used:0/31hits:34633misses:4367starving:0dirty:0changed:4367
act_log:used:0/127hits:11433119293misses:1051777270starving:11558178dirty:81149096changed:962834152
InstallLinux‐HA
• Compilefromsourceor
• yuminstallheartbeat
ConfigureLinux‐HA
• ha.cf• haresources• authkeys
ha.cf
deadBme120auto_failbackoff
udpport694
ucasteth110.30.0.10
nodenode‐001.zonarsystems.net
nodenode‐002.zonarsystems.net
haresources
cluster‐001.zonarsystems.netIPaddr::10.30.0.101/24drbddisk::postgres\
Filesystem::/dev/drbd0::/mnt/rod1::ext3\
postgresql\MailTo::[email protected]::'Zonar_Cluster‐001_Switch'
authkeys
auth1
1sha1f93e5e562d430dae92b7fd7b272c13bb0c5de0e2
StartLinux‐HA
• /etc/init.d/heartbeatstart
DEMO
Problems?
• Secondaryserveralwaysstand‐by• Failednodecancauseslowdowns• Kernelmodule
Performance?
• ddif=/dev/zeroof=/mnt/drbd/pgwest/testbs=8192count=2000000
• DRBDdevice– ~85MB/s
• Diskdevice– 121MB/s
30%write performancedegradaBon
Problems?
PerformanceImprovements
Ethernet
• Directcrossovercable• GigE,10GigE• 1500MTUvs9000MTU
RAIDCards
• CapableofWRITEBACK• Writecaching
• Ba_erybackupunit(BBU)
WritebackVS.Writethrough
• Writeback– 80MB/s–94MB/s
• Writethrough– 61MB/s–71MB/s
0102030405060708090
100
Run1
Run3
Run5
Run7
Run9
Run11
Run13
Run15
DRBDProtocols
• ProtocolA– DRBDwaitsforlocaldiskandlocalTCPsendbuffer
• ProtocolB– DRBDwaitsforlocaldiskandremotebuffercache
• ProtocolC– DRBDwaitsforbothlocalandremotedisk
– SAFEST
YMMV
• Remembertotestyourself• Fedora/e1000networkstackissues• DRBDissoYwaretoo,bugscomeup
OtherUsesHA‐Linux+DRBD+???
• Mail• NFS• MySQL
QuesBons?
Links
• Linux‐HA:h_p://www.linux‐ha.org• DRBD:h_p://www.drbd.org
Demo(backupplan)
Host:c6a
Host:c6a
Host:c6a
Host:c6a
Host:c6a
Host:c6b
Host:c6b
Host:c6a