Lahav Savir• 15 years in on-line industry• Architect and CEO @ Emind Systems (est. 2006)
• AWS solution provider• Over 30 AWS customers
Hobbies (that’s the . . .)• MTB cycling• Mountain hiking
Backup scenarios
On Premises to off-site• File servers• Backup files• Data base dumps
archiving• Disaster recovery
On the cloud to other site• File servers• Large data volumes• Data base dumps• Large S3 beckets
Storage scenarios
Storage appliances• NFS• CIFS
Disks & Servers• Windows shares• Linux exports• Linux servers• Sun exports
RequirementsBackup• Keep a replica of the data off-site• Keep history of the data for X month back• Secure transfer• Encrypt data sets• Large files• Delta transfer
Deployment• Don’t impact existing setup• Don’t install any SW on servers• No additional hardware
Few more . . .
• Control bandwidth throughput• Visibility and monitoring• Simplicity• Don’t pay much– License– Traffic– Storage
Alternatives
• Windows– Virtual drive to s3– Sync application– Cygwin / delta copy
• Linux– s3fs (fuse)– s3cmd
• Storage built-in– No monitoring– No visibility to status– No feedback
Simple solution
• Sync Manager– Linux appliance– cifs-utils– rsync– s3cmd– tc (traffic controller)– net-snmp– curl
Sync Configuration
• rsync (filer to filer)rsync;/filer/data1/; [email protected]:/data1/{A}rsync;/filer/data2/; sync@porticor_vpd:/data2
• s3 (filer to s3 with / without VPD)s3;/var/www/wordpress/;s3://bucket1/wordpress-{d}/;--no-delete-removeds3;/mnt/srv1/;s3://bucket2/
Bandwidth control• Tag user trafficiptables -t mangle -A OUTPUT -m owner --uid-owner $SYNCMGR_UID -j MARK --set-mark 0x1
• Create root qdisc for eth0$TC qdisc add dev $IF root handle 1: htb default 30
• Add a class (bucket) with bandwidth restrictions$TC class add dev $IF parent 1: classid 1:2 htb rate $MAXRATE
• Then add a filter to force packets through the class$TC filter add dev $IF protocol ip parent 1:0 prio 1 handle 1 fw classid 1:2
Tip: use iftop to see it in action
Monitoring## SNMP paramsSNMPTRAP=trueSNMPTRAP_HOST=nms_serverSNMPTRAP_PORT=162SNMPTRAP_COMMUNITY=publicSNMPTRAP_OID=.1.3.6.1.4.1.39731.2101
## support_routerSUPPRTR_NOTIF=trueSUPPRTR_PROJECT="SupportDispatcher“SUPPRTR_SYNCMGR_CLIENT=EmindSUPPRTR_BASEURL=https://support.emind.co/support_router/public/api.php
## snmpd.confrocommunity public# send all Emind Enterprise ID requests to the subagentpass .1.3.6.1.4.1.39731 /usr/local/emind/snmp_subagent
Cloud backup hosts
• ec2 instance (Linux server)– EBS volumes
• s3 buckets• Porticor VPB– EBS volumes– S3 proxy
Hosting on the cloud
• Public cloud– Instance behind security groups with SSH keys
• VPC– Instance behind VPN• AWS VPN Gateway• IPSec with CheckPoint in the VPC• IPSec with Swan in the VPC• SSL VPN with OpenVPN in the VPC
Restoring
Don’t be shocked
• rsync back from storagersync ; [email protected]:/data1/{A} ; /filer/data1/
• 3scmds3cmd get s3://bucket2/file /path/to/restore/file
Summary
• Simple & open solution• No impact to customer infrastructure• No additional HW• Control & visible• Fully integrated to NMS• Reliable• Secure
AWS Tips
• Don’t forget to set AWS console MFA• Setup a VPN to your AWS server• No public SSH• Monitor traffic coming into your servers• Multi region / AZ for high availability• Use ec2 tools• Backup backup backup . . .