Embed Size (px)
Citation preview
Inside Sqale’s Backend
Sapporo Ruby Kaigi 2012Gosuke Miyashitapaperboy&co. Inc.
A little bit about me
Technical Managerat
paperboy&co.
https://github.com/mizzyhttp://mizzy.org/
@gosukenator
Inside Sqale’s Backend
http://www.facebook.com/sqalejp
WARNINGThere are little topics
about Ruby in this talk
What is Sqale?
Cloud Application Platform like Heroku
Architecture Overview
AWS
SSH Router
Containers
Web Proxyto Containers
Deploy Servers
File Repositories
SFTPGit over SSHSSH
HTTP/HTTPS
Containers
AWS
SSH Router
Containers
Web Proxyto Containers
Deploy Servers
File Repositories
SFTPGit over SSHSSH
HTTP/HTTPS
Virtual Environments Assigned To Users
Similar to Dynos of Heroku
Containers made by LXC (Linux Containers)
EC2 Instance (1 Virtual Machine)
Container for
user B
Container for
user A
Container for
user A
Container for
user B
Container for
user B
Container for
user D
Container for
user D
Container for
user C
Container for
user E
Container for
user E
Container for
user F
Container for
user F
Container for
user E
Container for
user F
Container for
user F
NginxUnicorn
sshdsupervisrod
on each container
Amazon Linux+
Patched kernel(3.2.16)
grsecurity kernel patchfor various restrictions
original kernel patchesto restrict tcp port
bind and fork bomb
Anti fork bomb patch makes some changes to cgroup and fork process
Seepaperboy-sqale/sqale-patches
on GitHub
Web Proxy
AWS
SSH Router
Containers
Web Proxyto Containers
Deploy Servers
File Repositories
SFTPGit over SSHSSH
HTTP/HTTPS
nginx
Container for
user A
Container for
user B
Container for
user B
Container for
user C
Container for
user C
Container for
user C
ELB
nginx
HTTP/HTTPS
nginxlua-nginx-module
redis2-nginx-module
Container for
i4pc-mizzy
Container for
i4pc-mizzy
Container for
lokka-mizzy
Container for
lokka-mizzy
nginx
http://lokka-mizzy.sqale.jp/
Redis
nginx port 8081 nginx port 8082 nginx port 8083 nginx port 8084
Which containers?
host001:8083, host001:8084
host001
or
location / {set $container "";set $next_containers "";
error_page 502 = @failover;
rewrite_by_lua_file dynamic-proxy.lua;proxy_pass http://$container;
}
nginx.conf (excerpt)
local reply = ngx.location.capture("/redis")if reply.status ~= ngx.HTTP_OK thenngx.exit(503)
end
local containers, type = parser.parse_reply(reply.body)
dynamic-proxy.lua (excerpt)
while #containers > 0 dotmp = table.remove(
containers,math.random(#containers))
if ngx.shared.downed_containers:get(tmp) thenngx.log(ngx.DEBUG, tmp .. " is down")
elsecontainer = tmpbreak
endend
dynamic-proxy.lua (excerpt)
ngx.var.container = containerngx.var.next_containers= luabins.save(containers)
dynamic-proxy.lua (excerpt)
location / {set $container "";set $next_containers "";
error_page 502 = @failover;
rewrite_by_lua_file dynamic-proxy.lua;proxy_pass http://$container;
}
nginx.conf (again)
location @failover {error_page 502 = @failover;
rewrite_by_lua_file failover.lua;proxy_pass http://$container;
}
nginx.conf (excerpt)
failover.lua (excerpt)
local downed_container = ngx.var.containerif downed_container thenngx.shared.downed_containers:set(downed_container,1,sqale.NEGATIVE_CACHE_SECONDS
)end
failover.lua (excerpt)
while #containers > 0 dotmp = table.remove(
containers,math.random(#containers))
if ngx.shared.downed_containers:get(tmp) thenngx.log(ngx.DEBUG, tmp .. " is down")
elsecontainer = tmpbreak
endend
if not container thenngx.exit(503)
end
ngx.var.container = containerngx.var.next_containers= luabins.save(containers)
failover.lua (excerpt)
location @failover {error_page 502 = @failover;
rewrite_by_lua_file failover.lua;proxy_pass http://$container;
}
nginx.conf (agin)
Seehttp://bit.ly/UHbHIb
by @hiboma
SSH Router
AWS
SSH Router
Containers
Web Proxyto Containers
Deploy Servers
File Repositories
SFTPGit over SSHSSH
HTTP/HTTPS
SSH Router
File Repositories(Git Server)
Git SSH Login
File Repositories(File Server)
Containers
SFTP
How implement this routing?
OpenSSH with script authentication patch
Seemizzy/openssh-script-auth
on GitHub
Change routes by SSH_ORIGNAL_COMMAND
In case of SSH_ORIGINAL_COMMAND
is “git-*”
SSH Router
File Repository(Git Server)
git push(ssh [email protected] git-recieve-pack‘/mizzy/lokka.git’)
MySQL
Run AuthorizedKeysScript
Verify the public keyand get the user’s gitserver
command=“ssh [email protected]‘/var/repos/mizzy/lokka.git’”
In case of SSH_ORIGINAL_COMMAND
is “sftp-server”
SSH Router
File Repository
(File Server)
sftp [email protected](ssh [email protected] sftp-server)
MySQL
File Repository(Git Server)
git push
Run AuthorizedKeysScript
Verify the public keyand get the user’s file server
command=“ssh [email protected]”
In case of SSH_ORIGINAL_COMMAND
is empty
SSH Router
Container
MySQL
Run AuthorizedKeysScript
Verify the public keyand get the user’s cotainers list
command=“ssh sqale@ users001.sqale.lan -p 8081”
Display the user’s containers list and wait the user’s selection
Deploy Servers
AWS
SSH Router
Containers
Web Proxyto Containers
Deploy Servers
File Repositories
SFTPGit over SSHSSH
HTTP/HTTPS
Please ask to@kyanny
Other
About Sqale’s Server Build Automation
http://bit.ly/NBbj9Fby @lamanotrama
Thanks
