Scaling with event-based webservers

Scaling with event–based webserversMorten Siebuhr

Open Source DaysMarch 5th 2011

1/66

whoami(1)

Defining “web–server work”

Killing Apache

Event–based servers

Using Node.js

Questions

2/66

whoami(1)

3/66

Computer Scientist

Distributed Systems / Scientific Computing

(Web–)developer @ One.com

4/66

Web–development @ one.com:

Rich Internet Applicationse–mail, calendar, galleries, . . .

+ 900.000 customers

5/66

6/66

Defining“Web–server work”

7/66

Typical web–server work

Serving static files

Talking to databases

Not doing lots of computations

≈ 100→ 1000 connections served ASAP

8/66

Typical web–server work

Serving static files

Talking to databases

Not doing lots of computations

≈ 100→ 1000 connections served ASAP

8/66

Typical web–server work

Serving static files

Talking to databases

Not doing lots of computations

≈ 100→ 1000 connections served ASAP

8/66

Typical web–server work

Serving static files

Talking to databases

Not doing lots of computations

≈ 100→ 1000 connections served ASAP

8/66

Typical web–server work

Serving static files

Talking to databases

Not doing lots of computations

≈ 100→ 1000 connections served ASAP

8/66

(A)typical web–server work

Persistent connections

=∞ simultaneous connections

≈ 10000 simultaneous connections

(But we don’t care as much about latency. . . )

9/66

(A)typical web–server work

Persistent connections

=∞ simultaneous connections

≈ 10000 simultaneous connections

(But we don’t care as much about latency. . . )

9/66

(A)typical web–server work

Persistent connections

=∞ simultaneous connections

≈ 10000 simultaneous connections

(But we don’t care as much about latency. . . )

9/66

(A)typical web–server work

Persistent connections

=∞ simultaneous connections

≈ 10000 simultaneous connections

(But we don’t care as much about latency. . . )

9/66

(A)typical web–server work

Persistent connections

=∞ simultaneous connections

≈ 10000 simultaneous connections

(But we don’t care as much about latency. . . )

9/66

waiting...

10/66

sleep(1.0)

11/66

12/66

0 s

5 s

10 s

15 s

20 s

25 s

0 100 200 300 400 500 600 700 800 900 1000

Avera

ge r

esponse tim

e

Concurrent requests

Response time w. sleep(1).

WSGI

13/66

0 s

5 s

10 s

15 s

20 s

25 s

0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000

Avera

ge r

esponse tim

e

Concurrent requests

Response time w. sleep(1).

WSGI

14/66

Threading

15/66

Use threads

16/66

Wasted resources!

17/66

Add threads

18/66

0 s

5 s

10 s

15 s

20 s

25 s

0 100 200 300 400 500 600 700 800 900 1000

Avera

ge r

esponse tim

e

Concurrent requests

Response time w. sleep(1).

WSGIApache

19/66

0 s

5 s

10 s

15 s

20 s

25 s

0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000

Avera

ge r

esponse tim

e

Concurrent requests

Response time w. sleep(1).

WSGIApache

20/66

21/66

22/66

23/66

0 MB

250 MB

500 MB

750 MB

1000 MB

1250 MB

1500 MB

0 100 200 300 400 500 600 700 800 900 1000

Max V

irtu

al M

em

ory

Concurrent requests

Memory usage w. sleep(1).

WSGIApache

25/66

0 MB

250 MB

500 MB

750 MB

1000 MB

1250 MB

1500 MB

0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000

Max V

irtu

al M

em

ory

Concurrent requests

Memory usage w. sleep(1).

WSGIApache

26/66

A single connection

=

1 thread

=

1 C–stack & kernel data structures

+

Locking & switching overhead &c. . .

27/66

A single connection

=

1 thread

=

1 C–stack & kernel data structures

+

Locking & switching overhead &c. . .

27/66

A single connection

=

1 thread

=

1 C–stack & kernel data structures

+

Locking & switching overhead &c. . .

27/66

A single connection

=

1 thread

=

1 C–stack & kernel data structures

+

Locking & switching overhead &c. . .

27/66

28/66

Add memory!

Memory is cheap

Except, actually using it isn’t cheap. . .

29/66

Add memory!

Memory is cheap

Except, actually using it isn’t cheap. . .

29/66

Add memory!

Memory is cheap

Except, actually using it isn’t cheap. . .

29/66

Memory Wall

1986→ 2000:

+ 55% CPU speed P/A

+ 10% RAM speed P/A

(Source: http://www.cs.virginia.edu/papers/Hitting_Memory_Wall-wulf94.pdf)

30/66

Memory Wall

1986→ 2000:

+ 55% CPU speed P/A

+ 10% RAM speed P/A

(Source: http://www.cs.virginia.edu/papers/Hitting_Memory_Wall-wulf94.pdf)

30/66

Memory Wall

1986→ 2000:

+ 55% CPU speed P/A

+ 10% RAM speed P/A

(Source: http://www.cs.virginia.edu/papers/Hitting_Memory_Wall-wulf94.pdf)

30/66

31/66

Threads and connections mapped 1:1

We use a lot of threads

Which we don’t have the memory bandwidth to move around.

32/66

Threads and connections mapped 1:1

We use a lot of threads

Which we don’t have the memory bandwidth to move around.

32/66

Threads and connections mapped 1:1

We use a lot of threads

Which we don’t have the memory bandwidth to move around.

32/66

Events!

33/66

waiting...

34/66

35/66

36/66

Event-basics

Event queue

Event loop

“The Event Loop”

37/66

Event-basics

Event queue

Event loop

“The Event Loop”

37/66

Event-basics

Event queue

Event loop

“The Event Loop”

37/66

Event-basics

Event queue

Event loop

“The Event Loop”

37/66

A single connection

=

1 “Event”

=

Some variables & function

=

Hash table & some pointers

+

Queue of events waiting to be processed

38/66

A single connection

=

1 “Event”

=

Some variables & function

=

Hash table & some pointers

+

Queue of events waiting to be processed

38/66

A single connection

=

1 “Event”

=

Some variables & function

=

Hash table & some pointers

+

Queue of events waiting to be processed

38/66

A single connection

=

1 “Event”

=

Some variables & function

=

Hash table & some pointers

+

Queue of events waiting to be processed

38/66

A single connection

=

1 “Event”

=

Some variables & function

=

Hash table & some pointers

+

Queue of events waiting to be processed

38/66

39/66

39/66

39/66

39/66

Why not earlier

Simply not a problem!

Traditional back–end programmers not used to thinking this way

40/66

Why not earlier

Simply not a problem!

Traditional back–end programmers not used to thinking this way

40/66

Why not earlier

Simply not a problem!

Traditional back–end programmers not used to thinking this way

40/66

What we gain

Relatively low memory use

No threading issues

Client-side programmers have no preconditions.

41/66

What we gain

Relatively low memory use

No threading issues

Client-side programmers have no preconditions.

41/66

What we gain

Relatively low memory use

No threading issues

Client-side programmers have no preconditions.

41/66

What we gain

Relatively low memory use

No threading issues

Client-side programmers have no preconditions.

41/66

Node.js

42/66

Node.js is a set of bindings to the V8 JavaScript enginefor scripting network programs.

— Ryan Dahl

43/66

Node.js is a set of bindings to the V8 JavaScript enginefor scripting network programs.

— Ryan Dahl

43/66

JavaScript

Built for event-based programming.

Netscape needed something fast. . .. . . JavaScript “designed” and implemented in 14 days

44/66

JavaScript

Built for event-based programming.

Netscape needed something fast. . .. . . JavaScript “designed” and implemented in 14 days

44/66

JavaScript

Built for event-based programming.

Netscape needed something fast. . .. . . JavaScript “designed” and implemented in 14 days

44/66

JavaScript

Built for event-based programming.

Netscape needed something fast. . .

. . . JavaScript “designed” and implemented in 14 days

44/66

JavaScript

Built for event-based programming.

Netscape needed something fast. . .. . . JavaScript “designed” and implemented in 14 days

44/66

JavaScript

Built for event-based programming.

Netscape needed something fast. . .. . . JavaScript “designed” and implemented in 14 days

44/66

Google V8

JIT’ing JavaScript to native machine code

= Compiling server code

45/66

Google V8

JIT’ing JavaScript to native machine code

= Compiling server code

45/66

Node.js

Non–browser V8–bindings for various libraries

Everything non–blocking

46/66

A minimal webserver in Node.js. . .

// Import HTTP package

var http = require(’http’);

// Set up basic server

var server = http.createServer(function (req , res) {

res.writeHead (200, {’Content -Type’: ’text/plain ’});

res.end(’Hello World\n’);

});

// Start the server

server.listen (8124, "127.0.0.1");

console.log(’Server running at http ://127.0.0.1:8124/ ’);

47/66

. . . with “database”

// Import HTTP package

var http = require(’http’);

// Set up basic server

var server = http.createServer(function (req , res) {

setTimeout(function () {

res.writeHead (200, {’Content -Type’: ’text/plain ’});

res.end(’Hello World\n’);

}, 1000);

});

// Start the server

server.listen (8124, "127.0.0.1");

console.log(’Server running at http ://127.0.0.1:8124/ ’);

48/66

0 s

5 s

10 s

15 s

20 s

25 s

0 100 200 300 400 500 600 700 800 900 1000

Avera

ge r

esponse tim

e

Concurrent requests

Response time w. sleep(1).

WSGIApacheNode.js

49/66

0 s

5 s

10 s

15 s

20 s

25 s

0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000

Avera

ge r

esponse tim

e

Concurrent requests

Response time w. sleep(1).

WSGIApacheNode.js

50/66

0 MB

250 MB

500 MB

750 MB

1000 MB

1250 MB

1500 MB

0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000

Max V

irtu

al M

em

ory

Concurrent requests

Memory usage w. sleep(1).

WSGIApacheNode.js

51/66

Connections in Node.js

=

1 “Event”

=

1 Closure + callback function

=

1 struct + some pointers

+

Event queue operations + function calls

52/66

Connections in Node.js

=

1 “Event”

=

1 Closure + callback function

=

1 struct + some pointers

+

Event queue operations + function calls

52/66

Connections in Node.js

=

1 “Event”

=

1 Closure + callback function

=

1 struct + some pointers

+

Event queue operations + function calls

52/66

Connections in Node.js

=

1 “Event”

=

1 Closure + callback function

=

1 struct + some pointers

+

Event queue operations + function calls

52/66

Connections in Node.js

=

1 “Event”

=

1 Closure + callback function

=

1 struct + some pointers

+

Event queue operations + function calls

52/66

Alternatives

Well–written thread program!

Erlanghttp://www.erlang.org/

Google GOhttp://golang.org/

Lua Event Machinehttps://github.com/esmil/lem

Nginxhttp://wiki.nginx.org/

Varnishhttp://www.varnish-cache.org/

53/66

Alternatives

Well–written thread program!

Erlanghttp://www.erlang.org/

Google GOhttp://golang.org/

Lua Event Machinehttps://github.com/esmil/lem

Nginxhttp://wiki.nginx.org/

Varnishhttp://www.varnish-cache.org/

53/66

Alternatives

Well–written thread program!

Erlanghttp://www.erlang.org/

Google GOhttp://golang.org/

Lua Event Machinehttps://github.com/esmil/lem

Nginxhttp://wiki.nginx.org/

Varnishhttp://www.varnish-cache.org/

53/66

Alternatives

Well–written thread program!

Erlanghttp://www.erlang.org/

Google GOhttp://golang.org/

Lua Event Machinehttps://github.com/esmil/lem

Nginxhttp://wiki.nginx.org/

Varnishhttp://www.varnish-cache.org/

53/66

Alternatives

Well–written thread program!

Erlanghttp://www.erlang.org/

Google GOhttp://golang.org/

Lua Event Machinehttps://github.com/esmil/lem

Nginxhttp://wiki.nginx.org/

Varnishhttp://www.varnish-cache.org/

53/66

Alternatives

Well–written thread program!

Erlanghttp://www.erlang.org/

Google GOhttp://golang.org/

Lua Event Machinehttps://github.com/esmil/lem

Nginxhttp://wiki.nginx.org/

Varnishhttp://www.varnish-cache.org/

53/66

Alternatives

Well–written thread program!

Erlanghttp://www.erlang.org/

Google GOhttp://golang.org/

Lua Event Machinehttps://github.com/esmil/lem

Nginxhttp://wiki.nginx.org/

Varnishhttp://www.varnish-cache.org/

53/66

Working with Node.js

54/66

I’m new to JavaScript!

55/66

56/66

JavaScript WTFs

Always add ;!

Function arguments. . .

this scoped to callee object!

Stacktraces + events sometimes look strange

57/66

JSLint

www.crockfordfacts.com

58/66

JSLint

www.crockfordfacts.com

58/66

Nice code!

def process(user):

user = getAuth(user)

if not user:

return ’Fail’

db = getConnnection(params)

data = db.getUserData(user)

if not data:

return ’Fail’

return data

59/66

Boomerang code

function process(user , callback) {

getAuth(req.username , function (err , user) {

if (err) return callback(err);

getDBConnection(params , function(err , db) {

if (err) return callback(err);

db.getUserData(user , function(err , data) {

if (err) return callback(err);

return callback(null , data);

});

});

});

}

60/66

NPM

61/66

Express.js

var app = require(’express ’).createServer ();

app.get(’/’, function(req , res) {

setTimeout(function () {

res.send(’Hello World ’);

}, 1000);

});

app.listen (3000);

62/66

Express.js - Chat server

var app = require(’express ’).createServer (),

chat = [];

app.get(’/’, function(req , res) {

chat.push(res);

});

app.get(’/:msg’, function(req , res) {

chat.forEach(function (conn) {

conn.write(req.params.msg);

});

});

app.listen (3000);

63/66

Questions

(The end)

64/66

http://nodejs.org/

https://github.com/joyent/node

65/66

function process(user , callback) {

var user = null , db = null;

function done_ () {

if (user && db) {

db.getUserData(user , function(err , data) {

if (err) return callback(err);

return callback(null , data);

});

}

}

getAuth(req.username , function (err , retuser) {

if (err) return callback(err);

user = retuser;

done_();

});

getDBConnection(params , function(err , retdb) {

if (err) return callback(err);

db = retdb;

done_();

});

}

66/66