Copyright © 1998 Alex Chaffee Building a Robust Multithreaded Server in Java Alexander Day Chaffee jGuru Training by the MageLang Institute [email protected]

Copyright © 1998 Alex Chaffee

Building a Robust Multithreaded Server in Java

Alexander Day ChaffeejGuru Training by the MageLang [email protected]


Abstract

• Mark Andreesen says, ”Client-side Java is dead." Fortunately, server-side Java is alive and kicking! This session will quickly introduce you to the concepts of threading and networking in Java. We then build a robust, multithreaded server (using a simple chat as our application). We examine several different threading models, specifically comparing the one-thread-per-connection model with the queuing model.


Introduction

• jGuru Training by the Magelang Institute– http://www.jguru.com– Java Training and Consulting

• Alex Chaffee– Creator of Gamelan– Cool Java Dude


Background

• Java language basics• Java networking• Java threading


Overview

• Start slow, end up really really fast• Part I: Networking and I/O in Java• Part II: Threads and concurrency• Part III: Multithreaded server design• Part IV: Implementations

– Design decisions– Object models– Problems and solutions


Part I: Networking and I/O in Java


Java and Networking

• Built into language• One of the 11 buzzwords• Network classloader• Java.Net API• Based on TCP/IP, the internet protocol


Client-server

• Difference between client and server is semantic

• It’s all just peers talking to each other• Protocol - roles, vocabulary, rules for

communication


TCP/IP: The Internet Protocol

Physical Network

Transport Layer (TCP, UDP)

Internet Layer (IP)

Application Layer (HTTP, FTP, SMTP)


Sockets and Ports

• Port: a meeting place on a host– One service per port– 1-1023 = well-known services– 1024+ = experimental services, temporary

• Socket: a two-way connection


Client

Sockets and Ports (Diagram)

port 13

port 80

Time Service

Web Service

Socket

Server

Socket


The Socket Class

• Socket(String host, int port)• InputStream getInputStream()• OutputStream getOutputStream()• void close()


Socket Code

Socket s = new Socket(“www.sigs.com”, 80);

• Simple, huh?• Creates the connection• We still need to get the data there and

back again


Streams

• A stream is a sequence of bytes• I/O from disk, network, memory, etc. is

handled in exactly the same way


InputStream and OutputStream

• InputStream:abstract int read()

• OutputStream:abstract void write(int b)

• Common:abstract void close()


Filter Streams

• Like photographic filters• Change the input (or output) on the

way through• E.g.: BufferedInputStream,

PrintStream, LineNumberInputStream


Filter Streams (Diagram)

Abc

FileInputStream AllCapsInputStreamAbc

ABC

NoVowelInputStreamBC

DataInputStream

readLine()

“BC” DataInputStream in = new DataInputStream(new NoVowelInputStream(

new AllCapsInputStream(new FileInputStream(“input.txt”))));

String line = in.readLine();


PrintStream

• print()– outputs an object, primitive, or String

• println()– same, but adds a newline

• System.out is a PrintStream

PrintStream ps =

new PrintStream(outputstream)


DataInputStream

• int readInt()• type readType()• String readLine()

DataInputStream in =

new DataInputStream(inputstream)


Browser.java (source)

import java.net.*;

import java.io.*;

public class Browser {

public static void main(String[] args)

{

String host = "www.stinky.com";

int port = 80;

String file = "/index.html";



Socket s = new Socket(host, port);

InputStream strIn = s.getInputStream();

OutputStream strOut = s.getOutputStream();

PrintStream out =

new PrintStream(strOut);

DataInputStream in =

new DataInputStream(strIn);



out.println("GET " + file + " HTTP/1.0");

out.println();

String line;

while ((line = in.readLine()) != null) {

System.out.println(line);

}

in.close();

out.close();


Java Servers

• A server listens on a port and accepts connections

• Must be applications (or signed applets)

• Only one service per port for the entire host machine– Java throws an exception if you try to open

more than one


ServerSocket

ServerSocket ss = new ServerSocket(1234);

Socket socket = ss.accept();

• accept() returns only after a client makes the connection


ServerSocket

• Now we have a socket, so we can call...

InputStream in = socket.getInputStream();

OutputStream out = socket.getOutputStream();


EchoServer.java

ServerSocket ss =

new ServerSocket(1234);

Socket s = ss.accept();

InputStream in = s.getInputStream();

OutputStream out = s.getOutputStream();


EchoServer.java

int ch;

while ((ch = in.read()) != -1) {

out.write(ch);

}

out.close();

in.close();


EchoServer Demo

• Connect• Type some stuff• Disconnect• Problem: Multiple simultaneous

connections– Connections are made, but server doesn’t

respond– Even if we loop, can still handle only one at a

time


Multiple Connections

• Solution:– Java threads!


Part II: Threads


Java Threads

• A thread is not an object• A thread is a flow of control• A thread is a series of executed

statements• A thread is a nested sequence of

method calls


The Thread Object

• A thread is not an object• A Thread is an object

void start()– Creates a new thread and makes it runnable

void run()– The new thread begins its life inside this

method


Thread Creation Diagram

Thread t = new BThread();

t.start();

doMoreStuff();

BThread() {}

void start() {// create thread

}

void run() {doSomething();

}

Object A Object BThread (extends Thread)



t.start();

doMoreStuff();

Thread Creation DiagramObject A

BThread() {}


}


}

Object BThread (extends Thread)



t.start();

doMoreStuff();

BThread() {}


}


}

Thread Creation DiagramObject A Object BThread (extends

Thread)



t.start();

doMoreStuff();

BThread() {}


}


}


Thread)



t.start();

doMoreStuff();

BThread() {}


}


}


Thread)



t.start();

doMoreStuff();

BThread() {}


}


}


Thread)



t.start();

doMoreStuff();

BThread() {}


}


}


Thread)



t.start();

doMoreStuff();

BThread() {}


}


}


Thread)



t.start();

doMoreStuff();

BThread() {}


}


}


Thread)


Runnable Interface

• A helper to the thread object• The Thread object’s run() method calls

the Runnable object’s run() method• Allows threads to run inside any

object, regardless of inheritance


Runnable Example

Talker talker = new Talker();

Thread t = new Thread(talker);

t.start();

---

class Talker implements Runnable {

public void run() {

while (true) {

System.out.println(“yakitty yak”);

}

}

}


Blocking Threads

• When reading from a stream, if input is not available, the thread will block

• Thread is suspended (“blocked”) until I/O is available

• Allows other threads to automatically activate• When I/O available, thread wakes back up

again– Becomes “runnable”– Not to be confused with the Runnable interface


Thread Scheduling

• In general, the runnable thread with the highest priority is active (running)

• Java is priority-preemptive– If a high-priority thread wakes up, and a low-

priority thread is running– Then the high-priority thread gets to run

immediately

• Allows on-demand processing– Efficient use of CPU


Thread Starvation

• If a high priority thread never blocks• Then all other threads will starve• Must be clever about thread priority


Thread Priorities: General Strategies

• Threads that have more to do should get lower priority

• Counterintuitive• Cut to head of line for short tasks• Give your I/O-bound threads high priority

– Wake up, immediately process data, go back to waiting for I/O


Race Conditions

• Two threads are simultaneously modifying a single object

• Both threads “race” to store their value

• In the end, the last one there “wins the race”

• (Actually, both lose)


Race Condition Example

class Account {

int balance;

public void deposit(int val)

{

int newBal;

newBal = balance + val;

balance = newBal;

}

}Looks good, right?


Race Condition Example

class Account {

int balance;

public void deposit(int val)

{

int newBal;

newBal = balance + val;

balance = newBal;

}

}What if we are swapped out right here?


Thread Synchronization

• Language keyword: synchronized

• Takes out a monitor lock on an object– Exclusive lock for that thread

• If lock is currently unavailable, thread will block


Thread Synchronization

• Protects access to code, not to data– Make data members private

– Synchronize accessor methods

• Puts a “force field” around the locked object so no other threads can enter

• Actually, it only blocks access to other synchronizing threads

• A rogue thread can "sneak in" and modify a variable it has access to


Synchronization Example

class Account { private int balance; synchronized public void deposit(int val) { int newBal; newBal = balance + val; balance = newBal; } synchronized public void withdraw(int val) { int newBal; newBal = balance - val; balance = newBal; }}


Thread Deadlock

• If two threads are competing for more than one lock

• Example: bank transfer between two accounts– Thread A has a lock on account 1 and wants

to lock account 2

– Thread B has a lock on account 2 and wants to lock account 1


Avoiding Deadlock

• No universal solution

• Ordered lock acquisition

• Encapsulation (“forcing directionality”)

• Spawn new threads

• Check and back off

• Timeout

• Minimize or remove synchronization


Wait and Notify

• Allows two threads to cooperate

• Based on a single shared lock object


Wait and Notify: Code

• Consumer:synchronized (lock) {

while (!resourceAvailable()) {

lock.wait();

}

consumeResource();

}


Wait and Notify: Code

• Producer:produceResource();

synchronized (lock) {

lock.notifyAll();

}


Wait/Notify Sequence

Lock Object

ConsumerThread

ProducerThread

1. synchronized(lock){

2. lock.wait();

3. produceResource()4. synchronized(lock) {5. lock.notify();6.}

7. Reacquire lock8. Return from wait()

9. consumeResource();10. }



Lock Object

ConsumerThread

ProducerThread


2. lock.wait();






Lock Object

ConsumerThread

ProducerThread


2. lock.wait();






Lock Object

ConsumerThread

ProducerThread


2. lock.wait();






Lock Object

ConsumerThread

ProducerThread


2. lock.wait();






Lock Object

ConsumerThread

ProducerThread


2. lock.wait();






Lock Object

ConsumerThread

ProducerThread


2. lock.wait();






Lock Object

ConsumerThread

ProducerThread


2. lock.wait();






Lock Object

ConsumerThread

ProducerThread


2. lock.wait();






Lock Object

ConsumerThread

ProducerThread


2. lock.wait();






Lock Object

ConsumerThread

ProducerThread


2. lock.wait();





Wait/Notify: Details

• Often the lock object is the resource itself

• Sometimes the lock object is the producer thread itself


Wait/Notify: Details

• Must loop on wait(), in case another thread grabs the resource...– After you are notified– Before you acquire the lock and return from

wait()

• Use lock.notifyAll() if there may be more than one waiting thread


Wait/Notify Example: Blocking Queue

class BlockingQueue extends Queue {

public synchronized Object remove() {

while (isEmpty()) {

wait(); // really this.wait()

}

return super.remove();

}

public synchronized void add(Object o) {

super.add(o);

notifyAll(); // this.notifyAll()

}

}


Part III: Multithreaded Server Design


Different Models of Information Flow

• Pull model• Push model: One thread per client• Buffer model: One thread per task• Single-thread model


Pull-based Flow

• Consumer asks producer for new information

• Demand-driven problems– Examples

• Not appropriate for chat/message-passing server


Push-based Flow

• Producer triggers sequence of events• One thread per client• Receives message, delivers it, waits


Push-based: Pro

• Simple• Efficient use of CPU

– Events occur only when message available - no polling

– No time wasted passing information from one stage to the next


Push-based: Con

• Much thread swapping• Slow receivers can stall upcoming

messages from sender


Buffer Model

• Independent threads• Communicate via buffers• Producer puts message in buffer,

consumer takes message out


Buffer Model: Pro

• Can separate tasks into separate threads

• Optimize and prioritize per task– e.g. Processing existing messages is more

important than accepting new connections

• Small number of threads means fewer chances for deadlock– Easier to model the system and avoid any

chance of deadlock


Buffer Model: Con

• More complex code

• Must deal with deadlock and starvation


The One-Thread Model

• All work done by a single thread

• Finishes processing one request before going to the next


The One-Thread Model (Cont.)

• Pro:– Suitable for low-traffic scenarios

– Quick to implement

– Avoids any possibility of deadlock or race conditions

• Con– Unsuitable for high-traffic server

– Hard to implement things like timers or callbacks


Part IV: Implementations


Implementation 1: Push Model Chat Server

• Listens on a port• Accepts connections• Spawns threads, one per connection• Receives a message from one client• Dispatches it to all clients


Chat Server Objects

• ChatServer– Main

• ClientHandler– One per connection– Threaded

• Dispatcher– Vector


Chat Server Objects (Diagram)

ChatServer

ClientHandlerClientHandler

Dispatcher

Socket Socket


ClientHandler

• Stores information for a single clientSocket incoming;

int id;

Dispatcher dispatcher;


ClientHandler.run()

• generate filter streams• read a line• send it to the dispatcher• loop


ClientHandler.send()

• sends a single message to the client• called by the dispatcher• uses the socket


Dispatcher

• Keeps a list of all client handlers• Uses java.util.Vector

– synchronized list structure


Dispatcher.dispatch()

• Sends a message to all clients• Accepts a String and an identifying

integer• Enumerates down all client handlers

and calls each one’s send() method


ChatServer

• One method: main()• Creates a new ServerSocket• Creates a new Dispatcher• Listens for connections• When connection received, creates a

new ClientHandler and starts a new thread running inside it

• Adds it to the Dispatcher


Push model: Pro and Con

• Pro– Straightforward object, thread models– Efficient CPU usage

• Threads unblock only when message available

• Con– Must send message to all clients before

receive new one– No upper limit on number of threads

• May thrash or starve


Implementation 2: Buffer Model Message Server


Queuing Model

• Producer and consumer communicate via a buffer

• Buffer is a queue• First in, first out


Blocking Queue

• If a thread tries to remove an item from an empty queue

• Then it blocks until an item is placed inside

• Uses wait/notify technique


JDK 1.2 Collections

• Set• List• Map• Queue

– Built on top of LinkedList


Basic Object Model

• Acceptor (thread)• Clients (set)• Receiver (thread)• Incoming (queue)• Processor (thread)


Acceptor

Receiver

Clients

Incoming

Processor

Thread

Data

write

read

read

write

create

Basic Object Model (Fig.)


Acceptor Thread

• Listens for incoming connections• Creates a client object• Adds it to clients set


Clients Set

• Set of all active clients• Synchronized methods

– Only one thread at a time can access


Receiver Thread

• Walks down clients set• For each client, checks if message is

(fully) available• Reads message and places it on

incoming queue• Uses non-blocking input streams


Receiver Thread: Alternative

• Alternative: one receiver thread per client

• Each receiver thread blocks until input available

• Con:– Reach number of threads bottleneck– Scheduling not determined - could starve or

do in weird order


Incoming Queue

• Queue of messages• Synchronized methods

– Only one thread at a time can access

• Blocking queue– If a thread tries to get a message from an

empty queue, it blocks until another thread adds a message


Processor Thread

• While incoming queue is not empty• Pops message off incoming queue• Delivers it


Full Object Model

• Checker (thread)• Problems (list)• Auditor (thread)


Acceptor

Receiver

Clients

Incoming

Processor

CheckerProblems

Auditor

Thread

Data

Full Object Model (Fig.)


Problems List

• List of clients who have problems• Receiver can add a client to problems

list if it gets a socket read error• Processor can add a client to problems

list if it gets a socket write error• Checker thread periodically

disconnects them all


Checker Thread

• Wakes up every N seconds• Walks list of clients• Move to problems list

– If N seconds has passed (“timeout”)– If socket has a problem

• Walk problems list and actively close each client


Auditor Thread

• Every N seconds, wakes up• Collects and prints statistics


Non-Blocking Input Stream

• Needed for receiver to do its job• If a full message is not available, it

does NOT block• Instead, it returns null• Tricky to implement

– Must use available(), mark() and reset() cleverly


Thread Safety Policies

• All data objects synchronized• All data objects independent• Therefore no possibility of deadlock in

the data objects themselves


Thread Safety Policies

• Thread objects must be very careful to avoid deadlock

• Current model: no multi-object transactions• Example

– Checker does clients.remove(c) then problems.add(c)

– NOT synchronized(clients) { synchronized(problems) { clients.remove(c); problems.add(c);}}

– It's OK if it's interrupted between remove and add


Priority: Why?

• In low-volume server, all threads are normally blocked

• When something needs to happen, it happens

• That's the beauty of threads!• So priorities are irrelevant

– If you don’t know what you’re doing, don’t mess with priorities


Priority: Why?

• In high-volume server, many threads always have something to do

• Important to do things in the right order

• Deal with starvation issue


Priorities: General Strategies

• Threads that have more to do get lower priority

• Counterintuitive• Cut to head of line for short tasks


Priorities: Queuing Strategies

• Most important: keep incoming queue empty

• So we don't get bogged down• Minimize latency for received

messages


Priorities: Decisions

• Don't want to add new messages if we're already busy– Therefore processor > receiver

• Don't want to accept new connections if we're already busy– Therefore processor > acceptor

• Implication: – Receiver and acceptor may starve?– No: queue will eventually get cleared



• Want to be able to accept new connections, even if a currently connected client has something to say– Therefore acceptor > receiver



• Checker's job is brief but important• Needs to wake up, check all clients,

and go back to sleep• Therefore checker > processor



• Auditor's job is brief but important• Need to get accurate statistics --

timing is crucial• Can't wait around for other threads to

finish• Therefore auditor > checker


Priorities: Conclusion

• Auditor - 10• Checker - 8• Processor - 6• Acceptor - 4• Receiver - 2• Goes by twos to accommodate

Windows threading model


Processor Thread Behavior

• Processor thread does:1. Pop message off incoming queue2. Deliver it3. Loop

• Very independent


Processor Thread Blocking

• Step 1 (pop message) may block• If queue is empty• This is good• Allows other threads to wake up and

fill queue


Processor Thread Blocking

• Step 2 (delivery) may block– If network is busy– If OS socket buffer is full

• Bad, because queue is still full• Increases latency (delivery time) for

already-queued messages – While new messages are arriving, old

messages should be delivered instead


Multiple Processor Threads

• Solution: have many processor threads working simultaneously

• If one blocks, another picks up the next message


Multiple Processors (Fig.)

Acceptor

Receiver

Clients

Incoming

Processor

Thread

Data

write

read

read

write

create

ProcessorProcessorProcessor


How Many Threads?

1. Constant, tuned per application2. Dynamic a/k/a thread pooling


Thread Pooling

• Thread pool keeps a certain number of threads alive

• Other threads ask the thread pool to perform a task

• If an existing thread is available, that thread performs the task

• Else a new thread is created, or the task blocks until one is available – (depending on policy)


Thread Pooling (Cont.)

• Removes overhead of creating threads• Allows modular task strategy


Information Flow

• Decoupling threads with buffers is a good idea– Maximizes concurrency– Smooths out bursty differences in rate

• Can cause problems with information flow– Producers outpace Consumers– Too many threads


Limiting Flow

• Bounded Buffers– Stalls producers if they outpace consumers

• Bounded Thread Pools– Prevents thrashing

• Rate-adjustment / back-pressure– Consumer asks Producer to slow down


Another Problem

• One message, multiple recipients, one processor thread

• If one recipient is in Finland, it will stall the remaining recipients for that message


Another Problem: Solutions

• Solution 1: one message queue per client; thread pool applied to these delivery queues

• Solution 2: split multi-message into several single-recipient messages– Called a Splitter (or Fork or Multicaster)– Also possible: Routers, Mergers, Collectors,

Combiners, Conduits -- see Lea’s Concurrent Programming in Java


Debugging Techniques• Make sure to trap all Throwables in

your Thread object’s run() method– Otherwise a stray OutOfMemoryError or

NullPointerException will kill your thread and it will look like deadlock

• Good logging procedure is vital– I use a global log method that outputs the

name of the thread and the time in addition to the message

– Name your threads at creation time


Thread Dump

• Control-backslash in Solaris• Control-break in Windows

– Unfortunately, it scrolls– There’s no way to redirect standard error– Oops


Thread Watcher

• Data object• All tasks (threads) have a pointer to the

master ThreadWatcher• Put debug code in your tasks to inform the

ThreadWatcher when they change state• watcher.setIdle();• Message m = incoming.remove();• watcher.set(“Processing message “ + m);


Thread Watcher (cont.)

• ThreadWatcher has its own thread– Periodically prints status of threads– Prints an error message if one thread is

blocked for too long

• Alternative to fancy GUI thread debugger– Symantec Café– Compuware DevPartner– et al.


Bots

• PingBot– periodically sends a message to itself,

measures the latency

• QuoteBot– sends random Zippy quotes to mimic a chat

room


Measurements

• Number of bots before sharp latency increase

• Average latency of single PingBot for N ZippyBots


Test Architecture

• Need at least three machines– Server– Zippy Bot Host– Ping Bot Host

• So they don’t interfere with one another• Should also have some clients on a far

away machine– If your server will be running on the Internet


Conclusion


Thanks to

• Eric Malmstrom• Carl Muckenhoupt• Gerry Seidman• Greg Travis• For helping prepare this talk


Where to Get More Information: Network Programming

• Cornell & Horstmann, Core Java (Sunsoft press)

• Harold, Java Network Programming (O’Reilly)

• Hughes et al., Java Network Programming, (Manning, an imprint of Prentice-Hall)


Where to Get More Information: Multithreading

– Cornell & Horstmann, Core Java (sunsoft press)

– Oaks and Wong, Java Threads (o’reilly)

– Lea, Concurrent Programming in Java (Addison Wesley)

– Travis, Using thread pools to increase threading efficiency, developer.com, http://www.Developer.Com/journal/techworkshop/060498_thread.html

– Travis, Working with the blocking queue, http://www.Developer.Com/journal/techworkshop/091098_blockq.html


Where to Get More Information:

• Web sites– http://www.jGuru.com/ (Java Training)

– http://www.Developer.com/ (Gamelan)– http://www.Javaworld.com/ (magazine)– http://www.Purpletech.com/ (author’s site)

Documents

Copyright © 1998 Alex Chaffee Building a Robust Multithreaded Server in Java Alexander Day Chaffee jGuru Training by the MageLang Institute [email protected]