Copyright © 1998 Alex Chaffee Building a Robust Multithreaded Server in Java Alexander Day Chaffee...

Building a Robust Multithreaded Server in Java

Alexander Day ChaffeejGuru Training by the MageLang Institutealex@jguru.com

Abstract

• Mark Andreesen says, ”Client-side Java is dead." Fortunately, server-side Java is alive and kicking! This session will quickly introduce you to the concepts of threading and networking in Java. We then build a robust, multithreaded server (using a simple chat as our application). We examine several different threading models, specifically comparing the one-thread-per-connection model with the queuing model.

Introduction

• jGuru Training by the Magelang Institute– http://www.jguru.com– Java Training and Consulting

• Alex Chaffee– Creator of Gamelan– Cool Java Dude

Background

• Java language basics• Java networking• Java threading

Overview

• Start slow, end up really really fast• Part I: Networking and I/O in Java• Part II: Threads and concurrency• Part III: Multithreaded server design• Part IV: Implementations

– Design decisions– Object models– Problems and solutions

Part I: Networking and I/O in Java

Java and Networking

• Built into language• One of the 11 buzzwords• Network classloader• Java.Net API• Based on TCP/IP, the internet protocol

Client-server

• Difference between client and server is semantic

• It’s all just peers talking to each other• Protocol - roles, vocabulary, rules for

communication

TCP/IP: The Internet Protocol

Physical Network

Transport Layer (TCP, UDP)

Internet Layer (IP)

Application Layer (HTTP, FTP, SMTP)

Sockets and Ports

• Port: a meeting place on a host– One service per port– 1-1023 = well-known services– 1024+ = experimental services, temporary

• Socket: a two-way connection

Client

Sockets and Ports (Diagram)

port 13

port 80

Time Service

Web Service

Socket

Server

Socket

The Socket Class

• Socket(String host, int port)• InputStream getInputStream()• OutputStream getOutputStream()• void close()

Socket Code

Socket s = new Socket(“www.sigs.com”, 80);

• Simple, huh?• Creates the connection• We still need to get the data there and

back again

Streams

• A stream is a sequence of bytes• I/O from disk, network, memory, etc. is

handled in exactly the same way

InputStream and OutputStream

• InputStream:abstract int read()

• OutputStream:abstract void write(int b)

• Common:abstract void close()

Filter Streams

• Like photographic filters• Change the input (or output) on the

way through• E.g.: BufferedInputStream,

PrintStream, LineNumberInputStream

Filter Streams (Diagram)

FileInputStream AllCapsInputStreamAbc

NoVowelInputStreamBC

DataInputStream

readLine()

“BC” DataInputStream in = new DataInputStream(new NoVowelInputStream(

new AllCapsInputStream(new FileInputStream(“input.txt”))));

String line = in.readLine();

PrintStream

• print()– outputs an object, primitive, or String

• println()– same, but adds a newline

• System.out is a PrintStream

PrintStream ps =

new PrintStream(outputstream)

DataInputStream

• int readInt()• type readType()• String readLine()

DataInputStream in =

new DataInputStream(inputstream)

Browser.java (source)

import java.net.*;

import java.io.*;

public class Browser {

public static void main(String[] args)

String host = "www.stinky.com";

int port = 80;

String file = "/index.html";

Socket s = new Socket(host, port);

InputStream strIn = s.getInputStream();

OutputStream strOut = s.getOutputStream();

PrintStream out =

new PrintStream(strOut);

DataInputStream in =

new DataInputStream(strIn);

out.println("GET " + file + " HTTP/1.0");

out.println();

String line;

while ((line = in.readLine()) != null) {

System.out.println(line);

in.close();

out.close();

Java Servers

• A server listens on a port and accepts connections

• Must be applications (or signed applets)

• Only one service per port for the entire host machine– Java throws an exception if you try to open

more than one

ServerSocket

ServerSocket ss = new ServerSocket(1234);

Socket socket = ss.accept();

• accept() returns only after a client makes the connection

ServerSocket

• Now we have a socket, so we can call...

InputStream in = socket.getInputStream();

OutputStream out = socket.getOutputStream();

EchoServer.java

ServerSocket ss =

new ServerSocket(1234);

Socket s = ss.accept();

InputStream in = s.getInputStream();

OutputStream out = s.getOutputStream();

EchoServer.java

int ch;

while ((ch = in.read()) != -1) {

out.write(ch);

out.close();

in.close();

EchoServer Demo

• Connect• Type some stuff• Disconnect• Problem: Multiple simultaneous

connections– Connections are made, but server doesn’t

respond– Even if we loop, can still handle only one at a

Multiple Connections

• Solution:– Java threads!

Part II: Threads

Java Threads

• A thread is not an object• A thread is a flow of control• A thread is a series of executed

statements• A thread is a nested sequence of

method calls

The Thread Object

• A thread is not an object• A Thread is an object

void start()– Creates a new thread and makes it runnable

void run()– The new thread begins its life inside this

method

Thread Creation Diagram

Thread t = new BThread();

t.start();

doMoreStuff();

BThread() {}

void start() {// create thread

void run() {doSomething();

Object A Object BThread (extends Thread)

t.start();

doMoreStuff();

Thread Creation DiagramObject A

BThread() {}

Object BThread (extends Thread)

t.start();

doMoreStuff();

BThread() {}

Thread Creation DiagramObject A Object BThread (extends

Thread)

t.start();

doMoreStuff();

BThread() {}

Thread)

t.start();

doMoreStuff();

BThread() {}

Thread)

t.start();

doMoreStuff();

BThread() {}

Thread)

t.start();

doMoreStuff();

BThread() {}

Thread)

t.start();

doMoreStuff();

BThread() {}

Thread)

t.start();

doMoreStuff();

BThread() {}

Thread)

Runnable Interface

• A helper to the thread object• The Thread object’s run() method calls

the Runnable object’s run() method• Allows threads to run inside any

object, regardless of inheritance

Runnable Example

Talker talker = new Talker();

Thread t = new Thread(talker);

t.start();

class Talker implements Runnable {

public void run() {

while (true) {

System.out.println(“yakitty yak”);

Blocking Threads

• When reading from a stream, if input is not available, the thread will block

• Thread is suspended (“blocked”) until I/O is available

• Allows other threads to automatically activate• When I/O available, thread wakes back up

again– Becomes “runnable”– Not to be confused with the Runnable interface

Thread Scheduling

• In general, the runnable thread with the highest priority is active (running)

• Java is priority-preemptive– If a high-priority thread wakes up, and a low-

priority thread is running– Then the high-priority thread gets to run

immediately

• Allows on-demand processing– Efficient use of CPU

Thread Starvation

• If a high priority thread never blocks• Then all other threads will starve• Must be clever about thread priority

Thread Priorities: General Strategies

• Threads that have more to do should get lower priority

• Counterintuitive• Cut to head of line for short tasks• Give your I/O-bound threads high priority

– Wake up, immediately process data, go back to waiting for I/O

Race Conditions

• Two threads are simultaneously modifying a single object

• Both threads “race” to store their value

• In the end, the last one there “wins the race”

• (Actually, both lose)

Race Condition Example

class Account {

int balance;

public void deposit(int val)

int newBal;

newBal = balance + val;

balance = newBal;

}Looks good, right?

Race Condition Example

class Account {

int balance;

public void deposit(int val)

int newBal;

newBal = balance + val;

balance = newBal;

}What if we are swapped out right here?

Thread Synchronization

• Language keyword: synchronized

• Takes out a monitor lock on an object– Exclusive lock for that thread

• If lock is currently unavailable, thread will block

Thread Synchronization

• Protects access to code, not to data– Make data members private

– Synchronize accessor methods

• Puts a “force field” around the locked object so no other threads can enter

• Actually, it only blocks access to other synchronizing threads

• A rogue thread can "sneak in" and modify a variable it has access to

Synchronization Example

class Account { private int balance; synchronized public void deposit(int val) { int newBal; newBal = balance + val; balance = newBal; } synchronized public void withdraw(int val) { int newBal; newBal = balance - val; balance = newBal; }}

Thread Deadlock

• If two threads are competing for more than one lock

• Example: bank transfer between two accounts– Thread A has a lock on account 1 and wants

to lock account 2

– Thread B has a lock on account 2 and wants to lock account 1

Avoiding Deadlock

• No universal solution

• Ordered lock acquisition

• Encapsulation (“forcing directionality”)

• Spawn new threads

• Check and back off

• Timeout

• Minimize or remove synchronization

Wait and Notify

• Allows two threads to cooperate

• Based on a single shared lock object

Wait and Notify: Code

• Consumer:synchronized (lock) {

while (!resourceAvailable()) {

lock.wait();

consumeResource();

Wait and Notify: Code

• Producer:produceResource();

synchronized (lock) {

lock.notifyAll();

Wait/Notify Sequence

Lock Object

ConsumerThread

ProducerThread

1. synchronized(lock){

2. lock.wait();

3. produceResource()4. synchronized(lock) {5. lock.notify();6.}

7. Reacquire lock8. Return from wait()

9. consumeResource();10. }

Lock Object

ConsumerThread

ProducerThread

2. lock.wait();

Lock Object

ConsumerThread

ProducerThread

2. lock.wait();

Lock Object

ConsumerThread

ProducerThread

2. lock.wait();

Lock Object

ConsumerThread

ProducerThread

2. lock.wait();

Lock Object

ConsumerThread

ProducerThread

2. lock.wait();

Lock Object

ConsumerThread

ProducerThread

2. lock.wait();

Lock Object

ConsumerThread

ProducerThread

2. lock.wait();

Lock Object

ConsumerThread

ProducerThread

2. lock.wait();

Lock Object

ConsumerThread

ProducerThread

2. lock.wait();

Lock Object

ConsumerThread

ProducerThread

2. lock.wait();

Wait/Notify: Details

• Often the lock object is the resource itself

• Sometimes the lock object is the producer thread itself

Wait/Notify: Details

• Must loop on wait(), in case another thread grabs the resource...– After you are notified– Before you acquire the lock and return from

wait()

• Use lock.notifyAll() if there may be more than one waiting thread

Wait/Notify Example: Blocking Queue

class BlockingQueue extends Queue {

public synchronized Object remove() {

while (isEmpty()) {

wait(); // really this.wait()

return super.remove();

public synchronized void add(Object o) {

super.add(o);

notifyAll(); // this.notifyAll()

Part III: Multithreaded Server Design

Different Models of Information Flow

• Pull model• Push model: One thread per client• Buffer model: One thread per task• Single-thread model

Pull-based Flow

• Consumer asks producer for new information

• Demand-driven problems– Examples

• Not appropriate for chat/message-passing server

Push-based Flow

• Producer triggers sequence of events• One thread per client• Receives message, delivers it, waits

Push-based: Pro

• Simple• Efficient use of CPU

– Events occur only when message available - no polling

– No time wasted passing information from one stage to the next

Push-based: Con

• Much thread swapping• Slow receivers can stall upcoming

messages from sender

Buffer Model

• Independent threads• Communicate via buffers• Producer puts message in buffer,

consumer takes message out

Buffer Model: Pro

• Can separate tasks into separate threads

• Optimize and prioritize per task– e.g. Processing existing messages is more

important than accepting new connections

• Small number of threads means fewer chances for deadlock– Easier to model the system and avoid any

chance of deadlock

Buffer Model: Con

• More complex code

• Must deal with deadlock and starvation

The One-Thread Model

• All work done by a single thread

• Finishes processing one request before going to the next

The One-Thread Model (Cont.)

• Pro:– Suitable for low-traffic scenarios

– Quick to implement

– Avoids any possibility of deadlock or race conditions

• Con– Unsuitable for high-traffic server

– Hard to implement things like timers or callbacks

Part IV: Implementations

Implementation 1: Push Model Chat Server

• Listens on a port• Accepts connections• Spawns threads, one per connection• Receives a message from one client• Dispatches it to all clients

Chat Server Objects

• ChatServer– Main

• ClientHandler– One per connection– Threaded

• Dispatcher– Vector

Chat Server Objects (Diagram)

ChatServer

ClientHandlerClientHandler

Dispatcher

Socket Socket

ClientHandler

• Stores information for a single clientSocket incoming;

int id;

Dispatcher dispatcher;

ClientHandler.run()

• generate filter streams• read a line• send it to the dispatcher• loop

ClientHandler.send()

• sends a single message to the client• called by the dispatcher• uses the socket

Dispatcher

• Keeps a list of all client handlers• Uses java.util.Vector

– synchronized list structure

Dispatcher.dispatch()

• Sends a message to all clients• Accepts a String and an identifying

integer• Enumerates down all client handlers

and calls each one’s send() method

ChatServer

• One method: main()• Creates a new ServerSocket• Creates a new Dispatcher• Listens for connections• When connection received, creates a

new ClientHandler and starts a new thread running inside it

• Adds it to the Dispatcher

Push model: Pro and Con

• Pro– Straightforward object, thread models– Efficient CPU usage

• Threads unblock only when message available

• Con– Must send message to all clients before

receive new one– No upper limit on number of threads

• May thrash or starve

Implementation 2: Buffer Model Message Server

Queuing Model

• Producer and consumer communicate via a buffer

• Buffer is a queue• First in, first out

Blocking Queue

• If a thread tries to remove an item from an empty queue

• Then it blocks until an item is placed inside

• Uses wait/notify technique

JDK 1.2 Collections

• Set• List• Map• Queue

– Built on top of LinkedList

Basic Object Model

• Acceptor (thread)• Clients (set)• Receiver (thread)• Incoming (queue)• Processor (thread)

Acceptor

Receiver

Clients

Incoming

Processor

Thread

create

Basic Object Model (Fig.)

Acceptor Thread

• Listens for incoming connections• Creates a client object• Adds it to clients set

Clients Set

• Set of all active clients• Synchronized methods

– Only one thread at a time can access

Receiver Thread

• Walks down clients set• For each client, checks if message is

(fully) available• Reads message and places it on

incoming queue• Uses non-blocking input streams

Receiver Thread: Alternative

• Alternative: one receiver thread per client

• Each receiver thread blocks until input available

• Con:– Reach number of threads bottleneck– Scheduling not determined - could starve or

do in weird order

Incoming Queue

• Queue of messages• Synchronized methods

– Only one thread at a time can access

• Blocking queue– If a thread tries to get a message from an

empty queue, it blocks until another thread adds a message

Processor Thread

• While incoming queue is not empty• Pops message off incoming queue• Delivers it

Full Object Model

• Checker (thread)• Problems (list)• Auditor (thread)

Acceptor

Receiver

Clients

Incoming

Processor

CheckerProblems

Auditor

Thread

Full Object Model (Fig.)

Problems List

• List of clients who have problems• Receiver can add a client to problems

list if it gets a socket read error• Processor can add a client to problems

list if it gets a socket write error• Checker thread periodically

disconnects them all

Checker Thread

• Wakes up every N seconds• Walks list of clients• Move to problems list

– If N seconds has passed (“timeout”)– If socket has a problem

• Walk problems list and actively close each client

Auditor Thread

• Every N seconds, wakes up• Collects and prints statistics

Non-Blocking Input Stream

• Needed for receiver to do its job• If a full message is not available, it

does NOT block• Instead, it returns null• Tricky to implement

– Must use available(), mark() and reset() cleverly

Thread Safety Policies

• All data objects synchronized• All data objects independent• Therefore no possibility of deadlock in

the data objects themselves

Thread Safety Policies

• Thread objects must be very careful to avoid deadlock

• Current model: no multi-object transactions• Example

– Checker does clients.remove(c) then problems.add(c)

– NOT synchronized(clients) { synchronized(problems) { clients.remove(c); problems.add(c);}}

– It's OK if it's interrupted between remove and add

Priority: Why?

• In low-volume server, all threads are normally blocked

• When something needs to happen, it happens

• That's the beauty of threads!• So priorities are irrelevant

– If you don’t know what you’re doing, don’t mess with priorities

Priority: Why?

• In high-volume server, many threads always have something to do

• Important to do things in the right order

• Deal with starvation issue

Priorities: General Strategies

• Threads that have more to do get lower priority

• Counterintuitive• Cut to head of line for short tasks

Priorities: Queuing Strategies

• Most important: keep incoming queue empty

• So we don't get bogged down• Minimize latency for received

messages

Priorities: Decisions

• Don't want to add new messages if we're already busy– Therefore processor > receiver

• Don't want to accept new connections if we're already busy– Therefore processor > acceptor

• Implication: – Receiver and acceptor may starve?– No: queue will eventually get cleared

• Want to be able to accept new connections, even if a currently connected client has something to say– Therefore acceptor > receiver

• Checker's job is brief but important• Needs to wake up, check all clients,

and go back to sleep• Therefore checker > processor

• Auditor's job is brief but important• Need to get accurate statistics --

timing is crucial• Can't wait around for other threads to

finish• Therefore auditor > checker

Priorities: Conclusion

• Auditor - 10• Checker - 8• Processor - 6• Acceptor - 4• Receiver - 2• Goes by twos to accommodate

Windows threading model

Processor Thread Behavior

• Processor thread does:1. Pop message off incoming queue2. Deliver it3. Loop

• Very independent

Processor Thread Blocking

• Step 1 (pop message) may block• If queue is empty• This is good• Allows other threads to wake up and

fill queue

Processor Thread Blocking

• Step 2 (delivery) may block– If network is busy– If OS socket buffer is full

• Bad, because queue is still full• Increases latency (delivery time) for

already-queued messages – While new messages are arriving, old

messages should be delivered instead

Multiple Processor Threads

• Solution: have many processor threads working simultaneously

• If one blocks, another picks up the next message

Multiple Processors (Fig.)

Acceptor

Receiver

Clients

Incoming

Processor

Thread

create

ProcessorProcessorProcessor

How Many Threads?

1. Constant, tuned per application2. Dynamic a/k/a thread pooling

Thread Pooling

• Thread pool keeps a certain number of threads alive

• Other threads ask the thread pool to perform a task

• If an existing thread is available, that thread performs the task

• Else a new thread is created, or the task blocks until one is available – (depending on policy)

Thread Pooling (Cont.)

• Removes overhead of creating threads• Allows modular task strategy

Information Flow

• Decoupling threads with buffers is a good idea– Maximizes concurrency– Smooths out bursty differences in rate

• Can cause problems with information flow– Producers outpace Consumers– Too many threads

Limiting Flow

• Bounded Buffers– Stalls producers if they outpace consumers

• Bounded Thread Pools– Prevents thrashing

• Rate-adjustment / back-pressure– Consumer asks Producer to slow down

Another Problem

• One message, multiple recipients, one processor thread

• If one recipient is in Finland, it will stall the remaining recipients for that message

Another Problem: Solutions

• Solution 1: one message queue per client; thread pool applied to these delivery queues

• Solution 2: split multi-message into several single-recipient messages– Called a Splitter (or Fork or Multicaster)– Also possible: Routers, Mergers, Collectors,

Combiners, Conduits -- see Lea’s Concurrent Programming in Java

Debugging Techniques• Make sure to trap all Throwables in

your Thread object’s run() method– Otherwise a stray OutOfMemoryError or

NullPointerException will kill your thread and it will look like deadlock

• Good logging procedure is vital– I use a global log method that outputs the

name of the thread and the time in addition to the message

– Name your threads at creation time

Thread Dump

• Control-backslash in Solaris• Control-break in Windows

– Unfortunately, it scrolls– There’s no way to redirect standard error– Oops

Thread Watcher

• Data object• All tasks (threads) have a pointer to the

master ThreadWatcher• Put debug code in your tasks to inform the

ThreadWatcher when they change state• watcher.setIdle();• Message m = incoming.remove();• watcher.set(“Processing message “ + m);

Thread Watcher (cont.)

• ThreadWatcher has its own thread– Periodically prints status of threads– Prints an error message if one thread is

blocked for too long

• Alternative to fancy GUI thread debugger– Symantec Café– Compuware DevPartner– et al.

• PingBot– periodically sends a message to itself,

measures the latency

• QuoteBot– sends random Zippy quotes to mimic a chat

Measurements

• Number of bots before sharp latency increase

• Average latency of single PingBot for N ZippyBots

Test Architecture

• Need at least three machines– Server– Zippy Bot Host– Ping Bot Host

• So they don’t interfere with one another• Should also have some clients on a far

away machine– If your server will be running on the Internet

Conclusion

Thanks to

• Eric Malmstrom• Carl Muckenhoupt• Gerry Seidman• Greg Travis• For helping prepare this talk

Where to Get More Information: Network Programming

• Cornell & Horstmann, Core Java (Sunsoft press)

• Harold, Java Network Programming (O’Reilly)

• Hughes et al., Java Network Programming, (Manning, an imprint of Prentice-Hall)

Where to Get More Information: Multithreading

– Cornell & Horstmann, Core Java (sunsoft press)

– Oaks and Wong, Java Threads (o’reilly)

– Lea, Concurrent Programming in Java (Addison Wesley)

– Travis, Using thread pools to increase threading efficiency, developer.com, http://www.Developer.Com/journal/techworkshop/060498_thread.html

– Travis, Working with the blocking queue, http://www.Developer.Com/journal/techworkshop/091098_blockq.html

Where to Get More Information:

• Web sites– http://www.jGuru.com/ (Java Training)

– http://www.Developer.com/ (Gamelan)– http://www.Javaworld.com/ (magazine)– http://www.Purpletech.com/ (author’s site)

Copyright © 1998 Alex Chaffee Building a Robust Multithreaded Server in Java Alexander Day Chaffee...

Documents

County Report Chaffee County Colorado · COUNTY PROFILE: Chaffee County, Colorado Chaffee County, Colorado | page 1 US COUNTY PERFORMANCE The Institute for Health Metrics and Evaluation

270 Michigan Avenue Chaffee Facility - Chaffee Landfillchaffeelandfill.wm.com/images/pdf/permit_modification_letter.pdf · Chaffee Facility – Liquid Solidification Process Permitting

©1996-2000 jGuru.com Enterprise JavaBeans Fundamentals

Sticking Patterns - Gary Chaffee

Gary Chaffee - Time Functioning Patterns

Loomis Chaffee Summer Program - ctparent.com · Loomis Chaffee Summer Program Loomis Chaffee School 4 Batchelder Road Windsor, CT 06095 ... Jeff Scanlon, the LC SPDirector, at Jeff_Scanlon@loomis.org

Gary Chaffee - Linear Time Playing

Patterns - Sticking (Gary Chaffee)

Revolution Motorsport Scott Chaffee, Jonathan … Motorsport Scott Chaffee, Jonathan Jackson, Adil Nurgozhin Jon Jackson (406) 599-3572 jacksonmontana@gmai!.colll Scott Chaffee (253)

FRESNO CHAFFEE · 2015. 1. 22. · FRESNO CHAFFEE Fresno Chaffee Zoo. 894 West Belmont Avenue. Fresno, California 93728 Phone (559) 498-5910 • Fax (559) 264-9226 Disbursement Claim

1 Introduction to JFC/Swing Alex Chaffee jGuru Training by the MageLang Institute alex@jguru.com

The Loomis Chaffee Log

Chaffee Summer 2011chaffeelandfill.wm.com/pdfs/WMChaffeeNews_July_2011.pdf · 2011. 8. 29. · News from Waste Management’s Chaffee Landfill WM Chaffee Landfill 10860 Olean Road

Gary Chaffee - Sticking Patterns

Gary Chaffee - Technique Patterns

Are Agile Projects Doomed to Half-Baked Design? Alex Chaffee alex@PivotalLabs.com Leslie Chicoine leslie@GetSatisfaction.com

Gary Chaffee - Sticking Patterns.pdf

Chaffee Grant program update

S A M - Chaffee County Clerk and Recorder · Gary Johnson / James P Gray Libertarian Jill Stein / Cheri Honkala Green Stewart Alexander / Alex Mendoza ... Chaffee County, Colorado

Copyright © 1998 Alex Chaffee Bean Soup: Using Components With Java Alexander Day Chaffee alex@jguru.com jGuru Training by the Magelang Institute