Multi-threaded Lisp: Multi-threaded Lisp: Challenges and Solutions Challenges and Solutions

Roger CormanRoger CormanCorman TechnologiesCorman Technologies

International Lisp Conference International Lisp Conference 20022002

Threads: Why Bother?Threads: Why Bother? Most modern applications use multiple threads of Most modern applications use multiple threads of

execution.execution.

Nearly all servers require threads for reasonable Nearly all servers require threads for reasonable performance.performance.

New languages have threads support built into New languages have threads support built into the language.the language.

Most modern operating systems support threads.Most modern operating systems support threads.

Lisp is cool; threads are cool.Lisp is cool; threads are cool.

RightRight Way To Think About Threads Way To Think About Threads

All threads run simultaneously.All threads run simultaneously.

If they don’t, it doesn’t matter.If they don’t, it doesn’t matter.

If you have more than one processor, they If you have more than one processor, they sometimes do.sometimes do.

This model leads you to consider, early, issues This model leads you to consider, early, issues such as synchronization, data sharing, such as synchronization, data sharing, performance and messaging.performance and messaging.

WrongWrong Way to Think About Way to Think About ThreadsThreads

Multiple execution contexts which run one at a Multiple execution contexts which run one at a time.time.

Thread-switching can be intercepted to perform Thread-switching can be intercepted to perform arbitrary logic.arbitrary logic.

Thread-switching can be forced to happen only at Thread-switching can be forced to happen only at appropriate times.appropriate times.

This model will never support multiple processors, This model will never support multiple processors, and will not even perform as well on a single CPU and will not even perform as well on a single CPU system.system.

It is easy to develop bad programming habits, It is easy to develop bad programming habits, such as WITHOUT-INTERRUPTS.such as WITHOUT-INTERRUPTS.

Next up: Thread

What a What a thread thread looks like:looks like:

Personal BackgroundPersonal Background

Developed multi-threaded applications in Developed multi-threaded applications in C/C++/Java.C/C++/Java.

Designed Corman Lisp to include all the Designed Corman Lisp to include all the capabilities of these other language platforms.capabilities of these other language platforms.

Had to make a lot of decisions about how to Had to make a lot of decisions about how to implement multiple threads in Lisp.implement multiple threads in Lisp.

Threads Are an OS FunctionThreads Are an OS Function Just as the file system is a function of the Just as the file system is a function of the

operating system, threads should be allocated operating system, threads should be allocated and scheduled by the operating system.and scheduled by the operating system.

While lisp-implemented threads can be useful, While lisp-implemented threads can be useful, they will never be as powerful as operating they will never be as powerful as operating system threads.system threads.

Many Lisp implementations have supported lisp-Many Lisp implementations have supported lisp-implemented threads, and a standard API has implemented threads, and a standard API has developed which is supported by several developed which is supported by several implementations. It implies lisp-implemented implementations. It implies lisp-implemented threads, with fine-grained control over task threads, with fine-grained control over task scheduling.scheduling.

Lisp Process:Lisp Process:Lisp-Implemented ThreadsLisp-Implemented Threads

Operating System Thread 0

Lisp Scheduled Thread 0

Lisp Scheduled Thread 1

Lisp Scheduled Thread 2

ProcessorA

ProcessorB

ProcessorC

Lisp Process:Lisp Process:Operating System ThreadsOperating System Threads

Operating System Thread 0

Operating System Thread 1

Operating System Thread 2

ProcessorA

ProcessorB

ProcessorC

Corman Lisp Threading ModelCorman Lisp Threading Model

In the Corman Lisp implementation, we chose to In the Corman Lisp implementation, we chose to implement implement onlyonly operating system threads. operating system threads.

These give Lisp applications the maximum These give Lisp applications the maximum capabilities provided by the operating system.capabilities provided by the operating system.

The Win32 API allows the application very limited The Win32 API allows the application very limited control over thread scheduling. We don’t view control over thread scheduling. We don’t view this as a OS limitation, but rather as an OS this as a OS limitation, but rather as an OS feature.feature.

Threads and the Common Lisp Threads and the Common Lisp StandardStandard

While lightweight processes have been supported While lightweight processes have been supported in many Lisp implementations, the standard does in many Lisp implementations, the standard does not include any discussion of threads.not include any discussion of threads.

The standard contains language which could The standard contains language which could imply non-conformance of multi-threaded imply non-conformance of multi-threaded implementations.implementations.

Thread Implementation IssuesThread Implementation Issues Special variablesSpecial variables

Heap accessHeap access

Garbage collectionGarbage collection

Foreign calls and callbacksForeign calls and callbacks

Code generationCode generation

DebuggingDebugging

Lisp Image saving and loading Lisp Image saving and loading

Foreign ThreadsForeign Threads

From ANSI Specification: From ANSI Specification:

3.1.2.1.1.2 Dynamic Variables3.1.2.1.1.2 Dynamic Variables

The effect of binding a dynamic variable is to create The effect of binding a dynamic variable is to create a new binding to which a new binding to which allall references to that references to that dynamic variable in any program refer for the dynamic variable in any program refer for the durationduration of the evaluation of the form that creates of the evaluation of the form that creates

the dynamic binding.the dynamic binding.

From ANSI Specification Glossary:From ANSI Specification Glossary:

dynamic extentdynamic extent n.n. an extent whose an extent whose durationduration is is bounded by points of establishment and bounded by points of establishment and disestablishment within the execution of a disestablishment within the execution of a particular form. See indefinite extent. “Dynamic particular form. See indefinite extent. “Dynamic variable bindings have dynamic extent.” variable bindings have dynamic extent.”

extentextent n.n. the the interval of timeinterval of time during which a during which a reference to an object, a binding, an exit point, a reference to an object, a binding, an exit point, a tag, a handler, a restart, or an environment is tag, a handler, a restart, or an environment is defined. defined.

Special Variables And ThreadsSpecial Variables And Threads

Top level bindings should be shared between Top level bindings should be shared between threads.threads.

Dynamic bindings (via LET, etc.) should only Dynamic bindings (via LET, etc.) should only apply to the thread the created the binding.apply to the thread the created the binding.

Is there ever a reason for dynamic bindings to be Is there ever a reason for dynamic bindings to be visible to other threads? (I haven’t found one.)visible to other threads? (I haven’t found one.)

Implementation of Dynamic BindingImplementation of Dynamic Binding

Many Lisp implementations store special variable Many Lisp implementations store special variable bindings in a symbol’s value slot. This is a bindings in a symbol’s value slot. This is a problem in a multi-threaded system.problem in a multi-threaded system.

Special variable bindings need to be stack-Special variable bindings need to be stack-specific, yet efficient.specific, yet efficient.

““Deep binding” where bindings are kept in a list, Deep binding” where bindings are kept in a list, can be inefficient.can be inefficient.

Dynamic Bindings in Corman LispDynamic Bindings in Corman Lisp

Top level (shared) bindings are stored in symbol’s Top level (shared) bindings are stored in symbol’s value slot.value slot.

Dynamic bindings are stored in a per-thread Dynamic bindings are stored in a per-thread symbol table.symbol table.

Only symbols which are actually dynamically Only symbols which are actually dynamically bound (in loaded, compiled code) need a slot in bound (in loaded, compiled code) need a slot in the per-thread symbol table.the per-thread symbol table.

Special Variable BindingsSpecial Variable Bindings

*X* *Z**Y*

binding binding binding

*X* *Y* *Z* etc Master symbol table

Special Variable BindingsSpecial Variable Bindings

*X* *Z**Y*

binding binding binding

*X* *Y* *Z* etc

*X* *Y* *Z* etc

Master symbol table

Thread 0 Symbol Table

*X* *Y* *Z* etc Thread 1 Symbol Table

Threadlocal

binding

Threadlocal

binding

Thread Local VariablesThread Local Variables If you implement efficient per-thread special If you implement efficient per-thread special

variable binding, you end up with an variable binding, you end up with an implementation of thread local variables (à la C#, implementation of thread local variables (à la C#, Java).Java).

Surprise: they are far easier to use, elegant, and Surprise: they are far easier to use, elegant, and fit into the language better than most other fit into the language better than most other thread local variable models in other languages.thread local variable models in other languages.

Example: JavaExample: Java public class MyThread extends Threadpublic class MyThread extends Thread {{ // static Hashtable stuff;// static Hashtable stuff; public static ThreadLocal stuff = new ThreadLocal();public static ThreadLocal stuff = new ThreadLocal();

public void run() public void run() {{ // each thread must have a different hashtable// each thread must have a different hashtable stuff.set(new Hashtable);stuff.set(new Hashtable);

if (((Hashtable)stuff.get()).containsValue("Foo"))if (((Hashtable)stuff.get()).containsValue("Foo")) doSomething(); // run code which can refer to hash tabledoSomething(); // run code which can refer to hash table }} }}

Example: Common LispExample: Common Lisp

(defvar *hash-table*)(defvar *hash-table*)

(defun run () (defun run ()

(let ((*hash-table* (make-hash-table)))(let ((*hash-table* (make-hash-table)))

(do-something))) ; run code which can refer to hash table(do-something))) ; run code which can refer to hash table

Next Up: Binding

Dynamic Dynamic BindingBinding

Garbage CollectionGarbage Collection

2 Big Issues:2 Big Issues:

1) Do you stop the mutator, i.e. all threads, 1) Do you stop the mutator, i.e. all threads, to perform garbage collection?to perform garbage collection?

2) Impact on code generation.2) Impact on code generation.

Garbage Collection (cont.)Garbage Collection (cont.)

Stopping the mutator:Stopping the mutator:

Overall performance hit (higher cost) when Overall performance hit (higher cost) when multiple processors are available (unless your multiple processors are available (unless your garbage collection algorithm is a parallel garbage collection algorithm is a parallel algorithm).algorithm).

Generally lower overall cost on single processors.Generally lower overall cost on single processors.

Simpler algorithms.Simpler algorithms.

Garbage Collection (cont.)Garbage Collection (cont.)

Parallel execution:Parallel execution:

Threads keep running.Threads keep running.

Scales better to many processors.Scales better to many processors.

Higher overall cost on single processor due to Higher overall cost on single processor due to code generation, extra root scanning.code generation, extra root scanning.

Minimizes pauses for collection.Minimizes pauses for collection.

Complex algorithms, more error-prone.Complex algorithms, more error-prone.

Corman Lisp Garbage CollectorCorman Lisp Garbage Collector

We chose to use a generational collector which We chose to use a generational collector which stops the mutator (all threads) but keeps pauses stops the mutator (all threads) but keeps pauses short (a few milliseconds, usually).short (a few milliseconds, usually).

Most target systems have only one processor. Most target systems have only one processor. Windows systems do not support more than 4 (or Windows systems do not support more than 4 (or 8) processors. 8) processors.

Does not require a software write barrier, or any Does not require a software write barrier, or any type of read barrier.type of read barrier.

Code generation is minimally impacted.Code generation is minimally impacted.

Corman Lisp Garbage Collector (cont.)Corman Lisp Garbage Collector (cont.)

1.1. Suspends all executing threads.Suspends all executing threads.

2.2. If any thread is in an atomic block, resume it for If any thread is in an atomic block, resume it for a few milliseconds and go to step 1.a few milliseconds and go to step 1.

3.3. Performs root analysis on all stacks, heaps, and Performs root analysis on all stacks, heaps, and static memory.static memory.

4.4. Copies live data from generation 0 to 1 Copies live data from generation 0 to 1 (triggering other levels if necessary).(triggering other levels if necessary).

5.5. Resumes all executing threads.Resumes all executing threads.

Garbage Collection and Code Garbage Collection and Code GenerationGeneration

In a single-threaded system, garbage collection In a single-threaded system, garbage collection will only occur at “safe states” (usually when a will only occur at “safe states” (usually when a heap allocation is requested).heap allocation is requested).

In a multi-threaded system, garbage collection In a multi-threaded system, garbage collection can occur at any time, potentially between any can occur at any time, potentially between any pair of instructions. Restricting it to safe states pair of instructions. Restricting it to safe states can be difficult and require code generator can be difficult and require code generator assistance.assistance.

Some code optimizations are impractical.Some code optimizations are impractical.

Next up: Me

Me, after a Me, after a long night long night debugging debugging the garbage the garbage collector.collector.

Foreign Calls and Callbacks (FFI) Foreign Calls and Callbacks (FFI)

Foreign Call:Foreign Call:Lisp code calls operating system, C function or any Lisp code calls operating system, C function or any external library.external library.

Foreign Callback:Foreign Callback:Operating system, C function, or any other code Operating system, C function, or any other code calls Lisp function directly.calls Lisp function directly.

FFI and ThreadsFFI and Threads

A thread may be executing foreign code when the A thread may be executing foreign code when the garbage collector is called. It may not be in a garbage collector is called. It may not be in a predictable state for the collector.predictable state for the collector.

Need a way for the collector to determine which Need a way for the collector to determine which threads were executing foreign code and which threads were executing foreign code and which were executing Lisp code. Lisp code could have were executing Lisp code. Lisp code could have been called by foreign code.been called by foreign code.

Tricky issue: “thunking” from Lisp-to-foreign or Tricky issue: “thunking” from Lisp-to-foreign or foreign-to-Lisp code. The collector can run any foreign-to-Lisp code. The collector can run any time during the process.time during the process.

Multi-threading and Code Multi-threading and Code GenerationGeneration

The code generator must cooperate with the The code generator must cooperate with the collector to ensure that executing code can be collector to ensure that executing code can be analyzed by the collector as necessary. analyzed by the collector as necessary.

Areas of interest:Areas of interest:

Register ConventionsRegister Conventions Tagging ConventionsTagging Conventions Stack frame linksStack frame links Function calling/returningFunction calling/returning Foreign code transitionsForeign code transitions Stack-allocated data Stack-allocated data

Multi-threading and Code Multi-threading and Code GenerationGeneration

Many common optimizations are difficult or Many common optimizations are difficult or impossible if the collector can run at any time. impossible if the collector can run at any time.

Implementation Strategies:Implementation Strategies:

a) Restrict the collector to only run at “safe states” a) Restrict the collector to only run at “safe states” and allow these optimizations at any other places in and allow these optimizations at any other places in the code.the code.

b) Never allow optimizations or generated code b) Never allow optimizations or generated code which would trip up the collector at any point.which would trip up the collector at any point.

c) Allow code generator to designate portions of the c) Allow code generator to designate portions of the code which are non-safe.code which are non-safe.

Corman Lisp Code GenerationCorman Lisp Code Generation

We implemented option (c) using a pair of We implemented option (c) using a pair of instructions:instructions:

BEGIN_ATOMICBEGIN_ATOMICEND_ATOMICEND_ATOMIC

These mark blocks of code which are protected These mark blocks of code which are protected from garbage collection.from garbage collection.

The atomic sections may never span calls or The atomic sections may never span calls or branches, and typically should only include a few branches, and typically should only include a few instructions.instructions.

Corman Lisp Code Generation (cont.)Corman Lisp Code Generation (cont.)

An An atomic blockatomic block is either: is either:

a)a) Instruction sequence between Instruction sequence between BEGIN_ATOMIC/END_ATOMIC instructions.BEGIN_ATOMIC/END_ATOMIC instructions.

b)b) Instructions that build up or tear down a stack Instructions that build up or tear down a stack frame (beginning or end of function call).frame (beginning or end of function call).

The latter is not safe because the stack frame list The latter is not safe because the stack frame list may be invalid, and the collector could be may be invalid, and the collector could be tripped up. It is impractical to use tripped up. It is impractical to use BEGIN_ATOMIC/END_ATOMIC in all these places, BEGIN_ATOMIC/END_ATOMIC in all these places, so the collector has special logic to detect that so the collector has special logic to detect that the instruction pointer is at one of these points. the instruction pointer is at one of these points.

Corman Lisp Code Generation (cont.)Corman Lisp Code Generation (cont.)

A A safe statesafe state is any point during code execution in is any point during code execution in which the instruction pointer which the instruction pointer is notis not in an atomic in an atomic block, or block, or isis executing foreign code (which cannot executing foreign code (which cannot access the heap).access the heap).

The collector will delay collection until all threads The collector will delay collection until all threads are in safe states. Some sequences of instructions are in safe states. Some sequences of instructions can only be performed in safe states.can only be performed in safe states.

Next up: Code Generator

Corman Corman Lisp Code Lisp Code GeneratorGenerator

Lisp DebuggerLisp Debugger

The ANSI specification does not go into detail The ANSI specification does not go into detail about what the common Lisp debugger does. about what the common Lisp debugger does.

Implementations commonly will enter a special Implementations commonly will enter a special READ-EVAL-PRINT loop when an error is signaled READ-EVAL-PRINT loop when an error is signaled

or a BREAK is executed.or a BREAK is executed.

This works well for checking the stack of the This works well for checking the stack of the current thread, but what should it do about other current thread, but what should it do about other threads? threads?

Lisp Debugger (cont.)Lisp Debugger (cont.)

IssuesIssues

Should other threads be halted?Should other threads be halted?

Should it be possible to examine stack frames of Should it be possible to examine stack frames of other threads, and to switch logical threads in the other threads, and to switch logical threads in the debugger?debugger?

Which thread are you dropped into when you press Which thread are you dropped into when you press control-break? control-break?

Lisp Debugger (cont.)Lisp Debugger (cont.)

I believe the best debugger will do all these things, I believe the best debugger will do all these things, but to do so it really needs to run in a separate but to do so it really needs to run in a separate thread. thread.

I don’t think debugger behavior ought to be part of I don’t think debugger behavior ought to be part of a standard, but it certainly is something an a standard, but it certainly is something an implementer has to deal with.implementer has to deal with.

Next Up: Debugger

Corman Corman Lisp Lisp DebuggerDebugger

Lisp Image Saving and Loading Lisp Image Saving and Loading

A non-standard feature which many Common Lisp A non-standard feature which many Common Lisp implementations support is loading and saving of implementations support is loading and saving of the lisp heap as an the lisp heap as an imageimage. .

It is important that other threads be at least It is important that other threads be at least suspended when saving or loading an image file, suspended when saving or loading an image file, and possibly you need to disallow other threads and possibly you need to disallow other threads

from even existing.from even existing.

I found the implementation of this to be a bit tricky. I found the implementation of this to be a bit tricky.

Foreign Threads Foreign Threads

When the Lisp system allocates a thread, it can When the Lisp system allocates a thread, it can initialize it with any necessary data for the garbage initialize it with any necessary data for the garbage collector to use when scanning it. collector to use when scanning it.

If you want to allow threads not allocated by the lisp If you want to allow threads not allocated by the lisp system to call lisp threads, there must be some way system to call lisp threads, there must be some way for the collector to know about them and figure out for the collector to know about them and figure out what to do with them. what to do with them.

If you don’t allow foreign-created threads to call If you don’t allow foreign-created threads to call into Lisp, then it impairs the ability to embed the into Lisp, then it impairs the ability to embed the Lisp engine in other applications.Lisp engine in other applications.

Corman Lisp DirectCallCorman Lisp DirectCall

DirectCall allows foreign-created threads to call DirectCall allows foreign-created threads to call directly into Lisp code.directly into Lisp code.

BlessThread()BlessThread()

UnblessThread()UnblessThread()

These must be called by any thread that calls into These must be called by any thread that calls into Lisp directly.Lisp directly.

DLLs compiled with Corman Lisp perform this DLLs compiled with Corman Lisp perform this transparently as needed. transparently as needed.

Next up: Foreign Thread

Foreign Foreign ThreadThread

(Before (Before being being blessed)blessed)

Common Lisp Standard Threading Common Lisp Standard Threading SuggestionsSuggestions

I propose that threading be incorporated in the I propose that threading be incorporated in the Common Lisp standard, at least as an optional Common Lisp standard, at least as an optional feature specified by the standard. This involves feature specified by the standard. This involves amending some of the existing language. Mostly it amending some of the existing language. Mostly it involves adding a few new macros, functions and involves adding a few new macros, functions and probably some standard classes. probably some standard classes.

Standard Common Lisp Threading Standard Common Lisp Threading Suggested AdditionsSuggested Additions

Dynamic variable specification.Dynamic variable specification.

Specify which objects/structures are safe for use Specify which objects/structures are safe for use by multiple threads (such as hash-tables, by multiple threads (such as hash-tables, packages, readtables, etc.).packages, readtables, etc.).

Specify which language features are safe for Specify which language features are safe for multi-threaded operations (read, eval, print, etc.).multi-threaded operations (read, eval, print, etc.).

Standard special variable bindings for process id, Standard special variable bindings for process id, thread id.thread id.

WITH-LOCK macro for synchronization on objects WITH-LOCK macro for synchronization on objects and structures.and structures.

Standard Common Lisp Threading Standard Common Lisp Threading Suggested Additions (cont.)Suggested Additions (cont.)

Start with some of the features of the Franz MP Start with some of the features of the Franz MP API.API.

Do not include scheduling functions or macros, Do not include scheduling functions or macros, such as WITHOUT-SCHEDULING and WITHOUT-such as WITHOUT-SCHEDULING and WITHOUT-

INTERRUPTSINTERRUPTS..

Standard of classes for synchronization such as Standard of classes for synchronization such as Mutex, Critical Section, Semaphore.Mutex, Critical Section, Semaphore.

SummarySummary

Threads are good.Threads are good.

Threads, by design, run in parallel.Threads, by design, run in parallel.

There are many system implementation issues There are many system implementation issues involved.involved.

Common Lisp standard should add a section on Common Lisp standard should add a section on standard multiple thread support.standard multiple thread support.

Illustrations help to clarify abstract concepts.Illustrations help to clarify abstract concepts.

Original Illustrations by Emmett CormanOriginal Illustrations by Emmett Corman

Copyright (c) Corman Technologies 2002Copyright (c) Corman Technologies 2002