Pipes & Filters Architecture PatternSource: Pattern-Oriented Software Architecture, Vol. 1, Buschmann, et al

Problem• You are designing a system that needs to perform several

transformations on some data

• The transformations can be performed sequentially

• It should be easy to:

– Reorder the sequence of the transformations

– Add new kinds of transformations

– Remove transformations

• There are different sources of input data (files, network, programs)

• There are different destinations for output data (files, network, programs)

Solution

• Organize the system using Pipes & Filters

• Input data originates from Data Source components

• Output data is consumed by Data Sink components

• Each transformation required to convert the input into the output is implemented by a Filter component

• Data Sources, Filters, and Data Sinks arranged in a pipeline

• Pipes connect adjacent components, the output of one component becoming the input to the next component

DataSource

Filter FilterPipe Pipe Pipe Data Sink

Solution

• Sources, Filters, and Sinks can be Active or Passive

• Active components run on their own thread of control

• Passive components execute only when invoked, directly or indirectly, by an Active component

-Delivers input to processing pipeline

Data Source

-Consumes output of processing pipeline

Data Sink

-Reads input data-Performs a transformation on the input data-Writes output data

Filter

-Transfers data-Buffers data-Synchronizes active neighbors

Pipe

Dynamic Behavior : Scenario I

• Active Source / Passive Filter / Passive Sink

DataSource

Filter FilterPipe Pipe Pipe Data Sink

Data Source Filter 1 Filter 2 Data Sink

Write(data)Transform(data)

Write(data)Transform(data)

Write(data)

Dynamic Behavior : Scenario II

• Passive Source / Passive Filter / Active Sink

DataSource

Filter FilterPipe Pipe Pipe Data Sink

Data Source Filter 1 Filter 2 Data Sink

Read

Read

Read

dataTransform(data)

data

data

Transform(data)

Dynamic Behavior : Scenario III

• Passive Source / Active Filter / Passive Sink

DataSource

Filter FilterPipe Pipe Pipe Data Sink

Data Source Filter 1 Filter 2 Data Sink

Read

Read

data

Transform(data)

data

Write(data)

Transform(data)

Dynamic Behavior : Scenario IV

• Passive Source / Multiple Active Filters / Passive Sink

DataSource

Filter FilterPipe Pipe Pipe Data Sink

Buffering PipeData Source Filter 1 Filter 2 Data Sink

Read

data

Transform(data)

Read

Write(data)

data

Write(data)

Read

dataTransform(data)

Write(data)

Read

dataTransform(data)

Write(data)

Transform(data)

Buffering Pipe

Implementation

• Divide the system into a sequence of processing stages

• Define the data format to be passed along each pipe

• Decide how to implement each pipe connection

– The simplest Pipe is direct Read/Write method calls between adjacent filters

– If buffering or synchronization is required between filters, actual Pipe objects will be needed to perform these functions

Implementation• Design and implement the filters

– Passive filters can be implemented as regular methods

– Active filters can be implemented as processes or threads

• Design the error handling

– In pipelines with one active component, standard error handling mechanisms can be used (exceptions, return codes, etc.)

– In pipelines with multiple active components, how will an error in one thread of control become visible to other threads?

– If an error occurs, we could:

• Tear down the entire pipeline and restart it from scratch, or

• Restart only the parts that failed and resynchronize the pipeline

• Setup the pipeline

Known Uses: UNIX Command Pipelines

cat fall.txt win.txt | sort | gzip | mail fred@byu.edu

cat sort gzip mail

fall.txt

win.txt

Known Uses: Image Processing

Known Uses: Compilers

Lexical Analyzer

Parser

Source File

ASCII Text

Token Stream

Semantic Analysis

Abstract Syntax Tree

Code Generator

Augmented Abstract Syntax Tree

Optimizer

Object Code

Object File

Optimized Object Code

Consequences• Very flexible

– Filters can be reused and recombined in arbitrary ways

– Individual filters can be easily replaced (e.g., plug in a different kind of compression)

• No intermediate files necessary between processing stages

• Benefits from efficiencies inherent in parallel processing (multiple active components)

• Some filters don’t produce any output until they've consumed all of their input (e.g., sort), which is not conducive to parallelism

• Data transfer and context switching between processes and threads can be expensive