A Streaming Video Server for areaDetector

Tom Cobb (Tom.Cobb@diamond.ac.uk)

Presented by Nick Rees

Introduction• What is areaDetector?

– Supported detectors

• Example processing chain• Why do we need a streaming video server?• ffmpegServer components

– Performance

• The future…

What is areaDetector?• An EPICS module for controlling area (2-D)

detectors, including CCDs, pixel array detectors, and online imaging plates.

• Architecture– Plugin based system– Plugins can be configured to create processing chains– Configurable at runtime– Plugins can be used on many detectors– Processing can be spread over multiple threads– Consistent data format (NDArray)– NDArray objects passed between plugins

Supported Detectors• Firewire (IEEE 1394)• Prosilica GigE cameras• Rayonix (Mar USA) CCD• Marresearch GmbH 345• Dectris Pilatus• ADSC• Roper CCD detectors• PCO imaging pcocam• Thales Pixium flat panel detectors• Perkin Elmer flat panel detectors• Bruker SMART CCD• Generic GigE vision cameras (prototype)• XIA DXP-XMAP/Jupiter/Saturn

Example processing chain

• Camera Driver• Processing

– Background subtraction– Flat field correction– Scale and offset– Recursive average

• Statistics:– Histograms– Centroid calculation– Min and max values

• Overlay– Crosses or rectangles– Selectable colours

• MJPG web server

CAM

PROC OV

STAT MJPG

Example processing chain

CAM

PROC OV

STAT MJPG

Why do we need a streaming video server?

• Scalability

Server:

CPU: 32%

Rx: 15MB/s

Tx: 15MB/s

Each client:

CPU: 15%

Rx: 15MB/s

Camera:

1024x768

Mono

20 fps

Server:

CPU: 86%

Rx: 15MB/s

Tx: 60MB/s

Why do we need a streaming video server?

• Scalability

Server:

CPU: 61%

Rx: 15MB/s

Tx: 8MB/s

Each client:

CPU: 29%

Rx: 2MB/s

Camera:

1024x768

Mono

20 fps

ffmpegServer components• FFmpeg

– A complete, cross-platform solution to record, convert and stream audio and video.

• Nullhttpd– A very small, simple and multithreaded web server for

Linux and Windows.

• ffmpegStream– areaDetector plugin that compresses NDArrays using

the MJPG codec from FFmpeg and serves them up over TCP using Nullhttpd

• ffmpegFile– areaDetector plugin that compresses NDArrays using

any codec Ffmpeg supports and writes them to file

Performance• Tests with prosilica driver and ffmpegServer compression

– 3 cameras, 1024x768 Mono @ 30fps, 105% of one core CPU usage

– 3 cameras, 1024x768 Bayer -> RGB @ 30fps, 212% of one core CPU usage

– Raw data from cameras -> server = 70MB/s

– No dropped frames

• Same raw data rate from more cameras doesn't seem to work– 4 cameras, higher res @ 10fps, 67%, 55MB/s raw data

– 5 cameras, 1024*768 @ 10fps, 42%, 39MB/s raw data

– Increasing the frame rates or res produces dropped frames

• Adding more cameras with lower res more successful– 8 cameras, 400x300 @ 70-100 fps, 87%, 81MB/s

Manufacturers suggestions• Use Intel Pro 1000 GT/PT/CT NIC.• Enable JumboFrames on NIC. PacketSize should

= 8228.• Run as root• PvAPI 1.24 with filter 1.20 recommended.• Use a dedicated LAN/NIC/Switch for the

camera(s). If using multiple cameras, on a single port NIC, be sure the sum of StreamBytesPerSecond on all cameras doesn't exceed 120,000,000.

The future…• Prosilica driver

– Replace with open source reverse engineered generic GigE Vision driver: aravis

– Prototype of areaDetector driver complete, camera producing frames

– Need to understand the GeniCam XML of each vendor camera

• ffmpegServer– Use ffserver instead of the nullhttpd– Not cross platform– Supports multicast– Support for MPEG4 over rtsp link

Other developments• Gaussian fitting

– x, y, base, height, minor axis, major axis, angle.– Uses MPFIT, so easily extensible to any other defined

function

• Gaussian simulator– Gaussian noise

• Pixel noise and on function parameters

• Function parameters have optional IIR low-pass filter.

• For further info, contact James O’Hea– James.Ohea@diamond.ac.uk

EPICS Base Automatic Testing

Jon Thompson (Jon.Thompson@diamond.ac.uk)

Presented by Nick Rees

Introduction• An effort was started at the last codeathon to

automate the testing of EPICS base. This has continued since then at Diamond. There are two threads:– Convert the ‘soft tests’ to the Diamond testing

framework.– Extend the Diamond framework to support the ‘make

runtests’ and present results as a web page.

Results• The results are available at

http://controls.diamond.ac.uk/downloads/other/epics-base-report/

• Tests are run regularly on the 3.14 branch of the launchpad repository.

• Targets so far supported are:– Linux-x86 (host): ‘run tests’ and ‘soft tests’– vxWorks-ppc604_long: ‘run tests’ only– RTEMS-mvme5500: ‘run tests’ only

Code Coverage

• A statement based, code coverage report is also available. This indicates the coverage obtained during the linux-x86 host execution of ‘make runtests’ and ‘soft tests’.– Uncovered lines are indicated with a ‘#####’

and are highlighted.– Lines marked ‘-’ are not a complete statement

(continuation lines, declarations, etc.).– A decimal number indicates the number of

times the statement was executed.

Coverage Pitfalls• Be aware of the shortcomings of statement

based, code coverage analysis.– Just because it was executed does not mean it’s

correct. That’s the job of the test case.– Code can be structured in such a way that it hides

untested code. Single line ‘if’ statements are a big culprit here.

– Insensitive to the ‘short-cut’ features of ‘&&’ and ‘||’ operators.

Further Developments• More targets.

– A Windows target will be tackled next.

• Extending ‘soft tests’ to the embedded targets.• Cross target testing, especially between targets

with differing endianity.

Any questions?