eXtensible Data Cataloging System

XDCS

2

Problem

• The radar software group relies on capturing very high speed data of low-level digital signals for development.

• Examples:– GRA 5500 ADC capture: 4 GB per hour– GWX 70 ADC capture: 100 GB per hour– GWX 7000 ADC capture: > GWX 70

• Some of these captures take considerable effort to acquire.

3

What is XDCS?

• Efficient, scalable storage for very large files

• Consistent, well-defined retrieval of the contents of very large files

• Relate file contents between different sources and formats

• Document and annotate file contents

4

Why not network folders?

• File storage must scale in the range of 2 TB to 10 TB per year.

• It is difficult to maintain organization in a free-form file/folder structure.

• Finding, downloading, and re-parsing individual files larger than a gigabyte wastes time when all that’s needed is a few megabytes.

• No content-relevant searching.

5

Desired Features

• Abstract physical file storage to ease scaling and organization on the user

• Retrieve selectable segments of files• Stand-alone GUI for common activities• C and Python libraries allow streaming partial

files directly over the network• Web interface for browsing/retrieval (and

administration/monitoring)• File format translation, combining, and

filtering• Attach supporting files to capture sets• Offline parse/tag workflow

6

System Diagram

GUI(wxWidgets)

FeatureConsole

PythonConsole

XDCS: eXtensible Data Cataloging System

Real-timeCaptureModule

(optional)

ParserModule

(.py)

ParserModule

(.py)

EmbeddedPython

Interpretor

ParserModule(.dll/.so)

ParserModule(.dll/.so)

SQLiteParser

API

MySQLClient

Binary I/OClient

BinaryStorage

MySQLServer

Real-timeCaptureModule

(optional)

ChartPlotter

(OpenGL)

XDCSCore

PythonApplicationBindings

ExportModule

(.py)

ExportModule

(.py)

7

Typical Workflow

Project Setup(one-time)

Data Import(each capture file)

Document /Annotate

(as needed)

Browse / Search

Retrieve / ExportAcquire Data

Synchronize / Upload

Time passes, seasons

change, etc...

8

Typical Workflow

Project Setup(one-time)

Data Import(each capture file)

Document /Annotate

(as needed)

Browse / Search

Retrieve / ExportAcquire Data

Synchronize / Upload

Time passes, seasons

change, etc...

9

Project Setup Examples

10

Typical Workflow

Project Setup(one-time)

Data Import(each capture file)

Document /Annotate

(as needed)

Browse / Search

Retrieve / ExportAcquire Data

Synchronize / Upload

Time passes, seasons

change, etc...

11

GUI Demonstration

12

Export Modules

• Ability to plug in different export formats

13

Future Plans

• Web browse/search/retrieve interface• GUI polish/iteration

– Parametric search and visualization

• Project setup automation/verification• Portable library development• Import/export plug-in API• Server performance, scalability

analysis• Data acquisition front-end• Software/project update tools

14

Questions?