Hyperion :High Volume Stream Archival

Divya Muthukumaran

Area

Network Monitoring Identify problems due to overloaded and/or

crashed servers, network connections or other devices

Example: To determine the status of a webserver, monitoring software may periodically send an HTTP request to fetch a page

Live Monitoring

Packets are examined in real time Compute and continually update traffic

statistics Discard the captured packet headers once

examined Why the need to store packet headers?

Live Monitoring Packets are examined in real time

Compute and continually update traffic statistics Discard the captured packet headers once examined

Why the need to store packet headers? Example: Network forensics

To go back and examine the root cause of a problem Ex: See how an intruder gained entry, How a worm

infection happened

What is the need of such a system?

Querying and examining live data Data Archival

Capture the data at wire speeds, Index and store them

Efficiently support retrieval and processing of archived data

Specifically designed to handle needs of high volume stream archival

Why not traditional databases?

Some statistics A single GB link can generate over 100,000

packets and tens of MBs of archival data. A monitor may record from Multiple links.

Design Principles

Support Queries not reads Implies the need to maintain indexes

Writes Sequential and Immutable

Archive locally , summarize globally Scalability Vs Need to avoid flooding

Scalability: Favors local archiving and indexing to avoid network writes

Need to answer Distributed queries: favors sharing information across nodes

Hyperion

Three Key components Stream File System

High volume archiving and querying Multi-level index structure

High update rates + reasonable lookup performance

Distributed index layer Distributes a summary of local indices to

enable distributed querying

Design choices for the Hyperion Storage System Storage of multiple high-speed traffic streams

without loss Support for concurrent read activity without

loss of write performance Re-use of storage in a buffer-like fashion

Stream File System

Stores Streams as opposed to files Characteristics

Recycled : When storage is full new data replaces old data.

In a GP File system new data is lost old is retained

Immutable Record-oriented: data is written in fixed or

variable length records

Can we use a GP FS?

Need to map streams <=>files

LogFile Rotation

Stream FS

Stream FS Organization

Los-structured FS What problem?

Cleaning/Garbage collection StreamFS solves the cleaning problem

Guarantee : Storage guarantee for each stream

Small segment size Check if next segment is a surplus . If yes then

overwrite , otherwise skip.

Stream FS Organization Los-structured FS What problem? Cleaning/Garbage collection StreamFS solves the cleaning problem

Guarantee : Storage guarantee for each stream Small segment size (1 or ½ MB)

Check if next segment is a surplus . If yes then overwrite , otherwise skip.

Advantages? Storage Reservation Best effort use of remaining storage

Reads

First get index Use index to get data Persistent Handles

Returned from each write operation Passed to read op to retrieve data What does the handle contain?

Disk location , approximate length Allows data to be retrieved directly

Handle issues

Validate the handle. How? Self certifying record header

Id of the stream Permissions of the stream Record length Hash (used for validating the handle)

Stream FS Organization Record

Variable length On-disk record + header

Block Fixed length Multiple records of the same stream

Block Map Every nth block (stream ID + in-stream sequence number for each of

the preceding n-1 blocks) Used for easy write allocation

Stream FS Organization

Indexing

Uses signature based Indices Signature for each segment

Can check if a record with a key k is present in the segment or not

Does not tell you where the record is present in the segment

Multi-level Indices

Multi Level Indices

Uses a Bloom Filter Hash (key) -> b bits In b bits k bits are set to 1

H(key1)||H(key2)…||H(keyn) = Hs (Signature) How to check for presence of a record?

Compute hash of its key kr, H(kr) If a bit in H(kr) is set but not set in Hs then the

value is not present False positives

Distributed Index How to handle distributed queries without flooding?

Maintain distributed index Integrated view of all nodes Coarse-grain summary of data at each node is needed

Can use the top level index in the Hyperion One index node per time interval

All nodes send their top-level indices to this node Temporally–distributed index