Advanced Topics in Databases

Fahime Raja

Niloofar Razavi

Melody Siadaty

Summer 2005

Technical Report II : Main memory Databases

Outline:

Introduction MMDB vs. DRDB

Storage Issues Reliability Issues Data Structures

Concurrency Control Commit Processing Data Representation Access Methods

T-tree definition & structure Query Processing Recovery

Introduction:

In a Main Memory DBMS (MMDBMS) there is no central role for I/O management.

Still, an important question in a MMDBMS is how to do transactions and recovery in an efficient way: Some of the proposed algorithms assume that a (small) stable subset

of the main memory exists, a piece of memory whose content will not be lost in a power outage through

a battery backup. These stable memories can be used to place, e.g., a redo log.

Others do not assume stable memories, and still use I/O to write transaction information to stable storage. These algorithms hence do not eliminate I/O, but minimize it, as the critical

path in a MMDBMS transaction only needs to write the log; not data pages from the buffer manager.

Introduction:

Motivations: Requirement of short Access/Response time Telecommunication applications Applications handling high traffic of data, e.g. Router. Real-Time applications

End of popularity of MMDBMS techniques in the early 1990s

MMDBMS were thereafter only considered of specific interest to real-time database applications, like above

Main Memory Databases Vs.Disk Resident Databases M : the primary copy of data live permanently in

main memory D: the primary copy of data is permanently disk

resident M: there can be a backup copy ,resident on disk

D: data can be temporarily cached in main memory for access speed up.

Main Memory Databases Vs.Disk Resident Databases Storage issues:

Access time of MM, orders of magnitude less than for disks (100 nsec vs. 10 msec)

MMDBMS footprints range between 200 KB and 2 MB MM is normally volatile

stable memory still expensive! Disks have high fixed dcost per access independent of the amount

of the retrieved data (Block Oriented) MM does not care of sequential access MM data are more vulnerable to software errors, since they can

be directly accessed by the processor.

Main Memory Databases Vs.Disk Resident Databases Reliability issues:

even if special hardware can enhance MM reliability, periodic backup is necessary

MM content is lost if system crashes If a single memory board fails, the entire machine must

be powered down loosing all data!

Whatever power backup for main memory is, in turn, is less reliable than passive magnetic media.

Main Memory Databases Vs.Disk Resident Databases Data Structures:

MMDB are not DRDB with a very large cache!

DRDB: Cached data are accessed through

indexes designed for disk access. Access is made through a buffer

manager ,which, given the disk address, checks if the relevant block is in the MM-Cache and then copies it to the MM application working area .

MMDB: data are accessed by directly

referring to their memory address. The following pictures illustrate this fact.

API for DRDB

API for MMDB

MMDBMS Concurrency Control:

Lock duration is short : reduced contention The advantage of small locking granules (fields or records) (when data

are memory resident) ,is removed, Very large lock granules (e.g., relations) are most appropriate (up to

the entire database) for memory resident data.

Serial transaction processing almost eliminate the need of concurrency control highly reduces cache flushes.

CC is still necessary when, Mixed length transactions coexist A multiprocessor system shares the DB among different units.

MMDBMS Concurrency Control:

Implementation : Traditional:

a hash table that contains entries for the objects currently locked . No lock information attached to data

MMDB: Stuff some small number of locking information to data. the first bit is set the object is locked, If it is locked and the second bit is set,

there are one or more waiting transaction else it is free.

MMDBMS Commit Processing necessary to have a backup copy and to keep a log of

transaction activity Before a transaction can commit, its activity records must

be written to the log. Logging can impact response time, since each transaction

must wait for at least one stable write before committing. Logging can also affect throughput if the log becomes a

bottleneck In MMDBs, the logging represents the only disk

operation each transaction will require.

MMDBMS Commit Processing Typical log length 400 bytes

40 bytes for Begin/End 360 bytes for Old/New values

stable the log tail (< 100 pages ) in a small amount of stable main memory Reduces response time Doesn’t affect bottlenecks

Pre-committing: releases transaction’s locks as soon as its log record is placed in

the log, without waiting for the information to be propagated to the disk.

Doesn’t affect serialization Because the log is sequential

MMDBMS Commit Processing

Doesn’t reduce response time Enhances concurrency

Response time of others.

Group committing: accumulates enough commit records to fill up a log

page and then flushes it to disk Reduces the total # of disk accesses can be used to relieve a log bottleneck .

MMDB Data Representation:

Relational data are usually represented as flat files Tuples are stored sequentially Attribute values are embedded in the tuples. Access is local

Space consuming due to duplicate values Inefficient due to sequentially of computations Need of indexes :

Relational tuples can be represented as a set of pointers to data values. use of pointers is space efficient when large values appear multiple

times in the database, (the actual value needs to only be stored once.)

Pointers also simplify the handling of variable length fields since variable length data can be represented using pointers into a heap.

MMDB Access Methods:

Hashing: Fast lookup & updating Not space efficient Doesn’t support range queries

Tree indexing: With a single pointer get access both to an attribute

value and the entire tuple Pointers are of fixed short lengths Suited for range queries

MMDB Access Methods:

The T-Tree : Is a data structure whose ancestors are B-Trees and AVL-trees It is binary like AVL-trees

search is essentially binary A T-node contains many elements like B-tree

storage & update are done efficiently Insertions and deletions usually move data between a single node

like in B-trees Rebalancing is done by node rotation

Like in AVL-trees, but much less frequent.

MMDB Access Methods:

T-Tree

MMDB Query Processing:

Query processing for DRDB focus on reducing disk access costs

Query processing for MMDB must focus on reducing processing costs Operation costs vary from system to system No general optimization technique

Implementation for relational operators should benefit of main memory data 7 index representation Nested loop-join is preferred to sort-merge join:

Although the sorted relations could be represented easily in a main memory database using pointer lists, there is really no need for this since much of the motivation for sorting is already lost .

MMDBMS Backup and Recovery: Backups of memory resident databases must be

maintained on disk or other stable storage to insure against loss of the volatile data .

1.the procedure used during normal database operation to keep the backup up-to-date (Checkpointing),

2.the procedure used to recover from a failure. (Failure Recovery) loading blocks of the database “on demand” until all of the

data has been loaded. using disk striping or disk array

there must be independent paths from the disks to memory

MMDBMS Backup and Recovery:3 major steps in a recovery process:1. Logging log information can be recorded at two different levels of

abstraction, physical logging: records, for each update, the state of the modified database

and the physical location (such as page id and offset) to which the update is applied.

logical logging: records the state transition of the database and the logical location affected by the update.

The Write Ahead Logging (WAL) is the most popular and accepted type of logging rules

MMDBMS Backup and Recovery:2.Checkpointing MMDB checkpointing approaches fall into three

categories: non-fuzzy checkpoint

the checkpointer first obtains locks on the data items to be checkpointed so that consistency of the database is preserved

fuzzy checkpoint the checkpointer does not secure any locks. To synchronize with

normal transactions, the checkpointer writes a record to the log after a checkpoint is completed. In addition, the database is generally dumped in segments.

log-driven checkpoint the checkpointer applies the log to a previous dump to generate a

new dump rather than dumping from the main-memory database.

MMDBMS Backup and Recovery:3. Reloading is the last and the one of the most important parts of the recovery

process. Reloading must ensure that the database is consistent after a crash. The database is loaded from the last checkpointed image and the undo

and redo operations are applied to restore it to the most recent consistent state.

The existing reloading schemes can be divided into two classes: simple reloading

the system does not resume its normal operation until the entire database is brought back into the main memory

concurrent reloading reload activities and transaction processing are performed in parallel.