Linux Technology Center

18 April 2003

© 2003 IBM

LDAP Content Synchronization

Kurt D. Zeilenga Jong Hyuk ChoiOpenLDAP Project IBM Research

LDAP Content Synchronization2

Agenda

What is a directory?What is X.500?

What is LDAP?

What is the OpenLDAP Project?

LDAP Content Synchronization

Questions and Answers

LDAP Content Synchronization3

What is a Directory?

A database organized and/or optimized for lookup, searching, browsing, and other “read” activities.

Examples:address books

web catalogs

whois(1)

dict(1)

operating system / network information services

Domain Name System

LDAP Content Synchronization4

What is X.500?

The set of ITU standards covering electronic directory services.

X.500 defines the OSI Directory ServiceHierarchical

Distributed

Replicated

Schema driven

LDAP Content Synchronization5

What is LDAP?

Lightweight Directory Access Protocolan IETF Proposed Standard

A protocol for accessing X.500 directory services over the Internet (or networks using Internet Protocols).

LDAP Content Synchronization6

What is OpenLDAP?

OpenLDAP Software“community developed LDAP software”

OpenLDAP Project

“a collaborative effort to develop a robust, commercial-grade, fully featured, and open source LDAP suite of applications and development tools”

OpenLDAP FoundationProvides a legal umbrella for the OpenLDAP Project

http://www.openldap.org

LDAP Content Synchronization7

LDAP Overview

Request

Response

The DirectoryUser

LDAP Content Synchronization8

Polling for content changes

Request

The DirectoryUser

Response

X

LDAP Content Synchronization9

Problems with simple “polling”

Cannot reliably detect whether the entry named by a DN is the same entry previously named by that DN. DNs are not stable identifiers!

A complete copy of each entry is transferred with each poll

Changes not apparent until next poll

LDAP Content Synchronization10

Stable Identifiers

Add an Universally Unique Identifier (UUID) to each entry.Introduce entryUUID operational attribute

Add knowledge of superior’s UUID for each entry.Introduce parentUUID operational attribute

LDAP Content Synchronization11

Reducing traffic

Each entry in LDAP has create and modify time stamps(&(FILTER)(|(createTimeStamp>=TIME)(modifyTimeStamp>=TIME))

where

FILTER is our original assertion and

TIME is the greatest seen in previous content.

All entries created/modified before TIME are not transferred.

Does not detect deleted (or modified ‘out of scope’) entries.

Does not detect unchanged entries now ‘in scope’

Search operations are not necessarily isolated from directory modifications. One cannot assume all updates made before TIME appeared in the previous content.

LDAP Content Synchronization12

We need to extend LDAP to support a lightweight content synchronization mechanism!

LDAP Content Synchronization13

Target Applications

Heterogeneous Data Store Synchronization“meta directory” applications

Local content shadowingDirectory-enabled services (web, email, etc.)

Mobile clients

LDAP ReplicationSingle master

Partial (Sparse and/or Fractional) Replication

LDAP Content Synchronization14

Design Goals

Less chatty than “simple polling”

Maintain consistencyEventually convergent (inconsistencies are transient)

Support “polling” for changes Support “listening” for changes

Support partial synchronization

Do not require pre-arranged synchronization agreements Do not require server to maintain per client state information Do not require server to maintain histories of changes

Lightweight / Simple

LDAP Content Synchronization15

Non-goals

Support bi-directional synchronization

Support very small clients (e.g., PDAs)

Attribute (or finer) granularity

LDAP Content Synchronization16

Synchronization Overview

Old New

Changed Content

Renamed Content

Added Content

Deleted Content

Unchanged Content

LDAP Content Synchronization17

Basic “Refresh Mode” Protocol Design

Send Entries

Initial Content Request

Done with Cookie

Send “Changed” Entries

Content Refresh Request

Done with Cookie

Repeat

Send “Present” Messages

LDAP Content Synchronization18

First things first…

Stable object identifiersLDAP Distinguished Names are not stable

Solution: add entryUUID and parentUUID attributes to every entry

Time stampsLDAP create/modify time stamps are generally good enough, but

semantics are unclear

Solution: add change sequence numbers (CSNs) to every entry, well define the semantics

LDAP Content Synchronization19

The Cookie

Synchronization State Indicator

Greatest committed change sequence number (CSN) in context which is not greater than any outstanding CSN

e.g., Change Sequence Numbers != Commit Sequence Numbers

LDAP Content Synchronization20

Bandwidth

0

10

20

30

40

50

60

70

80

90

100

10K/1% 10K/10% 10K/50% 1K/1% 1K/10% 1K/50%

Poll

Updates

Presents

Refresh

LDAP Content Synchronization21

Refresh: updates+present v. updates+deletes

If server has no histories, it cannot determine what to “delete”

If server maintains “tombstone” information for deleted entries but no content change history, it must send deletes for all new tombstones as well as all out-of-scope updated entries.

If the server can reliably determine which entries previously present in the shadow copy which are no longer present in the content and the number of such entries is less than equal the number of unchanged entries, the server may send “deletes” instead of “present” messages.

LDAP Content Synchronization22

OpenLDAP Implementation

No naming/change history

Each entry has:entryUUID/parentUUID

entryCSN - updated on entry modify

namingCSN - updated on entry (not subtree) rename

Context has:Outstanding CSNs - ordered list

Outstanding Naming CSNs – ordered list

Allows server to detect some “no deletes” cases.

LDAP Content Synchronization23

“Refresh & Persist Mode” Protocol Design

Send Entries

Content Request

Refresh Done

Send “Changed” Entries and/or “Deleted” messages

Cancel

Done with Cookie / Acknowledge Cancel

New “Cookie”

Repeat

LDAP Content Synchronization24

Special Cases

Operational AttributesMust either be treated as part of the entry’s content if transferred.

structuralObjectClass

hasSubordinates

subschemaSubentry Alias and Referral objects

No dereferencing

Treat as normal object Collective Attributes

Not treated as content of entries of the collection

Client must ‘watch’ the collective attributes subentries Access and other administrative controls

Server must either generate appropriate changes to sync client

OR force reload

LDAP Content Synchronization26

Summary

The LDAP Content Synchronization Operation appears to met our needs:

Supports both “Polling” and “Listening” modes

“Polling” mode is less chatty than “simple polling”

“Listening” mode is significantly less chatty than “simple polling”

Eventual convergence is assured, inconsistencies are transient.

No pre-arranged synchronization agreements are needed.

Server need not maintain histories or per-client state information.

Able to support our target applications:

Meta Directory

Content caching

LDAP Replication

LDAP Content Synchronization27

Availability

IETF Technical Specification:draft-zeilenga-ldup-sync-xx.txt

draft-zeilenga-ldup-uuid-csn-xx.txt (available soon)

Last Call this summer (hopefully)

OpenLDAP Implementation:LDAP-sync client and server implementations committed to HEAD

First-cut LDAP Replication engine committed to HEAD

Release target: OpenLDAP 2.2 in Summer of 2003

LDAP Content Synchronization28

Questions?