SiLo: A Similarity-Locality based Near-Exact Deduplication Scheme withLow RAM Overhead and High Throughput

Wen Xia † Hong Jiang ‡ Dan Feng † � Yu Hua †,‡

wx.hust@gmail.com jiang@cse.unl.edu dfeng@hust.edu.cn csyhua@hust.edu.cn

† School of Computer, Huazhong University of Science and Technology, Wuhan, ChinaWuhan National Lab for Optoelectronics, Wuhan, China

‡Dept. of Computer Science and Engineering, University of Nebraska-Lincoln, Lincoln, NE, USA

Abstract

Data Deduplication is becoming increasingly popular instorage systems as a space-efficient approach to databackup and archiving. Most existing state-of-the-artdeduplication methods are either locality based or sim-ilarity based, which, according to our analysis, do notwork adequately in many situations. While the formerproduces poor deduplication throughput when there islittle or no locality in datasets, the latter can fail to iden-tify and thus remove significant amounts of redundantdata when there is a lack of similarity among files. In thispaper, we present SiLo, a near-exact deduplication sys-tem that effectively and complementarily exploits sim-ilarity and locality to achieve high duplicate elimina-tion and throughput at extremely low RAM overheads.The main idea behind SiLo is to expose and exploitmore similarity by grouping strongly correlated smallfiles into a segment and segmenting large files, and toleverage locality in the backup stream by grouping con-tiguous segments into blocks to capture similar and du-plicate data missed by the probabilistic similarity detec-tion. By judiciously enhancing similarity through the ex-ploitation of locality and vice versa, the SiLo approachis able to significantly reduce RAM usage for index-lookup and maintain a very high deduplication through-put. Our experimental evaluation of SiLo based on real-world datasets shows that the SiLo system consistentlyand significantly outperforms two existing state-of-the-art system, one based on similarity and the other basedon locality, under various workload conditions.

1 Introduction

As the amount of the important data that needs to bedigitally stored grows explosively to a worldwide stor-age crisis, data deduplication, a space-efficient method,has gained increasing attention and popularity in datastorage. It splits files into multiple chunks that are

each uniquely identified by a 20-byte SHA-1 hash sig-nature, also called a fingerprint [21]. It removes dupli-cate chunks by checking their fingerprints, which avoidsa byte-by-byte comparison. Data deduplication not onlyreduces the storage space overheads, but also minimizesthe network transmission of redundant data in the net-work storage system [19].

One of the main challenges for centralized backupservices based on deduplication is the scalability offingerprint-index search. For example, to backup adataset of 800TB and assuming an average chunk sizeof 8KB, at least 2TB of fingerprints have to be generated,which will be too large to be stored in the memory. Sincethe access to on-disk index is at least 1000 times slowerthan that to RAM, the frequent accesses to on-disk fin-gerprints are not acceptable for backup services and havebecome the main performance bottleneck of such dedu-plication systems.

Most of the existing solutions aim to make the fulluse of RAM, by putting the hot fingerprints into RAMto minimize accesses to on-disk index and improve thethroughput of deduplication. There are two primary ap-proaches to scaling data deduplication: locality basedacceleration of deduplication, and similarity based dedu-plication. Locality-based approaches exploit the inherentlocality in a backup stream, which is widely used in state-of-the-art deduplication systems such as DDFS [26] andChunkStash [8]. Locality in this context means that thechunks of a backup stream will appear in approximatelythe same order in each full backup with a high proba-bility. Exploitation of this locality increases the RAMutilization and reduces the accesses to on-disk index,thus alleviating the disk bottleneck. Similarity-based ap-proaches are designed to address the problem encoun-tered by locality-based approaches in backup streamsthat either lack or have very weak locality (e.g., incre-mental backups). They exploit data similarity insteadof locality in a backup stream, and reduce the RAM us-age by extracting similar characteristics from the backup

stream. A well-known similarity-based approach is Ex-treme Binning [3] that exploits the file similarity toachieve a single on-disk index access for chunk lookupper file.

While these scaling approaches have significantly alle-viated the disk bottleneck in data deduplication, there arestill substantial limitations that prevent them from reach-ing the peta- or exa-scale, as explained below. Basedon our analysis of experimental results, we find thatin general a locality-based deduplication approach per-forms very poorly when the backup stream lacks localitywhile a similarity-based approach underperforms for abackup stream with a weak similarity. Unfortunately, thebackup data in practice are quite complicated in how orwhether locality/similarity is exhibited. In fact, DDFS isshown to run very slowly in backup streams with littleor no locality (e.g., when users only do the incremen-tal backup). On the other hand, the similarity-based Ex-treme Binning approach is shown to fail to find signifi-cant amount of duplicate data in datasets with little or nofile similarity (e.g., when the files are edited frequently).Fortunately, our preliminary study indicates that the ju-dicious exploitation of locality can compensate for thelack of similarity in datasets, and vice versa. In otherwords, both locality and similarity can be complemen-tary to each other, and can be jointly exploited to improvethe overall performance of deduplication.

To this end, we propose SiLo, a scalable and low-overhead near-exact deduplication system, to overcomethe aforementioned shortcomings of existing state-of-the-art schemes. The main idea of SiLo is to considerboth similarity and locality in the backup stream simul-taneously. Specifically, we expose and exploit more sim-ilarity by grouping strongly correlated small files into asegment and segmenting large files, and leverage localityin the backup stream by grouping contiguous segmentsinto blocks to capture similar and duplicate data missedby the probabilistic similarity detection. The main con-tributions of this paper include:

• SiLo proposes a new similarity algorithm thatgroups many small strongly-correlated files into asegment or segments a large file to better expose andexploit their similarity characteristics. This group-ing of small files results in much smaller similar-ity index for segments than chunk index, which caneasily fit into RAM for a much larger dataset. Thesegmenting of large files can expose and thus ex-tract more similarity characteristics so as to removeduplicate data with a higher probability.

• SiLo proposes an effective approach to mining thelocality characteristics to capture similar and dupli-cate data missed by the probabilistic similarity de-tection by grouping multiple contiguous segments

into a block, the basic cache and write-buffer unit,while preserving the spatial locality inherent in thebackup stream on the disk. By keeping the similar-ity index and preserving spatial locality of backupstreams in RAM (i.e., hash table and locality cache),SiLo is able to remove large amounts of redun-dant data, dramatically reduce the numbers of ac-cesses to on-disk index, and substantially increasethe RAM utilization.

• Our experimental evaluation of SiLo, based on real-world datasets, shows that the SiLo system con-sistently and significantly outperforms two exist-ing state-of-the-art systems, the similarity-basedExtreme Binning system and the locality-basedChunkStash system, under various workload con-ditions. According to our evaluations on du-plicate elimination, SiLo can remove 1%∼28%more redundant data than Extreme Binning andonly 0.1%∼1% less than the exact-deduplicatingChunkStash. Our evaluations on deduplicationthroughput (MB/sec) suggest that SiLo outperformsChunkStash by a factor of about 3 and Extreme Bin-ning by a factor of about 1.5. On the RAM utiliza-tion for the same datasets, SiLo consumes a RAMcapacity that is only 1/41∼1/60 and 1/3∼1/90 re-spectively of that consumed by ChunkStash and Ex-treme Binning.

The rest of the paper is organized as follow. Section2 presents background and motivation for this research.Section 3 describes the architecture and the design of theSiLo system. Section 4 presents our experimental eval-uation of SiLo and discusses the results, including thecomparisons with the state-of-the-art ChunkStash andExtreme Binning systems. Section 5 gives an overviewof related work, and Section 6 draws conclusions andoutlines future work.

2 Background and Motivation

In this section, we first provide the necessary backgroundfor our SiLo research by introducing the existing acceler-ation approaches for data deduplication, and then moti-vate our research by analyzing our observations based onextensive experiments on locality- and similarity-baseddeduplication acceleration approaches under real-wordworkloads.

2.1 Deduplication Acceleration Ap-proaches

Previous studies have shown that the main challenge fac-ing data deduplication lies in the on-disk index-lookup

2

bottleneck [26, 15, 8]. As the size of dataset to be dedu-plicated increases, so does the total size of fingerprintsrequired to detect duplicate chunks, which can quicklyoverflow the RAM capacity for even high TB-scale andlow PB-scale datasets. This can result in frequent diskaccesses for fingerprint-index lookups, thus severely lim-iting the throughput of the deduplication system. Cur-rently, there are two general approaches to accelerat-ing the index-lookup of deduplication and alleviating thedisk bottleneck, namely, the locality based and the simi-larity based methods.

The locality in the former refers to the observationthat files, say A and B (thus their data chunks), in abackup stream appear in approximately the same orderthroughout multiple full backups with a high probability.DDFS [26] makes full use of this locality characteristicby storing the chunks in the order of the backup streamon the disk and preserving the locality in the RAM. Itsignificantly reduces accesses to the on-disk index by in-creasing the hit ratio in the RAM. It also uses Bloomfilters to quickly identify new (non-duplicate) chunks,which helps compensate for the cases where there isno or little locality, but at the cost of significant RAMoverhead. Sparse Indexing [15] improves this methodby sampling index instead of using Bloom filters. Ituses less than half of the RAM capacity of DDFS. As anovel content-defined chunking algorithm, Bimodal [14]suggests that the neighboring data of duplicate chunksshould be assumed to be good deduplication candidatesdue to backup-stream locality, which can be exploited tomaximize the chunk size.

Nevertheless, all these approaches still produce unac-ceptable performance in face of very large datasets withlittle or no locality. The similarity-based approaches areproposed to exploit the similar characteristics in backupstreams to minimize the chunk-lookup index in the mem-ory. For example, Extreme Binning [3] exploits the simi-larity among files instead of locality, allowing it to makeonly one disk access for chunk lookup per file. It sig-nificantly reduces the RAM usage by storing only thesimilarity-based index in the memory. But it often failsto find significant amounts of redundant data when simi-larity among files is either lacking or weak. It puts sim-ilar files in a bin whose size grows with the size of thedata, resulting in decreased throughput as the size of thesimilarity bin increases.

2.2 Small Files and Large Files

Our experimental observations, as well as intuition, sug-gest that the deduplication of small files can be veryspace and time consuming. A file system typically con-tains a very large number of small files [1]. Since thesmall files (e.g., ≤64KB) usually only take up a small

fraction (e.g., ≤20%) of the total space of a file systembut account for a large percentage (e.g., ≥80%) of thenumber of files, the chunk-lookup index for small fileswill be disproportionally large and likely out of mem-ory. Consequently, the inline deduplication [26, 15] ofsmall files will tend to be very slow and inefficient be-cause of the more frequent accesses to the on-disk indexfor chunk lookup and the higher network-protocol costsbetween the client and the server.

This problem of small files can be addressed by group-ing many highly correlated small files into a segment. Weconsider files with the logic sequence within the sameparent directory to be highly correlated and thus simi-lar. We exploit the similarity and the locality of a group(i.e., segment) of adjacent small files rather than one in-dividual file or chunk. As a result, at most one access toon-disk index is needed per segment instead of per fileor per chunk. The segmenting approach can also min-imize the network costs by avoiding the frequent inlineinteractions per file.

A typical file system also contains many large files(e.g., ≥2MB) that only account for a small fraction (e.g.,≤20%) of total number of files but occupy a very largepercentage (e.g., ≥ 80%) of the total space [1]. Ob-viously, these large files are an important considera-tion for a deduplication system due to their high space-capacity and bandwidth/time requirements in the backupprocess. When a large file is being deduplicated inline,the server must often wait for a long time for the chunk-ing and hashing processes, resulting in low efficiency ofthe deduplication pipeline. In addition, the larger thefiles, the less similar they will appear to be even if sig-nificant parts within the files may be similar or identical,which can cause the similarity-based approaches to missthe identification of significant redundant data in largefiles.

To address this problem of large files, our SiLo ap-proach divides a large file into many small segments tobetter expose similarity among large files while increas-ing the efficiency of the deduplication pipeline. Morespecifically, the probability that file S1 and file S2 sharethe same representative fingerprint is highly dependenton their similarity degree according to Broder’s theorem[5]:

Theorem 1: Consider two sets S1 and S2, with H(S1)and H(S2) being the corresponding sets of the hashesof the elements of S1 and S2 respectively, where H ischosen uniformly and randomly from a min-wise inde-pendent family of permutations. Let min(S) denote thesmallest element of the set of integers S. Then:

Pr[min(H(S1) = min(H(S2)))] =|S1 ∩ S2||S1 ∪ S2|

3

This probability can be increased by segmenting thefiles and detecting all the segments of the file, as follows:

Pr[min(H(S1) = min(H(S2)))] =|S1 ∩ S2||S1 ∪ S2|

≪

Pr[min(H(S11) = min(H(S21)))] ∪ · · · ∪ Pr[min(H(S1n) = min(H(S2n)))]

=

n∪i=1

Pr[min(H(S1i) = min(H(S2i)))]

= 1 −n∩

i=1

Pr[min(H(S1i) ̸= min(H(S2i)))]

= 1 −n∏

i=1

(1 −|S1i ∩ S2i||S1i ∪ S2i|

)

As files S1 and S2 are segmented into S11 ∼ S1n andS21 ∼ S2n respectively, the detection of similarity be-tween S1 and S2 is determined by the union of the prob-abilities of detections of similarity between S11 ∼ S1n

and S21 ∼ S2n. Based on the above probability analysis,this segmenting approach will only fail in the worst-casescenario where all the segments in file S1 are not similarto segments of file S2. This, based on the inherent lo-cality in the backup streams, happens with a very smallprobability because it is extremely unlikely that two filesare very similar but none or very few of their respectivesegments are detected as being similar.

2.3 Similarity and LocalityNow we further analyze the relationship between similar-ity and locality with respect to backup streams. As men-tioned earlier, chunk locality can be exploited to storeand prefetch groups of contiguous chunks that are likelyto be accessed together with a high probability in thebackup stream, while files’ similarity may be mined sothat the similarity characteristics instead of the wholesets of fingerprints of files, are indexed to minimize theindex size in the memory. The exploitation of localitymaximizes the RAM utilization to improve the through-put but can cause RAM overflows and frequent accessesto on-disk index when datasets lack or are weak in local-ity.

The similarity-based approaches minimize the RAMusage at the cost of potentially missing large amountsof redundant datawhich is dependent on the similar-ity degree of the backup stream. We have exam-ined the similarity degree and the duplicate-eliminationmeasure of our similarity-only deduplication approachon four datasets, as shown in Figure 1 and Figure 2.The four datasets represent one-backups, incremental-backups, Linux-versions and full-backups respectively,whose characteristics will be detailed in Section 4.

The similarity degree is computed by our similaritydetection on the Linux dataset as: Simi(Sinput) = Max(| Sinput ∩ Si |)/|Sinput|,(Si ∈ Sstore, Simi(Sinput) ∈[0,1]). Thus, the similarity degree “1” signifies that the

One-set Inc-set Linux-set Full-set0%

20%

40%

60%

80%

100%

The

sim

ilarit

y di

strib

utio

ns

The datasets

1 [0.9,1.0) [0.8,0.9) [0.7,0.8) [0.6,0.7) [0.5,0.6) [0.4,0.5) [0.3,0.4) [0.2,0.3) [0.1,0.2) (0,0.1) 0

Figure 1: The distribution of the segment similarity onfour datasets by our similarity-only approach. It can beused to describe the similarity characteristics of datasets.A large proportion of data with low similarity degrees isobserved here.

One-set Inc-set Linux-set Full-set0%

20%

40%

60%

80%

100%

% o

f dup

licat

e da

ta e

limin

ated

The datasets

Figure 2: Percentage of duplicate data eliminated by oursimilarity-only deduplication approach.

input segment is completely duplicate and the similaritydegree “0” states that the segment is detected to matchno other segments at all by our similarity-only approach.

Figure 3 further examines the duplicate eliminationmissed by the similarity approach on the Linux dataset.The missed portion of duplication elimination is de-fined as the difference between the measure achievedby the exact deduplication and that by the similarity-based deduplication. Therefore, Figure 3 shows that thesimilarity-based deduplication efficiency is heavily de-pendent on the similarity degree of the backup streamwhich is well consistent with Broder’s Theorem in (seeSection 2.2). The similarity approach often fails to re-move large amounts of duplicate data, especially whenthe backup stream has a low similarity degree.

Inspired by Bimodal [14], which shows that thebackup-stream locality can be mined to find more po-tentially duplicate data, we believe that such locality canalso be mined to expose and thus detect more data sim-ilarity, a point well demonstrated by our experimentalstudy in Section 4. More specifically, SiLo mines lo-

4

0

(0-0.1)

[0.1-0

.2)

[0.2-0

.3)

[0.3-0

.4)

[0.4-0

.5)

[0.5-0

.6)

[0.6-0

.7)

[0.7-0

.8)

[0.8-0

.9)

[0.9-1

.0) 10%

20%

40%

60%

80%

100%%

of d

uplic

ate

data

elim

inat

ed

The degree of segment similarity

Missed Similarity

Figure 3: Percentage of duplicate data eliminated asa function of different similarity degree on the Linuxdataset by our similarity-only deduplication approach.

cality in conjunction with similarity by grouping multi-ple contiguous segments in a backup stream into a block.While this exploitation of locality helps find more po-tential deduplication candidates by detecting similar seg-ments’ adjacent segments in a block, it also reduces theaccesses to on-disk index to improve the deduplicationthroughput.

Now we analyze the combined exploitation of simi-larity and locality. Given two blocks B1 and B2, eachcontaining n segments (S11 ∼ S1n, S21 ∼ S2n), accord-ing to the Broder’s theorem, the percentage of duplicateeliminated by the similarity-only approach can be com-puted as: DeDupSimi(B1, B2) = |B1∩B2| / |B1∪B2|.The combined and complementary exploitation of simi-larity and locality can be computed as follows:

DeDupSiLo(B1, B2)

=

n∪i=1

Pr[min(H(S1i) = min(H(S2i)))]

= 1 −n∩

i=1

Pr[min(H(S1i) ̸= min(H(S2i)))]

= 1 −n∏

i=1

(1 −|S1i ∩ S2i||S1i ∪ S2i|

)

= 1 − (1 − a)N(assume all the

|S1i ∩ S2i||S1i ∪ S2i|

= a)

Assume that the value a follows a uniform distribu-tion in the range [0,1] (It may be much more compli-cated in the real world datasets), the expected value ofduplicate elimination can be further calculated under theaforementioned assumption as:

ESimi =

∫ 1

0

(a)da =1

2

ESiLo =

∫ 1

0

(1− (1− a)N )da =N

N + 1

Thus the larger the value N (i.e., the number of seg-ments in a block), the more locality can be exploited

in deduplication. ESimi is equal to ESiLo when N=1.SiLo can remove more than 99% of duplicate data whenN > 99. Thus the combined exploitation of similar-ity and locality makes it possible to achieve the near-complete duplicate elimination (recall that exact dedu-plication achieves complete duplicate elimination) andrequires at most one disk access per segment (a groupof chunks or small files) rather than one access perchunk (as in locality-based approaches) or per file (as insimilarity-based approaches), thus avoiding the disk bot-tleneck of data deduplication. In addition, the throughputof the deduplication system also tends to be improved byreducing the expensive accesses to on-disk index. As aresult, our SiLo approach, through its judicious and jointexploitation of locality and similarity, is able to signifi-cantly improve the overall performance of the deduplica-tion system as demonstrated in Section 4.

3 Design and Implementation

SiLo is designed for large-scale and disk-inline backupstorage systems. In this section, we will first describe thearchitecture of SiLo, followed by detailed discussions ofits design and implementation issues.

3.1 System Architecture OverviewAs depicted in Figure 4, the SiLo architecture consistsof four key functional components, namely, File Dea-mon (FD), Deduplication Server (DS), Storage Server(SS), and Backup Server (BS), which are distributed inthe datacenters to serve the backup requests. BS and DSreside in the metadata server (MDS) while FD is installedon each client machine that requires backup/restore ser-vices.

• File Deamon is a deamon program providing afunctional interface (e.g., backup/restore) in users’computers. It is responsible for gathering backupdatasets and sending/restoring them to/from Stor-age Servers for backups/restores. The processesof chunking, fingerprinting and segmenting can bedone by FD in the preliminary phase of the inlinededuplication. It also includes a File Agent that isresponsible for communicating with BS and DS andtransferring backup data to/from SS.

• Backup Server is the manager of the backup sys-tem that globally manages all jobs of backup/restoreand directs all File Agents and Storage Servers. Itmaintains a metadata database for administering allbackup files’ information.

• The main function of Deduplication Server is tostore and look up all fingerprints of files and chunks.

5

Chunking

User Interface

File Agent Job Agent Deduplication Metadata Agent

Storage Agent

Contain Store

Job MetaData Cache HashTable Block Store

Storage Agent

Contain Store

Storage Agent

Contain Store

Disk Disk Disk

Figure 4: The SiLo system architecture.

• Storage Server is the repository for backed-updata. SS in SiLo manages multiple Storage Nodesfor scalability and provides fast, reliable and safebackup/restore services.

In this paper, we focus on Deduplication Server sinceit is the most likely performance bottleneck of the entirededuplication system. DS consists of the locality hash ta-ble (LHTable), the similarity hash table (SHTable), writebuffer and read cache. While SHTable and LHTable in-dex segments and blocks, the similarity and locality unitsof SiLo respectively, the write buffer and read cache pre-serve the similarity and locality of the backup stream, asshown in Figure 5.

Block Block

Block Block

Block Block

Seg Key

SHTableRead Cache

Write Buffer

Block

Block

RepChunk ID

Block ID

Chunk ID

LHTable

Segment

Figure 5: Data structures of Deduplication Server.

The notion of segment is used to exploit the similar-ity of the backup stream while the block preserves thestream-informed locality layout of segments on the disk.SHTable provides the similarity detection for input seg-ments and LHTable serves to quickly index and filterout duplicate chunks. Note that, since this paper mainlyaims at improving the performance of accessing on-diskfingerprints in the deduplication system, all write/read

operations in this paper are performed in the form ofwriting/reading chunks’ fingerprints rather than the realbackup data.

3.2 Similarity AlgorithmAs mentioned in Section 2.3, SiLo exploits similarityand locality jointly. It exploits similarity by groupingstrongly correlated small files and segmenting large files,while locality is exploited by grouping contiguous seg-ments in a backup stream to preserve the locality layoutof these segments as depicted in Figure 6. Thus, seg-ments are the atomic building units of a block that isin turn the atomic unit of the write buffer and the readcache.

Figure 6: Data structure of the SiLo similarity algorithm.

As a salient feature of SiLo, the SiLo similarity algo-rithm is implemented in File Deamon, which structuresdata from backup streams into segments according to thefollowing three principles.

• P1. Correlated small files in a backup stream (e.g.,those under the same parent directory) are to begrouped into a segment.

• P2. A large file in a backup stream is divided intoseveral independent segments.

• P3. All segments are of approximately the same size(e.g., 2MB).

Where, P1 aims to reduce the RAM overhead of index-lookup; P2 helps expose more similarity characteristicsof large files to eliminate more duplicate data; and P3simplifies the management of segments. Thus, the simi-larity algorithm exposes and then exploits more similar-ity by leveraging file semantics and preserving locality-layout of a backup stream to significantly reduce theRAM usage.

SiLo employs the method of representative finger-printing [3] to represent each segment by a similarity-index entry in the similarity hash table. By virtue of P1,the SiLo similarity design solves the problem of small

6

(a) Input backup stream

(big file)(Small files)

(big file)

(b) Segmenting large files and grouping small files

(c) Similarity detection

(block)

(segment)

(d) Locality-enhanced similartity detection by chunks filtering

N N NNN

(potentially duplicate) (potentially duplicate)

(segment)

(dup chunks) (new chunks) (dup chunks)

similar similar

Input segments

Segments in cache or on disk

Figure 7: The workflow of the locality algorithm: it helpsdetect more potentially duplicate chunks that are missedby the similarity algorithm.

files taking up disproportionally large RAM space. Forexample, assuming an average segment size of 2MB andan average chunk or small file size of 8KB, a segment ac-commodates 250 chunks or small files, thus significantlyreducing the required index size in the memory. If weassume a 60-byte primary key for the similarity index-ing of a 2MB segment of backup data, which is consid-ered economic, a 1TB backup stream only needs 30MBsimilarity-index for deduplication that can easily fit in thememory.

3.3 Locality Algorithm

As another salient feature of SiLo, the SiLo locality al-gorithm groups several contiguous segments in a backupstream into a block and preserves their locality-layout onthe disk. Since block is also the minimal write/read unitof the write buffer and read cache in the SiLo system, itserves to maximize the RAM utilization and reduce fre-quent accesses to on-disk index by retaining access lo-cality in the backup stream. By exploiting the inherentlocality in backup streams, the block-based SiLo localityalgorithm is able to eliminate more duplicate data.

Figure 7 shows the workflow of the locality algorithm.According the locality characteristic of backup streams,if input segment S1k in block B1 is determined to be sim-ilar to segment S2k by hitting in the similarity hash table,SiLo will consider the whole block B1 to be similar toblock B2 that contains S2k. As a result, this grouping ofcontiguous segments into a block can eliminate more po-tentially duplicate data that is missed by the probabilisticsimilarity detection, thus complementing the similaritydetection.

When SiLo reads the blocks from disk by the simi-larity detection, it puts the recently accessed block into

the read cache. By preserving the backup-stream local-ity in the read cache, the accesses to on-disk index due tosimilarity detection can be significantly reduced, whichalleviates the disk bottleneck and increases the dedupli-cation throughput. Since it is at the block level wherelocality is preserved and exploited, the block size is animportant system parameter that affects the system per-formance such as duplicate elimination and throughput.The smaller the block size, the more disk accesses will berequired by the server to read the index, weakening thelocality exploitation. The larger the block size, on theother hand, the more unrelated segments will be read bythe server from the disk, increasing system’s space andtime overheads. Therefore, a proper block size not onlyprovides good duplicate elimination, but also achieveshigh throughput and low RAM usage in the SiLo system.

Each block in SiLo has its own Locality Hash Ta-ble (i.e., LHTable shown in Figure 5) for chunk filter-ing. Since a block contains several segments, it needs anindexing tool for thousands of fingerprints. The finger-prints in a block are organized into the LHTable whenreading the block from the disk. The additional timerequired for constructing LHTable in a block is signifi-cantly compensated by its quick indexing.

3.4 Cache and RAM Considerations

SiLo uses a very small portion of RAM as its write bufferand read cache to store a small number of recently ac-cessed blocks to avoid the frequent and expensive diskread/write operations. In our current design of SiLo, theread cache and the write buffer each contains a fixednumber of blocks. As illustrated in Figures 5 and 6, alocality-block contains only metadata information suchas LHTable, segment information, chunk information,and file information, which enables a 1MB locality-blockto represent a 200MB data-block.

Since users of file systems tend to duplicate files or di-rectories under the same directories, a significant amountof duplicate data can be eliminated by detecting the du-plication in the write buffer that also preserves the local-ity of a backup stream. For example, a code directorymay include many versions of source code files or docu-ments that can be good deduplication candidates.

The largest portion of RAM in the SiLo system is oc-cupied by the similarity hash table (i.e., SHTable shownin Figure 5). Assuming an average segment size of 2MBand a primary-key size of 60B, the SiLo SHTable re-quires 300MB for an average backup data of 10 TB. Thusthe RAM usage for the cache becomes negligibly smallas the data size further increases.

7

3.5 SiLo WorkflowTo put things together and in perspective, Figure 8 showsthe main workflow of the SiLo deduplication process.For an incoming backup stream, SiLo goes through thefollowing key steps:

Look up the SHTable for

the input segment

Is existed in SHTable Is Similar or Dup?

Filter the fingerprints in

the cache's LHTable

No

Yes

Read the block from disk

to the cache.

Similar

Write input segment into

block in the write buffer.

Is block in the cache?

NoYes

Is write buffer Full?Flush the Write buffer to

the Disk.Yes

Update the Similarity

Hash Table

No

The segment is complete

duplicate !Dup

Figure 8: The SiLo deduplication workflow.

1. Files in the backup stream are first chunked, fin-gerprinted and packed into segments by groupingstrongly correlated small files and segmenting largefiles in the File Agent.

2. Each newly generated segment Snew is checkedagainst SHTable for similarity detection. If the newsegment hits in SHTable, SiLo checks if the blockBbk containing Snew’s similar segment is in thecache. If it is not in the cache, Bbk is read fromthe disk to the read cache, where a block is replacedin the FIFO order if the cache is full. If Snew missesin SHTable, it is then checked against recently ac-cessed blocks in the read cache for potentially sim-ilar segment in one of the cached blocks (Bbk).

3. The duplicate chunks in Snew are eliminated bychecking LHTable of Bbk in the read cache. Thenthe chunks in the neighbouring segments of Snew

in the backup stream are filtered by the locality-enhanced similarity detection (i.e., these chunks arechecked against LHTable of Bbk for possible dupli-cation).

4. After chunk filtering and constructing a new andnon-duplicate block Bnew from the backup stream,SiLo checks if the write buffer is full. If the writebuffer is full, a block there is replaced in the FIFOorder by Bnew and then written to the disk.

As demonstrated in Section 4, SiLo is able to minimizeboth the time and space overheads of indexing finger-

prints while maintaining a duplicate elimination perfor-mance comparable to exact deduplication methods suchas ChunkStash.

4 Evaluation

In order to evaluate SiLo, we have implemented a pro-totype of SiLo that allows us to examine several impor-tant design parameters to provide useful insights. Wecompare SiLo with the similarity-based and locality-based state-of-the-art approaches Extreme Binning andChunkStash in the key deduplication metrics of duplicateelimination, RAM usage and throughput. The evaluationis driven by four real-world traces collected from realbackup datasets that represent different workload char-acteristics.

4.1 The Experimental Setup

We use a standard server configuration to evaluate andcompare the inline deduplication performances of theSiLo, ChunkStash and Extreme Binning approaches run-ning on a Linux environment. The hardware configura-tion includes a quad-core CPU running at 2.4GHz, witha 4GB RAM, 2 gigabit network interface cards, and two500GB 7200rpm hard disks.

Due to our lack of access to the source code of ei-ther the ChunkStash or Extreme Binning scheme, wehave chosen to implement both of them. More specifi-cally, we have implemented the locality-based and exact-deduplication approach of ChunkStash incorporating theprinciples and algorithms described in the ChunkStashpaper [8]. The ChunkStash approach makes full useof the inherent locality of backup streams and uses anovel data structure called Cuckoo hash for fingerprintindexing. We have also implemented a simple ver-sion of the Extreme Binning approach, which repre-sents a similarity-based and approximate-deduplicationapproach according to the algorithms described in theExtreme Binning paper [3]. Extreme Binning exploitsfile similarity instead of locality in the backup streams.

Note that our evaluation platform is not a production-quality deduplication system but rather a research pro-totype. Hence, our evaluation results should be inter-preted as an approximate and comparative assessmentof the three systems above, and not be used for abso-lute comparisons with other deduplication systems. TheRAM usage in our evaluation is obtained by recordingthe space overhead of index-lookup. The duplicate elim-ination performance metric is defined as the percentageof duplicate data eliminated by the system. Throughputof the system is measured by the rate at which finger-prints of the backup stream are processed, not the real

8

Feature One-set Inc-set Linux Full-setTotal size 530GB 251GB 101 GB 2.51TBTotal files 3.5M 0.59M 8.8M 11.3MTotal chunks 51.7M 29.4M 16.9M 417.6MAvg.chunk size 10KB 8KB 5.9KB 6.5KBDedupe factor 1.7 2.7 19 25Locality weak weak strong strongSimilarity weak strong strong strong

Table 1: Workload characteristics of the four traces usedin the performance evaluation. All use SHA-1 for chunkfingerprints and the content-based chunking algorithm.The deduplication factor is defined as the Totalsize /(To-talsize - Dedupsize) ratio.

backup throughput in that it does not measure the rate atwhich the backup data is transferred and stored.

Four traces representing different strengths of localityand similarity are used in the performance evaluation ofthe three deduplication systems and are listed in Table1. The four traces are collected from real-world datasetsof One-backup, Incremental-backup, Linux-version andFull-backup respectively.

The One-set trace was collected from 15 graduate stu-dents of our research group. To obtain traces from thisbackup dataset, we have built a deduplication analysistool that crawls the backup directory, and generates thesequences of chunk and file hashes for traces. Since weobtain only one full backup for this group, this trace hasweak locality and weak similarity. The Inc-set is a subsetof the trace reported by Tan et al. [24] and was collectedfrom initial full backups and subsequent incrementalbackups of eight members in a research group. There are391 backups with a total of 251GB data. Therefore, Inc-set represents datasets with strong similarity but weaklocality.

Linux-set, downloaded from the website [16], consistsof 900 versions from version 1.1.13 to 2.6.33, and repre-sents the characteristics of small files. Full-set consistsof 380 full backups of 19 researchers’ PCs, which is alsoreported by Xing et al. [25] and can be downloaded fromthe website [10]. Full-set represents datasets with stronglocality and strong similarity. Both Linux-set and Full-set are used in [25] and [3] to evaluate the performance ofExtreme Binning, and our use of these datasets resultedin similar and consistent evaluation results with the pub-lished studies.

With the above traces representing different but typ-ical workload characteristics, this evaluation intends toanswer, among other things, the following questions:Can the SiLo locality algorithm compensate for theprobabilistic similarity detection that may miss detect-ing large amounts of duplicate data? How effective isthe SiLo similarity algorithm under different workloadconditions? How is SiLo compared with existing state-

of-the-art deduplication approaches in key performancemeasures?

4.2 Interplay between Similarity and Lo-cality

The mutually interactive nature of similarity and localityin SiLo dictates a good understanding of the relationshipbetween locality and similarity before a thorough perfor-mance evaluation is carried out. Thus, we first examinethe impact of the SiLo design parameters of block sizeand segment size on duplicate elimination and time over-head, which is critical for the SiLo locality and similarityalgorithms.

From Figure 9 that shows the percentage of duplicatedata not eliminated, we find that the duplicate elimina-tion performance, defined as the percentage of dupli-cate data eliminated, increases with the block size butdecreases with the segment size. This is because thesmaller the segment is (e.g., segment size of 512KB),the more similarity can be exposed and detected, en-abling more duplicate data to be removed. On the otherhand, the larger the block is (e.g., block size of 512MB),the more locality of the backup stream will be retainedand captured, allowing SiLo to eliminate more (i.e.,97%∼99.9%) of redundant data regardless of the seg-ment size.

Although more redundant data can be eliminated byreducing the segment size or filling a block with moresegmentsas indicated by the results shown in Figure 9,it results in more accesses to on-disks index and higherRAM usage due to the increased index entries in theSHTable (see Figure 5). As the deduplication-time-overhead results of Figure 10 clearly suggest, continu-ously decreasing the segment size or increasing the blocksize can become counterproductive after a certain point.From Figure 10, we further find that, for a fixed blocksize, the time overhead is inversely proportional to thesegment size. This is consistent with our intuition thatsmaller segment size results in more frequent similar-ity detections for the input segments, which in turn cancause more accesses to on-disk index.

Figure 10 also shows that there is a knee point for eachcurve, meaning that for a given segment size and work-load the time overhead decreases first and then increases(except Figure 10 (c)). This may be explained by the factthat, with a very small block size (e.g., 8MB), there islittle locality to be mined, resulting in frequent accessesto on-disk index. With a very large block size (e.g.,512MB), SiLo also runs slower because the increaseddisk accesses for locality exploitation may result in moreunrelated segments being read in. The Linux-set are dif-ferent from other datasets in Figure 10, because the av-erage size of a Linux version is 110MB, which enables

9

8 16 32 64 128 256 5120%

5%

10%

15%

20%%

of d

uplic

ate

not e

limin

ated

The block size (MB)

512KB 1 MB 2 MB 4 MB 8 MB

(a) One-set

8 16 32 64 128 256 5120%

3%

6%

9%

12%

% o

f dup

licat

e no

t elim

inat

ed

The block size (MB)

512KB 1 MB 2 MB 4 MB 8 MB

(b) Inc-set

8 16 32 64 128 256 5120%

2%

4%

6%

8%

% o

f dup

licat

e no

t elim

inat

ed

The block size (MB)

512KB 1 MB 2 MB 4 MB 8 MB

(c) Linux-set

8 16 32 64 128 256 5120.0%

0.2%

0.4%

0.6%

0.8%

1.0%

% o

f dup

licat

e no

t elim

inat

ed

The block size (MB)

512KB 1 MB 2 MB 4 MB 8 MB

(d) Full-set

Figure 9: Percentage of duplicate data eliminated as a function of block size and segment size.

8 16 32 64 128 256 5120

400

800

1200

1600

2000

Tim

e ov

erhe

ad (s

)

The block size (MB)

512KB 1 MB 2 MB 4 MB 8 MB

(a) One-set

8 16 32 64 128 256 5120

200

400

600

800

Tim

e ov

erhe

ad (s

)

The block size (MB)

512KB 1 MB 2 MB 4 MB 8 MB

(b) Inc-set

8 16 32 64 128 256 5120

80

160

240

Tim

e ov

erhe

ad (s

)The block size (MB)

512KB 1 MB 2 MB 4 MB 8 MB

(c) Linux-set

8 16 32 64 128 256 5120

2000

4000

6000

8000

Tim

e ov

erhe

ad (s

)

The block size (MB)

512KB 1 MB 2 MB 4 MB 8 MB

(d) Full-set

Figure 10: Time overhead of SiLo deduplication as a function of block size and segment size.

0(0-0.1

)

[0.1-0

.2)

[0.2-0

.3)

[0.3-0

.4)

[0.4-0

.5)

[0.5-0

.6)

[0.6-0

.7)

[0.7-0

.8)

[0.8-0

.9)

[0.9-1

.0) 10%

20%

40%

60%

80%

100%

% o

f dup

licat

e da

ta e

limina

ted

The degree of segment similarity

Missed locality Similarity

Figure 11: Percentage of duplicate data eliminated asa function of different similarity degrees on the Linux-dataset by the similarity-only approach and locality-onlyapproach respectively.

more related locality to be exploited at the block size of256MB.

As analyzed above, there evidently exist an optimumsegment size and an optimum block size, subject to agiven workload and deduplication requirements (e.g., du-plicate elimination or deduplication throughput). Thechoice of segment size and block size can be dynami-cally adjusted by the user’s specific requirements (e.g.,the backup throughput or duplicate elimination or theRAM usage).

Figures 11 and 12 suggest that the full exploitation oflocality jointly with that of similarity can remove almostall redundant data missed by the similarity detection un-

Missed locality Similarity

One-set Inc-set Linux-set Full-set0%

20%

40%

60%

80%

100%

% o

f dup

licat

e da

ta e

limin

ated

The datasets

Figure 12: Percentage of duplicate data eliminatedon four datasets by the similarity-only approach andlocality-only approach respectively.

der all workloads. These results can be compared withFigure 2 and 3, then well verify our motivation of simi-larity and locality in Section 2. In fact, only an extremelysmall amount of duplicate data is missed by SiLo even onthe datasets with weak locality and similarity.

4.3 Comparative Evaluation of SiLo

This subsection presents evaluation results comparingSiLo with two other state-of-the-art deduplication sys-tems, the similarity-based Extreme Binning system andthe locality-based ChunkStash system, by executing thefour real-world traces described in Section 4.1 on thesethree systems. Note that in this evaluation SiLo assumes

10

One-set Inc-set Linux-set Full-set0%

20%

40%

60%

80%

100%%

of d

uplic

ate

data

elim

inat

ed

The datasets

ChunkStash SiLo-2MB SiLo-4MB ExtremeBin

Figure 13: Comparison among ChunkStash, SiLo, andExtreme Binning in terms of percentage of duplicate dataeliminated on the four datasets.

a block size of 256MB, while SiLo-2MB and SiLo-4MBrepresent SiLo with a segment size of 2MB and 4MBrespectively.

A. Duplicate eliminationFigure 13 shows the duplicate elimination perfor-

mance of the three systems under the four workloads.Since ChunkStash does the exact deduplication, it elim-inates 100% of duplicate data. Compared with ExtremeBinning that eliminates 71%∼99% of duplicate data inthe four datasets, SiLo removes about 98.5%∼99.9% ofduplicate data. Note that, while Extreme Binning elimi-nates about 99% of duplicate data as expected in Linux-set and Full-set that has strong similarity and locality, itfails to detect almost 30% of duplicate data in One-setthat has weak locality and similarity, and about 25% ofduplicate data in Inc-set with weak locality but strongsimilarity. Although there is strong similarity in Inc-set, Extreme Binning still fails to eliminate a significantamount of duplicate data primarily due to its probabilisticsimilarity detection that simply chooses one representa-tive fingerprint for each file regardless of the file size.

On the contrary, SiLo-2MB eliminates 99% of dupli-cate data even in One-set with both weak similarity andlocality, and also removes almost 99.9% of duplicate datain Linux-set and Full-set with both strong similarity andlocality. These results show that SiLo’s joint and com-plementary exploitation of similarity and locality is veryeffective in detecting and eliminating duplicate data un-der all workloads evaluated, achieving near-complete du-plicate elimination (i.e., exact deduplication).

B. RAM usageFigure 14 shows the RAM usage for deduplication

among these three systems under the four workloads.For Linux-set that has a very large number of small filesand small chunks, the highest RAM usage is incurred forboth Chunkstash and Extreme Binning. There is also aclear negative correlation between the deduplication fac-

824

666

1197

8520 14 20 211 7 14 1

76 45

1791

9

One-set Inc-set Linux-set Full-set0

300

600

900

1200

1500

1800

RAM

usa

ge p

er M

B da

ta (B

)

The datasets

ChunkStash SiLo-2MB SiLo-4MB ExtremeBin

Figure 14: Comparison among ChunkStash, SiLo, andExtreme Binning in terms of RAM usage (B: RAM re-quired per MB backup data).

tor and the RAM usage for the approximate deduplica-tion systems of SiLo and Extreme Binning on the otherfour workloads. That is, for One-set that has the lowestdeduplication factor, the highest RAM usage is incurred,while for Full-set that has highest deduplication factor,the smallest RAM space is required.

The average RAM usage for ChunkStash is the high-est among the three approaches, except for the Linux-set trace, as it does the exact deduplication that needsa large hash table in the memory to put all the indicesof chunk fingerprints. Although ChunkStash uses theCuckoo hash to store compact key signatures instead offull chunk-fingerprints, it still requires at least 6 bytes foreach new chunk, resulting in a very large cuckoo hashtable for millions of fingerprints. In addition, accord-ing to the open-source code of Cuckoo Hash [14], theChunkStash system needs to allocate about two millionhash table slots in advance to support one million indexentries.

Since only the file similarity index needs to be storedin RAM, Extreme Binning only consumes about 1/9∼1/15 of the RAM space required of ChunkStash excepton the Linux-set where it consumes more RAM usagethan ChunkStash due to the extremely large number ofsmall files. However, SiLo-2MB’s RAM efficiency al-lows it to reduce the RAM consumption of Extreme Bin-ning by a factor of 3∼900. The extremely low RAMoverhead of the SiLo system stems from the interplay be-tween its similarity algorithm, which groups many smallcorrelated files into segments and extracts their similaritycharacteristics, and its locality algorithm, which groupscontiguous segments of the backup stream into blocksto effectively exploit the locality residing in the backup-streams. On the other hand, the RAM usage for ExtremeBinning depends on the average file size of the file set, inaddition to the deduplication factor. The smaller the av-erage file size is, the more RAM space Extreme Binningwill consume, which is demonstrated in the Linux-set.

11

1 4 16 641

10

100

1000

10000R

AM u

sage

(GB)

The deduplication factor

ChunkStash ExtremeBin SiLo-2MB SiLo-4MB

Figure 15: Comparison among ChunkStash, SiLo, andExtreme Binning in terms of RAM usage in PB-scalededuplication with different deduplication factors. Weassume that the average file size is 200KB, and 80% ofduplicate data are from duplicate files.

The RAM usage of the SiLo system remains relativelystable with the change in average file size in the fourtraces and is inversely proportional to the deduplicationfactor of the traces.

Now we analyze the RAM usage in a PB-scale dedu-plcation system for the three approaches. As a 2MB-segment needs 60 bytes of key index in the memory, SiLotakes up about 30GB of RAM in a PB-scale deduplica-tion system. With 4MB-segments, SiLo’s RAM usageis halved to 15 GB in a PB-scale deduplication systemwhile its performance degrades gracefully as shown inFigures 9 and 10. Extreme Binning needs almost 300GBof RAM space with an average file size of 200KB whileChunkStash consumes almost 2TB of RAM space tomaintain a global index in a PB-scale deduplication sys-tem.Figure 15 also shows RAM usage of these three ap-proaches with different deduplication factors. Accordingto [15], Sparse Indexing uses 170GB of RAM space for aPB-scale deduplication system, whereas it estimates thatDDFS would require 360GB RAM to maintain a partialindex depending on locality in backup streams.

C. Deduplication throughputFigure 16 shows a comparison among the three ap-

proaches in terms of deduplication throughput, where thethroughput is observed to more than double as the aver-age chunk size changes from 6KB (e.g., Linux-set) to10KB (e.g., One-set).

ChunkStash achieves an average throughput of about335MB/sec with a range of 24MB/sec∼ 654MB/sec onthe four datasets. The frequency of accesses to on-diskindex by Chunkstash’s compact key signatures algorithmon the Cuckoo hash lookup tends to increase with thesize of the dataset, thus adversely affecting the through-put. Extreme Binning achieves an average throughput of904MB/sec with a range of 158MB/sec∼1571/sec on thefour datasets, since it only needs to access the disk once

One-set Inc-set Linux-set Full-set0

500

1000

1500

2000

Dedu

plica

tion

thro

ughp

ut (M

B/s)

The datasets

ChunkStash SiLo-2MB SiLo-4MB ExtremeBin

Figure 16: Comparison among ChunkStash, SiLo, andExtreme Binning in terms of deduplication throughput(MB/sec).

per similar-file and eliminates the duplicate files in thememory. As SiLo-2MB makes at most one disk accessper segment, it deduplicates data at an average through-put of 1167 MB/sec with a range of 581MB/sec∼1486MB /sec on the four datasets.

Although Extreme Binning runs faster than SiLo-2MBunder Inc-set where many duplicate files exist, it runsmuch slower in other datasets. Since each bin stores allsimilar files and it tends to grow in size with the datasetsize. As a result, Extreme Binning will slow down asthe size of each bin increases since each similar file mustread its corresponding bin in its entirety. In addition, thedesign of bin fails to exploit the backup-stream localitythat can help reduce disk accesses and increase the RAMutilization by preserving the locality layout in the mem-ory.

Since SiLo uses significantly less RAM space thanother approaches for a given dataset, SiLo can also boostthe deduplication throughput by caching more index in-formation in RAM to reduce accesses to on-disk index.In fact, the SiLo system can be dynamically configuredwith users’ requirements such as the throughput and du-plicate elimination by tuning the appropriate system pa-rameters (e.g., the number of blocks in the cache, the seg-ment size and block size, etc.). Therefore, compared withExtreme Binning and ChunkStash, SiLo is shown to pro-vide robust and consistently good deduplication perfor-mance, achieving higher throughput and near-completeduplicate elimination at a much lower RAM overhead.

5 Related work

Data deduplication is an essential and critical compo-nent of backup/archiving storage systems. It not onlyreduces storage space requirements, but also improvesthe throughput of the backup/archiving systems by elim-inating the network transmission of redundant data. We

12

briefly review the work that is most relevant to our SiLosystem to put it in the appropriate perspective, as follows.LBFS [19] first proposes the content-based chunking al-gorithm with the adoption of the Rabin fingerprints [22],and applies it to the network file system to reduce trans-mission of redundant data. Venti [21] employs dedupli-cation in an archival storage system and significantly re-duces the storage space requirement. Policroniades etc.[20] compares the performance of several deduplicationapproaches, such as file-level, fixed-size chunking andcontent-based chunking.

In recent years, more attention has been paid to avoid-ing the fingerprint-lookup disk bottleneck and enablingmore efficient and scalable deduplication in mass storagesystems. DDFS [26] is the earliest research to proposethe idea of exploiting the backup-stream locality to re-duce accesses to on-disk index and avoid the disk bottle-neck of inline deduplication. Sparse Indexing [15] alsoexploits the inherent backup-stream locality to solve theindex-lookup bottleneck problem. Different from DDFS,Sparse Indexing is an approximate deduplication solu-tion that samples index for fingerprint-lookup and onlyrequires about half of the RAM usage of DDFS. But itsduplicate elimination and throughput are heavily depen-dent on the sampling rate and chunks locality of backupstreams.

ChunkStash [8] stores the chunk fingerprints on anSSD instead of an HDD to accelerate the index-lookup.It also preserves the backup-stream locality in the mem-ory to increase the RAM utilization and reduce ac-cesses to on-disk index. Cuckoo hash is used byChunkStash to organize the fingerprint index in RAM,which is shown to be more efficient than Bloom filtersin DDFS. ChunkStash study also shows that the disk-based Chunkstash scheme performs comparable to theflash-based ChunkStash scheme when there is sufficientlocality in the data stream.

The aforementioned locality-based approaches wouldproduce unacceptably poor performance of deduplica-tion in the case of the data streams with little or no lo-cality [3]. Several earlier studies [17, 5, 9, 4] propose toexploit similarity characteristics for small-scale dedupli-cation of documents in the field of knowledge discoveryand database. SDS [2] exploits the similarity of backupstreams in mass deduplication systems. It divides a datastream into large 16MB blocks and constructs signaturesto identify possibly similar blocks. A byte-by-byte com-parison is conducted to eliminate duplicate data, whichis also the first deduplication scheme that uses similar-ity matching. But the index structure in SDS appearsto be proprietary and no details are provided in the refer-ence paper. Extreme Binning [3] exploits the file similar-ity for deduplication to apply to non-traditional backupworkloads with low-locality (e.g., incremental backup).

It stores a similarity index of each new file in RAM andgroups many similar files into bins that are stored on thedisks, thus it eliminates duplicate files in RAM and du-plicate chunks inside each bin by similarity detection.

SiLo is in part inspired by the Cumulus system andBimodal algorithm. Cumulus is designed for file-systembackup over the Internet under the assumption of a thincloud [18]. It proposes the aggregation of many smallfiles to a segment to avoid frequent network transfers ofsmall files in the backup system, and implements a gen-eral user-level deduplication. Bimodal [14] aims to re-duce the size of index by exploiting data-stream local-ity. It merges some contiguous and duplicate chunks,produces a chunk size that is 2-4 times larger than thatof general algorithms, and finds more potential duplicatedata among the boundaries of duplicate chunks.

Most recently, there have also been studies that ex-plore the emerging applications of deduplication, suchas the virtual machines [12, 7], the buffer cache [23],I/O deduplication [13] and flash [6, 11], suggesting anincreasing popularity and importance of data deduplica-tion.

6 Conclusion and future work

In this paper, we present SiLo, a similarity-locality baseddeduplication system that exploits both similarity and lo-cality in backup streams to achieve higher throughputand near-complete duplicate elimination at a much lowerRAM overhead than existing state-of-the-art approaches.SiLo exploits the similarity of backup streams by group-ing small correlated files and segmenting large files toreduce the RAM usage for index-lookup. The backup-stream locality is mined in SiLo by grouping contigu-ous segments in backup streams to complement the sim-ilarity detection and alleviate the disk bottleneck dueto frequent accesses to on-disk index. The combinedand complementary exploitation of these two backup-stream properties overcomes the shortcomings of exist-ing approaches based on either property alone, achievinga robust and consistently superior deduplication perfor-mance.

Results from experiments driven by real-worlddatasets show that the SiLo similarity algorithm signif-icantly reduces the RAM usage while the SiLo localityalgorithm helps eliminate most of the duplicate data thatis missed by the similarity detection. And there existsa solution that optimizes the trade-off between duplicateelimination and throughput by appropriately tuning thelocality and similarity parameters (i.e., the size of seg-ment and block).

As our future work of SiLo, we plan to build a mathe-matical model to quantitatively analyze why SiLo workswell with the combined and complementary exploitation

13

of similarity and locality and learn and adapt to the opti-mal parameter automatically by the real-time deduplica-tion factor and other system status. Due to its low sys-tem overheads, we also plan to apply the SiLo system toother deduplication applications such as cloud storage orprimary storage environments that desire to deduplicateredundant data with extremely low system overheads.

7 Acknowledgments

This work was supported by the National Basic Research973 Program of China under Grant No.2011CB302301,the National High Technology Research and De-velopment Program (“863” Program) of China un-der Grant No.2009AA01A401 and 2009AA01A402,NSFC No.60703046, 61025008, 60933002, 60873028,Changjiang innovative group of Education of ChinaNo.IRT0725, Fundamental Research Funds for the cen-tral universities, HUST, under grant 2010MS043, andthe US NSF under Grants NSF-IIS-0916859, NSF-CCF-0937993 and NSF-CNS-1016609.The authors are alsograteful to anonymous reviewers and our shepherd, AndyTucker, for their feedback and guidance.

References[1] AGRAWAL, N., BOLOSKY, W., DOUCEUR, J., AND LORCH, J.

A five-year study of file-system metadata. ACM Transactions onStorage (TOS) 3, 3 (2007), 9.

[2] ARONOVICH, L., ASHER, R., BACHMAT, E., BITNER, H.,HIRSCH, M., AND KLEIN, S. The design of a similarity baseddeduplication system. In Proceedings of SYSTOR 2009: The Is-raeli Experimental Systems Conference (2009), ACM, pp. 1–14.

[3] BHAGWAT, D., ESHGHI, K., LONG, D., AND LILLIBRIDGE,M. Extreme binning: Scalable, parallel deduplication for chunk-based file backup. In Modeling, Analysis & Simulation ofComputer and Telecommunication Systems, 2009. MASCOTS’09.IEEE International Symposium on (2009), IEEE, pp. 1–9.

[4] BHAGWAT, D., ESHGHI, K., AND MEHRA, P. Content-baseddocument routing and index partitioning for scalable similarity-based searches in a large corpus. In Proceedings of the 13th ACMSIGKDD international conference on Knowledge discovery anddata mining (2007), ACM, pp. 105–112.

[5] BRODER, A. On the resemblance and containment of documents.In Compression and Complexity of Sequences 1997. Proceedings(2002), IEEE, pp. 21–29.

[6] CHEN, F., LUO, T., AND ZHANG, X. CAFTL: A content-awareflash translation layer enhancing the lifespan of flash memorybased solid state drives. In FAST11: Proceedings of the 9th Con-ference on File and Storage Technologies (2011), USENIX Asso-ciation.

[7] CLEMENTS, A., AHMAD, I., VILAYANNUR, M., AND LI, J.Decentralized deduplication in SAN cluster file systems. In Pro-ceedings of the 2009 conference on USENIX Annual technicalconference (2009), USENIX Association, p. 8.

[8] DEBNATH, B., SENGUPTA, S., AND LI, J. ChunkStash: speed-ing up inline storage deduplication using flash memory. In Pro-ceedings of the 2010 USENIX conference on USENIX annualtechnical conference (2010), USENIX Association, p. 16.

[9] FORMAN, G., ESHGHI, K., AND CHIOCCHETTI, S. Findingsimilar files in large document repositories. In Proceedings of theeleventh ACM SIGKDD international conference on Knowledgediscovery in data mining (2005), ACM, pp. 394–400.

[10] FULL-DATASET. http://en.amazingstore.org/xyj/.[11] GUPTA, A., PISOLKAR, R., URGAONKAR, B., AND SIVA-

SUBRAMANIAM, A. Leveraging Value Locality in OptimizingNAND Flash-based SSDs. In FAST11: Proceedings of the 9thConference on File and Storage Technologies (2011), USENIXAssociation.

[12] JIN, K., AND MILLER, E. The effectiveness of deduplica-tion on virtual machine disk images. In Proceedings of SYS-TOR 2009: The Israeli Experimental Systems Conference (2009),ACM, pp. 1–12.

[13] KOLLER, R., AND RANGASWAMI, R. I/O deduplication: utiliz-ing content similarity to improve I/O performance. ACM Trans-actions on Storage (TOS) 6, 3 (2010), 1–26.

[14] KRUUS, E., UNGUREANU, C., AND DUBNICKI, C. Bimodalcontent defined chunking for backup streams. In Proceedingsof the 8th USENIX conference on File and storage technologies(2010), USENIX Association, p. 18.

[15] LILLIBRIDGE, M., ESHGHI, K., BHAGWAT, D., DEOLALIKAR,V., TREZISE, G., AND CAMBLE, P. Sparse indexing: large scale,inline deduplication using sampling and locality. In Proccedingsof the 7th conference on File and storage technologies (2009),USENIX Association, pp. 111–123.

[16] LINUX-DATASET. http://www.cn.kernel.org/pub/linux/kernel/.[17] MANBER, U., ET AL. Finding similar files in a large file system.

In Proceedings of the USENIX winter 1994 technical conference(1994), Citeseer, pp. 1–10.

[18] MICHAEL, V., STEFAN, S., AND GEOFFREY, M. Cumulus:Filesystem backup to the cloud. In Proceedings of 7th USENIXConference on File and Storage Technologies (2009).

[19] MUTHITACHAROEN, A., CHEN, B., AND MAZIERES, D. Alow-bandwidth network file system. In Proceedings of the eigh-teenth ACM symposium on Operating systems principles (2001),ACM, pp. 174–187.

[20] POLICRONIADES, C., AND PRATT, I. Alternatives for detectingredundancy in storage systems data. In Proceedings of the annualconference on USENIX Annual Technical Conference (2004),USENIX Association, p. 6.

[21] QUINLAN, S., AND DORWARD, S. Venti: a new approach toarchival storage. In Proceedings of the FAST 2002 Conference onFile and Storage Technologies (2002), vol. 4.

[22] RABIN, M. Fingerprinting by random polynomials. Center forResearch in Computing Techn., Aiken Computation Laboratory,Univ., 1981.

[23] REN, J., AND YANG, Q. A New Buffer Cache Design Ex-ploiting Both Temporal and Content Localities. In 2010 Inter-national Conference on Distributed Computing Systems (2010),IEEE, pp. 273–282.

[24] TAN, Y., JIANG, H., FENG, D., TIAN, L., YAN, Z., ANDZHOU, G. SAM: A Semantic-Aware Multi-Tiered Source De-duplication Framework for Cloud Backup. In 2010 39th Interna-tional Conference on Parallel Processing (2010), IEEE, pp. 614–623.

[25] XING, Y., LI, Z., AND DAI, Y. PeerDedupe: Insights into thePeer-Assisted Sampling Deduplication. In Peer-to-Peer Comput-ing (P2P), 2010 IEEE Tenth International Conference on (2010),IEEE, pp. 1–10.

[26] ZHU, B., LI, K., AND PATTERSON, H. Avoiding the disk bottle-neck in the data domain deduplication file system. In Proceedingsof the 6th USENIX Conference on File and Storage Technologies(2008), USENIX Association, pp. 1–14.

14