1 Highly Scalable Algorithms for Robust String Barcoding Bhaskar DasGupta * Kishori M. Konwar Ion...

1

Highly Scalable Algorithms for Robust String Barcoding

Bhaskar DasGupta*

Kishori M. Konwar

Ion Mandoiu

Alex Shavartsman

Computer Science & Engineering Department

University of Connecticut

Storrs, CT

*Department of Computer Science University of Illinois at Chicago

Chicago, IL

2

Motivation• There are many critical situations when one needs to rapidly identify an unknown genomic sequence from among a given set of known sequences

Rapid identification of pathogens in epidemic outbreaks Monitoring of microbial communities, e.g., in environmental studies Fast database search

3

Possible Approaches

• Sequencing based: sequence the unknown DNA sequence, then use similarity search programs such as BLAST to identify the unknown virus sequence for pathogens in databaseSequencing is prohibitively expensive and time consuming

• Hybridization Based: identify the unknown sequence by testing for the presence of certain subsequencesSubsequence tests can be performed quickly and at low cost

using a variety of hybridization based methods

4

Sequence Fingerprints• For each sequence, find a subsequence that appears

in that sequence and only in that sequence

GTTGC GTTC CAT

CAGTTGC 1 0 0

CAGTTC 0 1 0

CATGGA 0 0 1

• Sequence barcodes: 0/1 vectors• When using fingerprints, barcode length = #sequences

5

String Barcoding

• (Borneman et al.’01, Rash & Gusfield’02): Unique occurrence of tested subsequences not needed, as long as 0/1 barcodes are unique

TG CAGT

CAGTTGC 1 1

CAGTTC 0 1

CATGGA 1 0

• When using non-unique subsequences, barcode length can be much smaller than #sequences

6

Overview• Problem Formulation and Previous Work

• Greedy Setcover Algorithm

• Experimental Results

• Conclusions

7

Given:

Genomic sequences g1,…, gn

Find:

Minimum number of distinguisher strings t1,…,tk

Such that:

For every gi gj there exists a distinguisher tl which is a substring of gi or gj but not of both

- At least log2n distinguishers needed

- n distinguishers are always sufficient

Problem Definition

8

Computational Complexity

• [Berman et al.’04] Cannot be approximated within a factor of (1-)ln(n) unless NP=DTIME(nloglog(n))

9

• A non-redundant set of candidate distinguishers is generated using a suffix tree• One variable vi for each candidate distinguisher xi

vi = 1 xi is selected vi = 0 xi is not selected

Rash & Gusfield Integer Program

10

Integer Program ExampleMinimize VVTGTG + V + VATGGAATGGA+ V+ VCAGTCAGT + V + VTTC +V +VGTGC #objective function

Such that VVTGTG + VVTTC + VVGTGC >= 1 #constraint to cover pair 1,2 VVATGGAATGGA + VVCAGTCAGT + VVGTGC >= 1 #constraint to cover pair 1,3 VVTGTG + VVATGGAATGGA + VVCAGTCAGT + VVTTC >= 1 #constraint to cover pair 2,3Binaries #all variables are 0/1 VVTGTG V VATGGAATGGA VVCAGTCAGT VVTTC VVGTGC End

TG ATGGA

1. CAGTGC 1 0

2. CAGTTC 0 0

3. CCATGGA 1 1

11

Limitations of Integer Program Method

• Works only for moderately sized datasets 50-150 sequencesAverage sequence length ~1000 nucleotidesUp to 4 hours needed to come within 20% of optimum

12

Information Content Heuristic

• [Berman et al. 2004]Keep track of the partition defined by distinguishers

selected so far In every step, choose candidate that reduces partition

entropy by largest amount

• Theorem: Information Content Heuristic is always finding a #distinguishers within 1+ln(n) of optimum

17

Limitations of ICH• Real genomic sequences contain degenerate

nucleotides (e.g., N for any of {A,T,C,G}) due to sequencing errors and known single nucleotide polymorphisms

• Distinguisher-to-sequence matches:Perfect matchesPerfect mismatchesUncertain matches

• Information Content cannot be defined in the presence of uncertain matches

ATCNAT

ATC 1

CCC 0

CCA ?

18

Other Heuristics

• (Cazalis et al 2004): greedy setcover, simulated annealing, and genetic algorithms for distinguisher selection

• To achieve practical running time, only a small random subset (2000 candidates) of all candidate distinguishers is consideredNo data provided on the loss of solution quality due

to this restriction

19

Overview• Problem Formulation and Previous work

• Greedy Setcover Algorithm

• Experimental Results

• Conclusions

20

Setcover Greedy Heuristic

• Phase I: Candidate Generation Generate a representative set of candidate

distinguishers from the source sequences

• Phase II: Greedy Distinguisher SelectionIn every step, choose candidate that distinguishes the

largest number of not yet distinguished pairs

21

Candidate Generation

• A set of candidate distinguishers guaranteed to contain an optimum solution is generated from the sequences

• We do not generate certain redundant candidates A candidate is redundant if there is another candidate

that appears exactly in the same set of sequencesFor every sequence we generate only one of the

substrings that appear exclusively in that sequence

22

Efficient Candidate Generation• Our implementation uses simple array datastructures

We generate candidates in increasing order of lengthExact match positions for candidates of length l-1 used to

generate the exact matches for candidates of length l

• Candidates that do not satisfy individual given biochemical constraints, such as minimum/maximum length, GC content, melting temperature, are discarded

23

Setcover Greedy Heuristic

• Phase I: Candidate Generation Generate a set of candidate distinguishers from the

source sequences

• Phase II: Greedy Distinguisher SelectionIn every step, choose candidate that distinguishes the

largest number of not yet distinguished pairs

24

Distinguisher Selection as Set Cover

• Set Cover Problem: given a universal set U and a family of subsets, find a minimum number of subsets covering U

• Distinguisher selection is a special case of set cover:Elements to be covered are the pairs of sequencesEach candidate distinguisher defines a set of pairs that it

separates

• By a classical result, the greedy algorithm has an approximation factor of 1+ln(|U|) Setcover greedy has approximation factor of 2*ln(n) for

distinguisher selection with n sequences

25

Distinguisher Selection• Start with an empty set D of distinguishers

• While there are pairs of sequences not yet distinguished, do:

Compute for each remaining candidate c its coverage gain (c, D) – the number of not yet distinguished pairs of sequences that are distinguished by c

Add the candidate with maximum coverage gain to D

• Return D

26

Computation of (c, D):

CATCAGA

TTCAGT

TAT

AATAG

AATCAG

D = { }

27

Computation of (c, D):

CATCAGA

TTCAGT

TAT

AATAG

AATCAG

c=TCAG

D = { }

28

Computation of (c, D):

CATCAGA

TTCAGT

TAT

AATAG

AATCAG

c=TCAG

D = { }

(c, D)= 3 x (5 –3) = 6

29

Computation of (c, D):

CATCAGA

TTCAGT

TAT

AATAG

AATCAG

D = {TCAG}

30

Computation of (c, D):

CATCAGA

TTCAGT

TAT

AATAG

AATCAG

D = {TCAG}

c=AAT

31

Computation of (c, D):

CATCAGA

TTCAGT

TAT

AATAG

AATCAG

D = {TCAG}

c=AAT

(c,D)= 1 x (2-1) + 1 x (3-1) = 3

32

Computation of (c, D):

CATCAGA

TTCAGT

TAT

AATAG

AATCAG

D = {TCAG,AAT}

33

Computation of (c, D)

|\|||),(1

cic

k

ii MSMSDc

• S1, S2, …, Sk are the subsets in the partition defined by D

• Mc is the set of matches of candidate c

• Using simple datastructures, computation can be done in linear time (in the number of sequences)

34

Lazy Update of Gains

• Coverage gains are monotonically non-increasing during the algorithm

• Re-compute coverage gain for a candidate only if last saved gain is higher than the gain of current best candidate

• In practice this speeds-up the selection algorithm by a factor of ~2

35

• Degenerate basesA pair of sequences is separated by candidate c if

c has at least one perfect match with one of the sequences, andc has perfect mismatches at all positions of the other sequence

Gain computation done in O(n2) time using a simple coverage matrix data-structure

• Redundancy rA pair of sequences is counted in the gain function until r

distinguishers separate it

• Distinguisher cross-hybridizationMinimum edit distance, or maximum common substring

weight, bound for every pair of selected distinguishersCandidates incompatible with a selected distinguisher

removed from candidate list

Algorithm Extensions

36

Overview• Problem Formulation and Previous work

• Greedy Setcover Algorithm

• Experimental Results

• Conclusions

37

• Randomly generated instances Equal probabilities assigned to each of the four

nucleotides

• Microbial genomes extracted from NCBI databasesSequence lengths between 490 Kbases to 4.75

MbasesSmall number of degenerate bases

Testcases

38

Selection time, L=10k, r=1

basic – O(n2) computation of gains using matrix datastructure

partition – O(n) computation of gains using partition-based datastructure

39

Candidate Sampling, n=1000, L=10k, r=1

40

Comparison to ICH, L=10k, r=1

Algo

n

10 20 50 100 200 500 1000

log2n

ICH

SGA

4 5 6 7 8 9 10

4.0 5.0 7.0 8.0 10.0 12.2 14.1

4.0 5.0 7.0 8.0 10.0 12.3 14.1

41

Varying Redundancy, L=10k

0

10

20

30

40

50

60

70

80

0 5 10 15 20

Redundancy

#Dis

tin

gu

ish

ers

n=10 n=20 n=50 n=100

n=200 n=500 n=1000

42

• 20 NCBI microbial genomic sequences

• Distinguisher melting temperature range of 55-60 oC

• GC content range of 40-60%

• Max common subsequence weight bound of 5 weight(A)=weight(T)=1, weight(C)=weight(G)=2

NCBI testcase

43AACTGTCTCACGACGTTCTGAA

GATTCGAACCCCCGA

GTGGATGCCTTGGCA

GGACTACCAGGGTATCTAATCCTG

AAAGAAGATAGAGCAGCAGCT

AAGCGCGTCGCAAA

CACAAGGAGTGAGTGTTGC

CGGTTTTGTGCTTCATGG

CCATTGACAATTTCAACACC

Organism Mb Barcode

Nanoarchaeum equitans Kin4-M 0.49 0 0 0 0 0 0 0 0 1Mycobacterium tuberculosis CDC1551 4.40 0 0 0 0 0 0 1 0 0Brucella suis 1330 chromosome 1 2.11 0 0 0 0 1 1 0 1 0Leifsonia xyli subsp. xyli str. CTCB07 2.58 0 0 0 0 0 0 1 0 1Mannheimia succiniciproducens MBEL55E 2.31 0 0 0 0 1 1 1 0 0Geobacter sulfurreducens PCA 3.81 0 0 0 1 0 0 0 0 0Rickettsia typhi str. Wilmington 1.11 0 0 0 0 0 1 1 0 1Picrophilus torridus DSM 9790 1.55 0 1 0 0 0 0 0 0 1Mesoplasma florum L1 0.79 0 0 0 0 0 0 0 1 1Methylococcus capsulatus str. Bath 3.30 0 0 0 0 0 1 0 0 1Propionibacterium acnes KPA171202 2.56 0 0 0 0 0 0 1 1 0 Mycoplasma mobile 163K 0.78 0 0 0 0 0 1 0 1 1Mycoplasma hyopneumoniae 232 0.89 1 0 0 0 0 1 0 1 1Bacillus licheniformis DSM 13 4.22 0 0 0 0 0 1 1 1 0 Legionella pneumophila subsp. pneumophila str. Philadelphia 1

3.40 0 0 0 0 0 1 1 0 0

Onion yellows phytoplasma OY-M DNA 0.86 0 0 0 0 1 1 1 1 0 Staphylococcus aureus subsp. Aureus strain MRSA252

2.90 0 0 1 0 0 1 1 1 1

Staphylococcus aureus strain MSSA476 2.80 0 0 0 0 0 1 1 1 1 Burkholderia pseudomallei strain K96243 chromosome 1

4.07 0 0 0 0 0 1 0 0 0

Bartonella henselae strain Houston-1 1.93 0 0 0 0 0 1 0 1 0GC (%) 60.0 45.5 60.0 50.0 57.1 50.0 52.6 42.9 40.0Tm (oC) 55.6 59.6 55.4 59.3 56.9 58.6 55.1 55.4 56.3

NCBI testcase, r=1

44

Results on 29 Microbial Sequences (76 Mb)

Redun lmin lmax MinEdit Select Time #Distinguishers

1

1

0 0 14.2 6.015 40 6 2.6 8.0

5

5

0 0 20.3 21.015 40 6 8.7 31.0

10

10

0 0 22.9 41.015 40 6 16.4 60.0

20

20

0 0 26.8 76.015 40 6 33.4 123.0

45

Overview• Problem Formulation and Previous work

• Greedy Setcover Algorithm

• Experimental Results

• Conclusions

46

• We provided highly scalable algorithms for the robust string barcoding problem, capable of handling whole genomic sequences of up to bacterial size

• Distinguisher selection based whole genomic sequences results in a number of distinguishers nearly matching the information theoretic lower bounds for the problem

• The software can be used online at http://dna.engr.uconn.edu/~software/DNA-BAR/

Conclusions