Data Organization

Data Organization

Problem

• Huge amounts of information

• How do I find– Information that I know I want– Information related to what I want

• How do I understand– Particular pieces of information– The whole collection of information

Limitations

• Screen space

• Network bandwidth– Bandwidth - how much information can be

transmitted per second

• Human attention

Kinds of things to organize

• Menu items– MS Word - about 150 menu items

• Text– Pages in a book - 500– Documents on the WWW - gazillions

• Images– All of the pictures created in a commercial

advertising company

Kinds of things to organize

• Sounds– Sound tracks to all TV and Radio news broadcasts

• Video– A complete collection of classic movies

• Structured information (records)– People– Cars– Students– Electronic appliance parts

A question of scale

• 10 things

• 100 things - menu

• 1,000 things - files on your computer

• 10,000 things - students at a university

• 1,000,000 things - books in a library

• gazillion things - WWW pages

Three ways to find things

• Lists – arrays

• Trees – organize in to categories

• Search – describe what you want and have the computer

find it

Finding things in lists

• How long will it take to find “Ron Dallin” in the Provo/Orem phone book?

• How long will it take to find “764-0588” in the Provo/Orem phone book?

Binary search - for “Goodrich”0 Anderson Lars 920-32311 Bilinski Phrank 123-31232 Clark Terry 764-23543 Garcia Jose 238-12344 Goodrich Carrie 824-92945 Gump George 232-03126 Jones Phred 723-92327 Randolph Jane 823-12428 Schwartz Gertrude 923-31349 Spimoni Maria 921-329410 Watt Harold 130-2313


Lower = 0Upper = 10

Guess = (0+10)/2 = 5


Lower = 0Upper = 5

Guess = (0+5)/2 = 2


Lower = 2Upper = 5

Guess = (2+5)/2 = 3


Lower = 3Upper = 5

Guess = (3+5)/2 = 4

Binary search

• If there are 64 things in a list, how many times can you divide that list in half?– 32, 16, 8, 4, 2, 1

• 6 times

Binary search

• If there are 1024 things in a list, how many times can you divide that list in half?– 512, 256, 128, 64, 32, 16, 8, 4, 2, 1

• 10 times

Binary search

• If the size of the list doubles, how many more steps are required in a binary search?

1

Binary search

• If there are N items in a list then binary search takes

• log2(N) steps

Binary search

• Estimating log2(N)– Count the number of digits and multiply by 2.5

• 1000– 4*2.5 = 10 steps

• 1,000,000– 7*2.5 = 17-18 steps

• 1,000,000,000– 10*2.5= 25 steps

Provo/Orem phone book

• How long to find “Ron Dallin”– 400,000 in Utah county

– Log2(400,000) approx 6*2.5 = 15 steps

How to find a phone number

0 Anderson Lars 920-32311 Bilinski Phrank 123-31232 Clark Terry 764-23543 Garcia Jose 238-12344 Goodrich Carrie 824-92945 Gump George 232-03126 Jones Phred 723-92327 Randolph Jane 823-12428 Schwartz Gertrude 923-31349 Spimoni Maria 921-329410 Watt Harold 130-2313

• 920-3231– 1 step

• 130-2313– 11 steps

• Average?– 5 steps

• Average N?– N/2

Provo/Orem phone book

• How many steps to find a phone number?– 400,000/2 = 200,000 average

• How can we improve this?

Sort the phone book by phone number

• What if I want to search on both name and number?

0 Bilinski Phrank 123-31231 Watt Harold 130-23132 Gump George 232-03123 Garcia Jose 238-12344 Jones Phred 723-92325 Clark Terry 764-23546 Randolph Jane 823-12427 Goodrich Carrie 824-92948 Anderson Lars 920-32319 Spimoni Maria 921-329410 Schwartz Gertrude 923-3134

Using an Index0 Goodrich Carrie 824-92941 Gump George 232-03122 Schwartz Gertrude 923-31343 Watt Harold 130-23134 Randolph Jane 823-12425 Garcia Jose 238-12346 Anderson Lars 920-32317 Spimoni Maria 921-32948 Bilinski Phrank 123-31239 Jones Phred 723-923210 Clark Terry 764-2354

0 61 82 103 54 05 16 97 48 29 710 3

Last Name

0 81 32 13 54 95 106 47 08 69 710 2

Phone number


0 61 82 103 54 05 16 97 48 29 710 3

Last Name

0 81 32 13 54 95 106 47 08 69 710 2

Phone number

Anderson


0 61 82 103 54 05 16 97 48 29 710 3

Last Name

0 81 32 13 54 95 106 47 08 69 710 2

Phone number

Anderson, Bilinski


0 61 82 103 54 05 16 97 48 29 710 3

Last Name

0 81 32 13 54 95 106 47 08 69 710 2

Phone number

Anderson, Bilinski, Clark


0 61 82 103 54 05 16 97 48 29 710 3

Last Name

0 81 32 13 54 95 106 47 08 69 710 2

Phone number

Anderson, Bilinski, Clark, Garcia


0 61 82 103 54 05 16 97 48 29 710 3

Last Name

0 81 32 13 54 95 106 47 08 69 710 2

Phone number

123-3123


0 61 82 103 54 05 16 97 48 29 710 3

Last Name

0 81 32 13 54 95 106 47 08 69 710 2

Phone number

123-3123, 130-2313


0 61 82 103 54 05 16 97 48 29 710 3

Last Name

0 81 32 13 54 95 106 47 08 69 710 2

Phone number

123-3123, 130-2313, 232-0312


0 61 82 103 54 05 16 97 48 29 710 3

Last Name

0 81 32 13 54 95 106 47 08 69 710 2

Phone number

123-3123, 130-2313, 232-0312, 238-1234

Search for Goodrich

0 Goodrich Carrie 824-92941 Gump George 232-03122 Schwartz Gertrude 923-31343 Watt Harold 130-23134 Randolph Jane 823-12425 Garcia Jose 238-12346 Anderson Lars 920-32317 Spimoni Maria 921-32948 Bilinski Phrank 123-31239 Jones Phred 723-923210 Clark Terry 764-2354

0 61 82 103 54 05 16 97 48 29 710 3

Last Name

Lower = 0Upper = 10

Guess = 5

lower

Search for Goodrich


0 61 82 103 54 05 16 97 48 29 710 3

Last Name

Lower = 0Upper = 5

Guess = 2

above

Search for Goodrich


0 61 82 103 54 05 16 97 48 29 710 3

Last Name

Lower = 2Upper = 5

Guess = 3

above

Search for Goodrich


0 61 82 103 54 05 16 97 48 29 710 3

Last Name

Lower = 3Upper = 5

Guess = 4

above

Search for 823-1242


Lower = 0Upper = 10

Guess = 5

above

0 81 32 13 54 95 106 47 08 69 710 2

Phone number

Search for 823-1242


Lower = 5Upper = 10

Guess = 7

below

0 81 32 13 54 95 106 47 08 69 710 2

Phone number

Search for 823-1242


Lower = 5Upper = 7

Guess = 6

MATCH

0 81 32 13 54 95 106 47 08 69 710 2

Phone number

Using an Index


0 61 82 103 54 05 16 97 48 29 710 3

Last Name

0 81 32 13 54 95 106 47 08 69 710 2

Phone number

• What about first name or city?– another index

Data Organization

• What are we organizing for?• Scale

– 10 - 1,000 - 1,000,000 - 1,000,000,000

• Lists– Unsorted (N/2)– Sorted Log2(N)

• count the digits and multiply by 2.5

• To access in many ways– Use many indices into the same data