Data Organization

Problem

• Huge amounts of information

• How do I find– Information that I know I want– Information related to what I want

• How do I understand– Particular pieces of information– The whole collection of information

Limitations

• Screen space

• Network bandwidth– Bandwidth - how much information can be

transmitted per second

• Human attention

Kinds of things to organize

• Menu items– MS Word - about 150 menu items

• Text– Pages in a book - 500– Documents on the WWW - gazillions

• Images– All of the pictures created in a commercial

advertising company

Kinds of things to organize

• Sounds– Sound tracks to all TV and Radio news broadcasts

• Video– A complete collection of classic movies

• Structured information (records)– People– Cars– Students– Electronic appliance parts

A question of scale

• 10 things

• 100 things - menu

• 1,000 things - files on your computer

• 10,000 things - students at a university

• 1,000,000 things - books in a library

• gazillion things - WWW pages

Three ways to find things

• Lists – arrays

• Trees – organize in to categories

• Search – describe what you want and have the computer

find it

Finding things in lists

• How long will it take to find “Ron Dallin” in the Provo/Orem phone book?

• How long will it take to find “764-0588” in the Provo/Orem phone book?

Binary search - for “Goodrich”0 Anderson Lars 920-32311 Bilinski Phrank 123-31232 Clark Terry 764-23543 Garcia Jose 238-12344 Goodrich Carrie 824-92945 Gump George 232-03126 Jones Phred 723-92327 Randolph Jane 823-12428 Schwartz Gertrude 923-31349 Spimoni Maria 921-329410 Watt Harold 130-2313

Lower = 0Upper = 10

Guess = (0+10)/2 = 5

Lower = 0Upper = 5

Guess = (0+5)/2 = 2

Lower = 2Upper = 5

Guess = (2+5)/2 = 3

Lower = 3Upper = 5

Guess = (3+5)/2 = 4

Binary search

• If there are 64 things in a list, how many times can you divide that list in half?– 32, 16, 8, 4, 2, 1

• 6 times

Binary search

• If there are 1024 things in a list, how many times can you divide that list in half?– 512, 256, 128, 64, 32, 16, 8, 4, 2, 1

• 10 times

Binary search

• If the size of the list doubles, how many more steps are required in a binary search?

Binary search

• If there are N items in a list then binary search takes

• log2(N) steps

Binary search

• Estimating log2(N)– Count the number of digits and multiply by 2.5

• 1000– 4*2.5 = 10 steps

• 1,000,000– 7*2.5 = 17-18 steps

• 1,000,000,000– 10*2.5= 25 steps

Provo/Orem phone book

• How long to find “Ron Dallin”– 400,000 in Utah county

– Log2(400,000) approx 6*2.5 = 15 steps

How to find a phone number

0 Anderson Lars 920-32311 Bilinski Phrank 123-31232 Clark Terry 764-23543 Garcia Jose 238-12344 Goodrich Carrie 824-92945 Gump George 232-03126 Jones Phred 723-92327 Randolph Jane 823-12428 Schwartz Gertrude 923-31349 Spimoni Maria 921-329410 Watt Harold 130-2313

• 920-3231– 1 step

• 130-2313– 11 steps

• Average?– 5 steps

• Average N?– N/2

Provo/Orem phone book

• How many steps to find a phone number?– 400,000/2 = 200,000 average

• How can we improve this?

Sort the phone book by phone number

• What if I want to search on both name and number?

0 Bilinski Phrank 123-31231 Watt Harold 130-23132 Gump George 232-03123 Garcia Jose 238-12344 Jones Phred 723-92325 Clark Terry 764-23546 Randolph Jane 823-12427 Goodrich Carrie 824-92948 Anderson Lars 920-32319 Spimoni Maria 921-329410 Schwartz Gertrude 923-3134

Using an Index0 Goodrich Carrie 824-92941 Gump George 232-03122 Schwartz Gertrude 923-31343 Watt Harold 130-23134 Randolph Jane 823-12425 Garcia Jose 238-12346 Anderson Lars 920-32317 Spimoni Maria 921-32948 Bilinski Phrank 123-31239 Jones Phred 723-923210 Clark Terry 764-2354

0 61 82 103 54 05 16 97 48 29 710 3

Last Name

0 81 32 13 54 95 106 47 08 69 710 2

Phone number

0 61 82 103 54 05 16 97 48 29 710 3

Last Name

0 81 32 13 54 95 106 47 08 69 710 2

Phone number

Anderson

0 61 82 103 54 05 16 97 48 29 710 3

Last Name

0 81 32 13 54 95 106 47 08 69 710 2

Phone number

Anderson, Bilinski

0 61 82 103 54 05 16 97 48 29 710 3

Last Name

0 81 32 13 54 95 106 47 08 69 710 2

Phone number

Anderson, Bilinski, Clark

0 61 82 103 54 05 16 97 48 29 710 3

Last Name

0 81 32 13 54 95 106 47 08 69 710 2

Phone number

Anderson, Bilinski, Clark, Garcia

0 61 82 103 54 05 16 97 48 29 710 3

Last Name

0 81 32 13 54 95 106 47 08 69 710 2

Phone number

123-3123

0 61 82 103 54 05 16 97 48 29 710 3

Last Name

0 81 32 13 54 95 106 47 08 69 710 2

Phone number

123-3123, 130-2313

0 61 82 103 54 05 16 97 48 29 710 3

Last Name

0 81 32 13 54 95 106 47 08 69 710 2

Phone number

123-3123, 130-2313, 232-0312

0 61 82 103 54 05 16 97 48 29 710 3

Last Name

0 81 32 13 54 95 106 47 08 69 710 2

Phone number

123-3123, 130-2313, 232-0312, 238-1234

Search for Goodrich

0 Goodrich Carrie 824-92941 Gump George 232-03122 Schwartz Gertrude 923-31343 Watt Harold 130-23134 Randolph Jane 823-12425 Garcia Jose 238-12346 Anderson Lars 920-32317 Spimoni Maria 921-32948 Bilinski Phrank 123-31239 Jones Phred 723-923210 Clark Terry 764-2354

0 61 82 103 54 05 16 97 48 29 710 3

Last Name

Lower = 0Upper = 10

Guess = 5

Search for Goodrich

0 61 82 103 54 05 16 97 48 29 710 3

Last Name

Lower = 0Upper = 5

Guess = 2

Search for Goodrich

0 61 82 103 54 05 16 97 48 29 710 3

Last Name

Lower = 2Upper = 5

Guess = 3

Search for Goodrich

0 61 82 103 54 05 16 97 48 29 710 3

Last Name

Lower = 3Upper = 5

Guess = 4

Search for 823-1242

Lower = 0Upper = 10

Guess = 5

0 81 32 13 54 95 106 47 08 69 710 2

Phone number

Search for 823-1242

Lower = 5Upper = 10

Guess = 7

0 81 32 13 54 95 106 47 08 69 710 2

Phone number

Search for 823-1242

Lower = 5Upper = 7

Guess = 6

0 81 32 13 54 95 106 47 08 69 710 2

Phone number

Using an Index

0 61 82 103 54 05 16 97 48 29 710 3

Last Name

0 81 32 13 54 95 106 47 08 69 710 2

Phone number

• What about first name or city?– another index

Data Organization

• What are we organizing for?• Scale

– 10 - 1,000 - 1,000,000 - 1,000,000,000

• Lists– Unsorted (N/2)– Sorted Log2(N)

• count the digits and multiply by 2.5

• To access in many ways– Use many indices into the same data

Data Organization

Documents

DATA WAREHOUSING & DATA MINING IN AN ORGANIZATION

Data organization: hive meetup

Data organization in spreadsheets

MELJUN CORTES DATA Organization

Data Organization Btrees - Brown University · PDF fileDatabase System Concepts 11.2 Data organization and retrieval File organization can improve data retrieval time SELECT *

Physical Data Organization and Indexing

How to become a data-driven government organization · a data-driven organization? A lot of questions spring to mind when thinking about becoming a data-driven government organization,

Data Input and Organization

Maine Health Data Organization Hospital Financial ... 1_2017_Final.pdf · Maine Health Data Organization Hospital Financial Data Release Notes - November 2018 Hospital Mergers/Acquisitions:

Building a data fluent organization

Lecture 9 Aggregate Data Organization

Data Organization and Processing - siret.ms.mff.cuni.czsiret.ms.mff.cuni.cz/sites/default/files/doc/david.hoksza/lectures/ndbi007/2014W/... · Data organization - terminology •Database

6 Cache Data Organization - ECE:Course Pagecourse.ece.cmu.edu/~ece548/handouts/06idcach.pdf · 18-548/15-548 Cache Data Organization 9/14/98 1 6 Cache Data Organization 18-548/15-548

Data Scraping, Data Organization

Becoming a Data-Driven Organization

DATA ORGANIZATION FOR DATA …eprints.usm.my/29307/1/Data_organization_for_data_broadcasting_in...DATA ORGANIZATION FOR DATA BROADCASTING IN MOBILE COMPUTING by ROZAIDA BINTI MOHD

Creating a Data-Driven Organization (Data Day Seattle 2015)

Becoming a Data-Driven Organization - WordPress.com · Becoming a Data-Driven Organization: Aligning Business & Data Strategy Donna Burbank & Nigel Turner Global Data Strategy Ltd

Computing Basics. DATA ORGANIZATION Data/Information Organization Bit Byte Nybble (½ byte) Word Double word Etc

Data-driven Organization The Journey