64
1 NUS.SOC.CS5248 Ooi Wei Tsang CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 [email protected]

NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 [email protected]

Embed Size (px)

Citation preview

Page 1: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

1

NUS.SOC.CS5248Ooi Wei Tsang

CS6282 Very Large Scale Distributed

Systems

Ooi Wei TsangSOC1-04-20

[email protected]

Page 2: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

Distributed Systems

Page 3: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

Very Large ScaleDistributed Systems

Page 4: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

New technological trends lead to VLDS

Page 5: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

Examples

Page 6: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

P2P File Sharing

Page 7: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

MMORPG

Page 8: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

News Alert

Page 9: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

Web Services

Page 10: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

Sensor Network

Page 11: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

Examples of VLDS

P2P File SharingMMORPG

News AlertVideo on Demand

Page 12: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

Scope of CS6282

Explore basic techniques and

their applications

Page 13: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

Goal

At the end of the class, students are expected to understand how to

build large scale distributed systems.

Page 14: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

Target Audience

Anyone interested in distributed systems

ordoing related research(db, grid, sensors, media etc.)

Page 15: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

Learning Mode

I am here to light you on fire, not to fill your

bucket.

Page 16: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

Learning Mode

We learn together

Page 17: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

Activities

ReadThink

DiscussInnovate

Page 18: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

Continuous Assessments

50

20

20

10

Project

Exam

Assignments/Quizzes

Class Participation

Page 19: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

Exam (20%)

Open BookEssay Type

Page 20: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

Project (50%)

Individual/PairResearch Based

Expect Something New

Page 21: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

Types of Project

Identify a new problemPropose a solution

Page 22: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

Types of Project

Existing ProblemNew Solution

Page 23: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

Types of Project

Existing SolutionsNew Evaluations

Page 24: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

Types of Project

Existing SolutionsNew Application

Page 25: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

Types of Project

Collect and Analyze Data

New Insights

Page 26: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

Project ComponentsDefine ProblemFind SolutionImplementEvaluatePresent

Page 27: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

Project Deliverables

8-page research paperPoster presentations

Page 28: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

Combine Project

with your current research

(subj. to advisor’s approval)

with another module (subj to instructor’s approval)

Page 29: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

In Class Participation

DiscussionsScribing

Presentations

Page 30: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

Scribing

Scribe notesDiscuss and mergeTypeset in LaTeX

Post online

Page 31: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

Presentations

Read papersPresent in class

Answer questions

Page 32: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

Assignments + QuizzesQuizzes

Paper ReviewsSurvey

Page 33: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

WARNING

Zero-Tolerance for Plagiarism

Page 34: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

Related Class

CS5223 Distributed Systems

Tue 6:30-8:30pmLT33

Gary Tan

Page 35: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

Very Large ScaleDistributed Systems

Page 36: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

Design Requirements

Page 37: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

Correct

Page 38: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

Scalable

Page 39: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

Robust

Page 40: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

Efficient

Page 41: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

Simple

Page 42: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

Secure

Page 43: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

Design Requirements

CorrectScalableRobustEfficientSimpleSecure

Page 44: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

UndesirableCharacteristics

Page 45: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

Numbers

Page 46: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

Scattered

Page 47: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

Dynamic

Page 48: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

Unreliable

Page 49: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

Untrustable

Page 50: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

Undesirable Properties

NumbersScatteredDynamic Unreliable

Untrustable

Page 51: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

MotivatingExamples

Page 52: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

DNS

(lookup)

Page 53: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

P2P File Sharing

(lookup, storage)

Page 54: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

VoD

(multicast)

Page 55: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

MMORPG

(global states)

Page 56: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

Network Management

(data collection)

Page 57: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

Olympics News Alert

(event notification)

Page 58: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

Operations

LookupMulticast

Data CollectionStorage

Event Notifications:

Page 59: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg
Page 60: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

StandardTechniques

Page 61: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

Build a Tree (or some form of

Hierarchy)

Page 62: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

Build a Mesh/Graph(or some structure)

Page 63: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

Gossip(send to random nodes)

Page 64: NUS.SOC.CS5248 Ooi Wei Tsang 1 CS6282 Very Large Scale Distributed Systems Ooi Wei Tsang SOC1-04-20 ooiwt@comp.nus.edu.sg

Flood(send to everyone)