Embed Size (px)
Citation preview
How IPFS Works(approximately)
Sawood Alam (@ibnesayeed)Old Dominion University
Original slides by Steven Allen (@Stebalien)
IPFS:
Lifecycle
AddingFiles
GettingFiles
IPFS:
Adding Files
IPFS:
Adding Files
-> CID: Content Identifier
-> IPFS Path: /ipfs/QmS4ustL54uo8FzR9455qaxZwuMiUhyvMcX9Ba8nUH4uVv
-> Gateway URL: https://ipfs.io/ipfs/QmS4ustL54uo8FzR9455qaxZwuMiUhyvMcX9Ba8nUH4uVv
CID
IPFS:
Getting Files
CID
IPFS:
Lifecycle
Import Name Find Fetch
AddingFiles
GettingFiles
Import Name Find Fetch
ChunkingUnixFSIPLD
CIDPathIPNS
RoutingDHTKademlia
Bitswap
ChunkingUnixFSIPLD
CIDPathIPNS
Bitswap
Import FetchFindName
Contiguous File:
Chunked File:
● Deduplication
● Piecewise Transfer
● Seeking
(each chunk is hashed)
RoutingDHTKademlia
ChunkingUnixFSIPLD
CIDPathIPNS
Bitswap
Import FetchFindName
Contiguous File:
Chunked File:
● Deduplication
● Piecewise Transfer
● Seeking
RoutingDHTKademlia
ChunkingUnixFSIPLD
CIDPathIPNS
Bitswap
Import FetchFindName
Contiguous File:
Chunked File:
● Deduplication
● Piecewise Transfer
● Seeking
RoutingDHTKademlia
ChunkingUnixFSIPLD
CIDPathIPNS
Bitswap
Import FetchFindName
Contiguous File:
Chunked File:
● Deduplication
● Piecewise Transfer
● Seeking
Deduplicated:
RoutingDHTKademlia
ChunkingUnixFSIPLD
CIDPathIPNS
Bitswap
Import FetchFindName
Contiguous File:
Fetched:
● Deduplication
● Piecewise Transfer
● Seeking
Chunked File:
RoutingDHTKademlia
ChunkingUnixFSIPLD
CIDPathIPNS
Bitswap
Import FetchFindName
Contiguous File:
Chunked File:
● Deduplication
● Piecewise Transfer
● Seeking
RoutingDHTKademlia
ChunkingUnixFSIPLD
CIDPathIPNS
Bitswap
Import FetchFindName
Contiguous File:
Chunked File:
● Deduplication
● Piecewise Transfer
● Seeking
RoutingDHTKademlia
ChunkingUnixFSIPLD
CIDPathIPNS
Bitswap
Import FetchFindName
File Chunks:
RoutingDHTKademlia
ChunkingUnixFSIPLD
CIDPathIPNS
Bitswap
Import FetchFindName
File Chunks:
UnixFS File:
(merkle-link)(a hash)
(merkle-tree)
0-200 200-350
0-100 100-200 100-200 200-250
RoutingDHTKademlia
ChunkingUnixFSIPLD
CIDPathIPNS
Bitswap
Import FetchFindName
File Chunks:
UnixFS File:
(merkle-tree-dag) - directed acyclic graph
(merkle-link)
0-200 200-350
0-100 100-200 100-200 200-250
RoutingDHTKademlia
ChunkingUnixFSIPLD
CIDPathIPNS
Bitswap
Import FetchFindName
UnixFS File(s):
UnixFS Directory: README.md main.go go.mod
File Chunks:
(merkle-link)
RoutingDHTKademlia
ChunkingUnixFSIPLD
CIDPathIPNS
Bitswap
Import FetchFindName
Meta-format for understanding, encoding, and decoding merkle-linked data.
RoutingDHTKademlia
ChunkingUnixFSIPLD
CIDPathIPNS
Bitswap
Import FetchFindName
http://b.com/Bar.json -> { "foo": http://a.com/Foo.json}http://a.com/Foo.json -> { "content": "I am foo"}
Linked Data
RoutingDHTKademlia
ChunkingUnixFSIPLD
CIDPathIPNS
Bitswap
Import FetchFindName
http://b.com/Bar.json -> { "foo": http://a.com/Foo.json}http://a.com/Foo.json -> { "content": "I am foo"}
Linked DataAuthority
RoutingDHTKademlia
ChunkingUnixFSIPLD
CIDPathIPNS
Bitswap
Import FetchFindName
QmBar -> { "foo": QmFoo}QmFoo -> { "content": "I am foo"}
Merkle-Linked Data● Immutable
● Authority Less
RoutingDHTKademlia
ChunkingUnixFSIPLD
CIDPathIPNS
Bitswap
Import FetchFindName
Content Identifier
● Used for content addressing
● Are self describing
● Used to name every piece of data in IPFS/IPLD
● Are basically a hash with some metadata
QmS4ustL54uo8FzR9455qaxZwuMiUhyvMcX9Ba8nUH4uVv
bafybeibxm2nsadl3fnxv2sxcxmxaco2jl53wpeorjdzidjwf5aqdg7wa6u
RoutingDHTKademlia
ChunkingUnixFSIPLD
CIDPathIPNS
Bitswap
Import FetchFindName
Digression:Content Addressing / Location Addressing
RoutingDHTKademlia
ChunkingUnixFSIPLD
CIDPathIPNS
Bitswap
Import FetchFindName
LocationAddressing
"My cat, Ozzy, is here."
Digression: Content Addressing
RoutingDHTKademlia
ChunkingUnixFSIPLD
CIDPathIPNS
Bitswap
Import FetchFindName
Content Addressing
"This is my cat, Ozzy."
RoutingDHTKademlia
ChunkingUnixFSIPLD
CIDPathIPNS
Bitswap
Import FetchFindName
LocationAddressing
Digression: Content Addressing
RoutingDHTKademlia
ChunkingUnixFSIPLD
CIDPathIPNS
Bitswap
Import FetchFindName
LocationAddressing
Digression: Content Addressing
RoutingDHTKademlia
ChunkingUnixFSIPLD
CIDPathIPNS
Bitswap
Import FetchFindName
LocationAddressing
Digression: Content Addressing
RoutingDHTKademlia
ChunkingUnixFSIPLD
CIDPathIPNS
Bitswap
Import FetchFindName
LocationAddressing
"That's the wrong cat!"
Digression: Content Addressing
RoutingDHTKademlia
ChunkingUnixFSIPLD
CIDPathIPNS
Bitswap
Import FetchFindName
LocationAddressing
"That's the wrong cat!"
(But you can't know that!)
Digression: Content Addressing
RoutingDHTKademlia
RoutingDHTKademlia
Verifiable, Immutable, Trustless
Permanent
ChunkingUnixFSIPLD
CIDPathIPNS
Bitswap
Import FetchFindName
ChunkingUnixFSIPLD
CIDPathIPNS
Bitswap
Import FetchFindName
Digression:Multiformats: Self Describing Data
RoutingDHTKademlia
ChunkingUnixFSIPLD
CIDPathIPNS
Bitswap
Import FetchFindName
Digression: Multiformats
● Multicodec: a non-magic number to uniquely identify a format, protocol, etc.
● Multihash: a self describing hash digest.
● Multibase: a self describing base-encoded string.
RoutingDHTKademlia
ChunkingUnixFSIPLD
CIDPathIPNS
Bitswap
Import FetchFindName
Digression: Multiformats
Multicodec: a non-magic number.
name, tag, code, description
identity, multihash, 0x00, raw binary
ip4, multiaddr, 0x04,
dccp, multiaddr, 0x21,
dnsaddr, multiaddr, 0x38,
protobuf, serialization, 0x50, Protocol Buffers
cbor, serialization, 0x51, CBOR
raw, ipld, 0x55, raw binary
...
github.com/multiformats/multicodec
RoutingDHTKademlia
ChunkingUnixFSIPLD
CIDPathIPNS
Bitswap
Import FetchFindName
Digression: Multiformats
Multihash: a self-describing hash digest:
● Hash Function (multicodec)
● Hash Digest Length
● Hash Digest
RoutingDHTKademlia
github.com/multiformats/multihash
ChunkingUnixFSIPLD
CIDPathIPNS
Bitswap
Import FetchFindName
Digression: A bit of metadata
Multibase: a self-describing base encoding.
● A multibase prefix.
○ b - base32
○ z - base58
○ f - base16
● Followed by the base encoded data.
bafybeibxm2...
RoutingDHTKademlia
ChunkingUnixFSIPLD
CIDPathIPNS
Bitswap
Import FetchFindName
Self Describing
● CIDv0: QmS4u...
○ Base58 encoded sha256 multihash
● CIDv1: bafybei...
○ Multibase encoded (ipld format multicodec, multihash) tuple.
● Why CIDv1?
○ Can be encoded in arbitrary bases (base32, base58, etc.).
○ Can link between merkle-dag formats using the ipld format multicodec.
RoutingDHTKademlia
ChunkingUnixFSIPLD
CIDPathIPNS
RoutingDHTKademlia
Bitswap
Import FetchFindName
IPFS uses paths, not URIs/URLs:
Unlike URLs, paths are recursive:
/dns/github.com/tcp/22/ssh/git
Versus:
git+ssh://github.com:22
Like URLs, paths are namespaced:
/ipfs/QmFoo/welcome.txt/ipns/QmBar/index.html
Not Composable!
ChunkingUnixFSIPLD
CIDPathIPNS
RoutingDHTKademlia
Bitswap
Import FetchFindName
IPNS maps Public Keys to paths
/ipns/QmMyKey -> /ipfs/QmFoo (signed)
/ipns/QmMyKey -> /ipfs/QmSomethingNew
IPNS is mutable
/ipns/QmMyKey -> /ipns/QmYourKey
IPNS can point to arbitrary paths
ChunkingUnixFSIPLD
CIDPathIPNS
Bitswap
Import FetchFindName
Content Address (CID) Location Address (Peer)
RoutingDHTKademlia
Solution: Keep a "routing table"
ChunkingUnixFSIPLD
CIDPathIPNS
Bitswap
Import FetchFindName
What Who
QmFoo Ozzy
QmBar Izzy
RoutingDHTKademlia
But the table is too BIG!
ChunkingUnixFSIPLD
CIDPathIPNS
RoutingDHTKademlia
Bitswap
Import FetchFindName
What Who
QmFoo Ozzy
QmBar Izzy
... millions of lines later ...
QmXXXXXXXXXXXXXXXX Seth (not a cat!)
Solution: Distribute the routing table and give a little bit to each peer.
ChunkingUnixFSIPLD
CIDPathIPNS
RoutingDHTKademlia
Bitswap
Import FetchFindName
What Who
QmBar Izzy
...
Ozzy Knows Izzy Knows
What Who
QmFoo Ozzy
...
How do we know who has what piece of the routing table?
ChunkingUnixFSIPLD
CIDPathIPNS
RoutingDHTKademlia
Bitswap
Import FetchFindName
How do we know who has what piece of the routing table?Solution: Deterministically distribute the routing table.
ChunkingUnixFSIPLD
CIDPathIPNS
RoutingDHTKademlia
Bitswap
Import FetchFindName
Distance Metric: Is peer X closer to content C than peer Y?Query Algorithm: Given the distance metric, how do we find the peers closest to C.
ChunkingUnixFSIPLD
CIDPathIPNS
RoutingDHTKademlia
Bitswap
Import FetchFindName
Distance Metric: XOR(HASH(C), HASH(Peer))Query Algorithm:1. Ask the closest peers you know for closer peers.2. Remember the closest peers.
ChunkingUnixFSIPLD
CIDPathIPNS
RoutingDHTKademlia
Bitswap
Import FetchFindName
Distance Metric: "Is this closer?"Query Algorithm: "How do I get closer?"
ChunkingUnixFSIPLD
CIDPathIPNS
RoutingDHTKademlia
Bitswap
Import FetchFindName
ChunkingUnixFSIPLD
CIDPathIPNS
Bitswap
Import FetchFindName
RoutingDHTKademlia
Izzy Wants
● QmTreats● QmToy
Ozzy Wants
● QmCuddles● QmFood● QmAttention
Izzy Ozzy
ChunkingUnixFSIPLD
CIDPathIPNS
Bitswap
Import FetchFindName
RoutingDHTKademlia
RoutingDHTKademlia
Izzy Wants
● QmTreats● QmToy
Ozzy Wants
● QmCuddles● QmFood● QmAttention
Izzy Ozzy
ChunkingUnixFSIPLD
CIDPathIPNS
Bitswap
Import FetchFindName
RoutingDHTKademlia
Izzy Wants
● QmTreats● QmToy
Ozzy Wants
● QmCuddles● QmFood● QmAttention
Izzy Ozzy
ChunkingUnixFSIPLD
CIDPathIPNS
Bitswap
Import FetchFindName
RoutingDHTKademlia
Izzy Wants
● QmTreats● QmToy
Ozzy Wants
● QmCuddles● QmFood● QmAttention
Izzy Ozzy
ChunkingUnixFSIPLD
CIDPathIPNS
Bitswap
Import FetchFindName
QmFood
QmAttention
QmToy
RoutingDHTKademlia
Izzy Wants
● QmTreats
Ozzy Wants
● QmCuddlesIzzy Ozzy
ChunkingUnixFSIPLD
CIDPathIPNS
Bitswap
Import FetchFindName
ChunkingUnixFSIPLD
CIDPathIPNS
Bitswap
Import FetchFindName
RoutingDHTKademlia
How IPFS Works(approximately)
Sawood Alam (@ibnesayeed)Old Dominion University
Original slides by Steven Allen (@Stebalien)
