Analyzing Blockchain and Bitcoin Transaction Data as Graph...Financial Services •Model customer relationship to products, services, people, places. •Analyze money customer [s flow

Copyright © 2015 Oracle and/or its affiliates. All rights reserved.

Analyzing Blockchain and Bitcoin Transaction Data as Graph

Xavier Lopez Zhe Wu

Senior Director Architect

Oracle Code BostonApril 17th, 2018


Safe Harbor Statement

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

2


Outline

3

• Overview Graph for Blockchain Analytics

• Understanding bitcoin transactions

• Graph modeling of bitcoin transactions

• Analysis of Bitcoin Transaction Graph (BTG) on Oracle Cloud

• Summary and future work


Graph Analysis for Business Insight

4

Identify Influencers

Discover Graph Patterns in Big Data

Generate Recommendations


Financial Services

• Model customer relationship to products, services, people, places.

• Analyze money customer’s flow between non-bank to bank accounts

• Combine internal CRM data with enterprise and social media content

• Identify high-value customers across banking divisions

• Enhance new product/service opportunities

5

Applying Graph Analysis To Improve Customer Service


Oracle Graph DatabasesSupport Spatial and Graph use cases on every platform

Property GraphProperty Graph Property Graph

Oracle Database 12cOracle Big Data

Spatial and GraphDatabase Cloud Service (DBCS)Big Data Cloud Service (BDCS)

.


Property Graph Database:

• Scalability and Performance

• OLTP and/or OLAP

• Graph Query Language (PGQL)

• Graph analytics

• Standard interfaces

• Graph visualization

• Integration with machine learning tools

7

Courtesy Tom Sawyer Perspectives

Courtesy Linkurious


Blockchain: Permissioned, Replicated Shared Ledger

8


Terminology

9

• Blockchain

• Bitcoin

• ICO

• BTC, ETH, LiteCoin, …

• Satoshi (0.00000001 BTC)

• Nonce (number used once)

• SHA256 (a cryptographic hash/digest that generates 256 bits signature)

A few important names:

- Wei Dai

- Satoshi Nakamoto


Terminology (2)

10

• Mining- Doing the work of finding nonce s.t.

SHA256(SHA256(data+nonce)) < difficulty

- Difficulty adjusted every 2016 blocks (roughly 2 weeks)

- Purpose- Find the next block to append to the chain,

maintain consistency

- Create new bitcoins (get rewarded)

- Mining is computationally expensive- Trivial to validate results though

- Presentation of a block with the correct nonce value Proof of Work (POW)

https://i2.wp.com/1stminingrig.com/wp-content/uploads/2017/09/bitcoin-mining-farm-1.jpg

http://clipart-library.com/images/gceo4pXMi.jpg

Copyright © 2018 Oracle and/or its affiliates. All rights reserved. 11

set timing on

DECLARE

vcData VARCHAR2(2000) := ‘<Transaction Data>';

rawData RAW(20480); digest varchar2(20480); nonce integer;

BEGIN

for nonce in 0..90000 loop -- max 32Bits

rawData := utl_raw.cast_to_raw(vcData || nonce);

digest := to_char(dbms_crypto.hash(dbms_crypto.hash(rawData,

dbms_crypto.hash_sh256),dbms_crypto.hash_sh256));

if (instr(digest, '000') = 1) then

dbms_output.put_line(' nonce: ' || nonce);

dbms_output.put_line('SH2-256: ' || digest);

exit;

end if;

end loop;

END;

/

nonce: 451

SH2-256: 000E9DBD2D39D66F64991848BAD3A826F2FA1FB0210174B3D1FC98C51D380C72

PL/SQL procedure successfully completed.

Elapsed: 00:00:00.02

Terminology (3) – Mining Illustrated


Understanding Blockchain

12

Reference: https://anders.com/blockchain/blockchain.html


Motivations

13

• Bitcoins/Blockchain are fun

• Bitcoins can be used in many ways

• All bitcoin transactions are in public domain

• A great resource to go deep into real-world transactions


Happy Story about Bitcoin

14

Man buys $27 of bitcoin, forgets about them, finds they're now worth $886k

Bought in 2009, currency’s rise in value saw small investment turn into enough to buy an apartment in a wealthy area of Oslo

Kristoffer Koch invested 150 kroner ($26.60) in 5,000 bitcoins in 2009, after discovering them during the course of writing a thesis on encryption. He promptly forgot about them until

widespread media coverage of the anonymous, decentralised, peer-to-peer digital currency in April 2013 jogged his memory.

Bitcoins are stored in encrypted wallets secured with a private key, something Koch had forgotten. After eventually working out what the password could be, Koch got a pleasant surprise:

“It said I had 5,000 bitcoins in there. Measuring that in today’s rates it’s about NOK5m ($886,000),” Koch told NRK.

https://www.theguardian.com/technology/2015/dec/09/bitcoin-forgotten-currency-norway-oslo-home


Sad Story about Bitcoin

15

Bitcoin Pizza Day: Celebrating the Pizzas Bought for 10,000 BTC

Today, bitcoiners the world over will celebrate the anniversary of the most expensive pizzas in history.

Bought on 22nd May 2010 by Laszlo Hanyecz, the programmer paid a fellow Bitcoin Talk forum user 10,000 BTC for two Papa John’s pizzas. Back then – when the technology was just over a year

old – that equated to roughly $25, but is $5.12m by today’s exchange rate.

At bitcoin’s all-time high last December, the pizzas would have been worth an eye-watering $11.47m, making them likely candidates for the most expensive pizzas of all time.

Now widely recognised as the first real-world transaction with bitcoin, May 22nd has come to celebrate 'Bitcoin Pizza Day', with cryptocurrency enthusiasts raising a slice to Hanyecz’s infamous

hunger pangs that paved the way for early merchant adoption.

http://www.coindesk.com/bitcoin-pizza-day-celebrating-pizza-bought-10000-btc/


Bitcoins Can Be Used in Many Ways

16


Bitcoin As an Investment

17

Caution: highly volatile. Not for faint hearted. Risk from quantum computing


Bitcoin As an Investment

18

Caution: highly volatile. Not for faint hearted. Risk from quantum computing


Bitcoin for Financial Purpose

19

• Purchase things

• Pay ransomware (Wannacry)

• Move asset across border

• Store of value (as opposed to USD & inflation)

• Free of banking system

• Wealth privacy

• …

• The underlying blockchain technology has wider applications


Bitcoin for Dark Side

20

• Purchase drugs

• Demand ransomware (Wannacry) payment in bitcoin

• Be sure to use multiple distinct payment addresses

• Wannacry: 115p7UMMngoj1pMvkpHijcRdfJNXj6LrLn

• Hide wealth from government/tax department

• Bribe VIPs

•…


What Does a Bitcoin Transaction Look Like

21

• A transaction has input(s) and output(s)

• An input comes from an output of a transaction

TX hash : 6f7cf9580f1c2dfb3c4d5d043cdbb128c640e3f20161245aa7372e9666168516

TX outputSum : 10000000000

-- TX Input from : ff3dc8b461305acc5900d31602f2dafebfc406e5b050b14a352294f0965e0bf6:0

-- TX Input from : 2db69558056d0132d9848851fd20329be9cd590fa5ae2b3c55f58931f42e27f7:0

-- TX Output value 10000000000

-- TX Output scriPubAddr 12higDjoCCNXSA95xZMWUdPvXNmkAduhWv

Note: 1,000,000 is 0.01 BTC


What Does a Bitcoin Transaction Look Like

22

• A transaction has input(s) and output(s)

• An input comes from an output of a transaction

TX9

TX1

TX8

TX3

Addr X

Addr K

Addr L

Addr Y

$

$

$

$

$$

$

Addr Z

$


What Is Graph

23

• A set of vertices (entities), edges (relationships), and properties

• Graph is also known as linked data

TX9

TX1

TX8

TX3

Addr X

Addr K

Addr L

Addr Y

$

$

$

$

$$

$

Addr Z

$


Bitcoin Transaction As a Graph

24

• Graph Model 1• Transaction as vertices

• Address as vertices

• Transaction references as edges (Trans Trans, Trans Addr)



25

• Graph Model 2• Transaction as vertices

• Address as vertices

• Transaction’s indirect reference to Address as edges (Addr Trans Addr)



26

• Graph Model 3•Address as vertices

•Address to address payment as edges (Addr Addr)

Note: 1,000,000 is 0.01 BTC

TX9

TX1

TX8

TX3

Addr X

Addr K

Addr L

Addr Y

$

$

$

$

$$

$

Addr Z

$


Bitcoin Transaction As a Graph: Money Flow

27

• Graph Model 3• Address to address payment as edges (Addr Addr)

• What is Addr X’s contribution to Addr K?

Note: 1,000,000 is 0.01 BTC

TX9

TX1

TX8

TX3

Addr X

Addr K

Addr L

Addr Y

$

$

$

$

$$

$

Addr Z

$


Bitcoin Transaction As a Graph: Money Flow

28

• Graph Model 3• Address to address payment as edges (Addr Addr)

• What is Addr X’s contribution to Addr K?

• Given an input address i, output address o

• Contribution of i to o:

o

i i

i AmountAmount

Amount•


Bitcoin Transactions As Graph Workflow

29

Graph Generation

Data preparation

Graph Data loading

Graph vizquery and analytics

BlockchainBlock data parsing


Functions of a Graph Database


30

Graph Generation

Data preparation

Graph Data loading




Functions of a Graph Database


31

Graph Generation

Data preparation

Graph Data loading



OCI (BMC) &Oracle Database Cloud Service 12.2 (18.1)


Overview of Oracle’s Graph Database

32


Graph Data Access Layer (DAL)

Architecture of Property Graph Support

Graph Analytics

Blueprints & Lucene/SolrCloud RDF (RDF/XML, N-Triples, N-Quads,

TriG,N3,JSON)R

EST/We

b Se

rvice/N

oteb

oo

ks

Java, Gro

ovy, P

ytho

n, …

Java APIs

Java APIs/JDBC/SQL/PLSQL

Property Graph formats

GraphMLGML

Graph-SON Flat Files

33

Scalable and Persistent Storage Management

Parallel In-Memory Graph Analytics/Graph Query (PGX)

Oracle NoSQL DatabaseOracle RDBMS Apache HBase




Graph Analytics


TriG,N3,JSON)R

EST/We

b Se

rvice/N

oteb

oo

ks

Java, Gro

ovy, P

ytho

n, …

Java APIs



GraphMLGML


34







Graph Analytics


TriG,N3,JSON)R

EST/We

b Se

rvice/N

oteb

oo

ks

Java, Gro

ovy, P

ytho

n, …

Java APIs



GraphMLGML


35




Apache Spark




Graph Analytics


TriG,N3,JSON)R

EST/We

b Se

rvice/N

oteb

oo

ks

Java, Gro

ovy, P

ytho

n, …

Java APIs



GraphMLGML


36




Apache Spark




Graph Analytics


TriG,N3,JSON)R

EST/We

b Se

rvice/N

oteb

oo

ks

Java, Gro

ovy, P

ytho

n, …

Java APIs



GraphMLGML


37



Oracle NoSQL DatabaseApache HBase

Apache Spark


Property Graph with Oracle Database Cloud Service

38


Property Graph with Big Data Cloud Service

39


Rich set of built-in parallel graph algorithms … and parallel graph mutation operations

Computational Analytics: Built-in Package

Detecting Components and Communities

Tarjan’s, Kosaraju’s, Weakly Connected Components, Label Propagation (w/ variants), Soman and Narang’sSpacification

Ranking and Walking

Pagerank, Personalized Pagerank,Betwenness Centrality (w/ variants),Closeness Centrality, Degree Centrality,Eigenvector Centrality, HITS,Random walking and sampling (w/ variants)

Evaluating Community Structures

∑ ∑

Conductance, ModularityClustering Coefficient (Triangle Counting)Adamic-Adar

Path-Finding

Hop-Distance (BFS)Dijkstra’s, Bi-directional Dijkstra’sBellman-Ford’s

Link Prediction SALSA (Twitter’s Who-to-follow)

Other Classics Vertex CoverMinimum Spanning-Tree(Prim’s)

a

d

b e

g

c i

f

h

The original grapha

d

b e

g

c i

f

h

Create Undirected Graph

Simplify Graph

a

d

b e

g

c i

f

h

Left Set: “a,b,e”

a d

b

e

g

c

i

Create Bipartite Graph

ge b d i a f c h

Sort-By-Degree (Renumbering)

Filtered Subgraph

d

bg

i

e

40


Text Search through Apache Lucene/SolrCloud

Why?

– Contribute to the performance of graph traversal queries

– Constrained to be uniform in type among the indexed elements (vertices or edges)

42

Automatic Indexes

– Automatic update based on a subset of property keys– Avoid linear scan to access an element by key/value

Manual Indexes

–Maintained by users– Fasten up text searches by a particular key/value pair– Sub-graphs based on a set of (existing or temporary) properties


• Cytoscape supports Property Graph

• Connects to Oracle Database, Oracle NoSQL Database, or Apache HBase

• Runs Page Rank, Clustering, Shortest Path, etc

• Alternative to command-line for in-memory analytics once base graph created

Visualizing Property Graphs(with Cytoscape)


Additional Graph Visualization PartnersTomSawyer, Cambridge Intelligence, Linkurios, Vis.js,...


• SQL-like syntax but with graph pattern description and property access

– Interactive (real-time) analysis

– Supporting aggregates, comparison, such as max, min, order by, group by

• Finding a given pattern in graph– Fraud detection

– Anomaly detection

– Subgraph extraction

– ...

• Proposed for standardization by Oracle

– Specification available on-line

–Open-sourced front-end (i.e. parser)

Pattern matching using PGQL

https://github.com/oracle/pgql-lang


• Apache Zeppelin

– Multi-purpose notebook for data analysis and visualization

– Enables to embed interactive execution inside Browsers

– Renders execution results with plots and tables within Browsers

• PGX provides a hook (interpreter) for Zeppelin integration

46

Zeppelin Frontend


Interacting with the Graph

• Access through APIs

– Implementation of Apache Tinkerpop Blueprints APIs

– Based on Java, REST plus SolR Cloud/Lucene support for text search

– SQL/PLSQL for property graph functions in Oracle Database

• Scripting

– Groovy, Python, Javascript, ...

– Zeppelin integration, Javascript (Node.js) language binding

• Graphical UIs– Cytoscape, plug-in available for BDSG

– Commercial Tools such as TomSawyer Perspectives, Ogma


Some Details on Applying Graph Database to Bitcoin Transactions

48


Oracle Databasejoins, ID generation,

mapping, etc.


49

Graph Generation

Data preparation

Graph Data loading



12.2+


Oracle Databasejoins, ID generation,

mapping, etc.


50

Graph Generation

Data preparation

Graph Data loading



12.2+


Oracle Spatial and GraphProperty Graph:

RDBMS Table => Flat Files


51

Graph Generation

Data preparation

Graph Data loading



OraclePropertyGraphUtils.convertRDBMSTable2OPV(E)

12.2+





52

Graph Generation

Data preparation

Graph Data loading




12.2+





53

Graph Generation

Data preparation

Graph Data loading




12.2+



54

Graph Generation

Data preparation

Graph Data loading



12.2+

opgdl=OraclePropertyGraphDataLoader.getInstance();

vfile="/home/graph/btc.opv" // vertex flat file

efile="/home/graph/btc.ope" // edge flat file

opgdl.loadDataWithSqlLdr(opg, "pg", "pg", "orcl122",

vfile, efile, 8/*dop*/, false, "sqlldr",

true, "PDML=T,PDDL=T,NO_DUP=T,");

Oracle Spatial and GraphProperty Graph



55

Graph Generation

Data preparation

Graph Data loading



A subset of all bitcoin transactions was converted into a property graph:- 364K vertices- 751K edges



56

Graph Generation

Data preparation

Graph Data loading



3bda4918180fd55775a24580652f4c26d898d5840c7e71313491a05ef0b743d8


Querying Bitcoin Transactions

57

• Find “top” transactions with SQL

400,000 BTC’s!


Finding Critical Bitcoin Addresses

58

•Using Groovy with BDSG


Finding Bitcoin Addresses with Many Sends

59

•Using PGQLopg-nosql> pgxResultSet = pgxGraph.queryPgql("SELECT n, count(m) WHERE (n) -> (m) GROUP BY n ORDER BY count(m) DESC ")==>PgqlResultSetImpl[graph=btc,numResults=322783]opg-nosql> pgxResultSet.print(10);+---------------------------------+| n | count(m) |+---------------------------------+| PgxVertex[ID=273095] | 23164 || PgxVertex[ID=40669] | 7715 || PgxVertex[ID=154277] | 7580 || PgxVertex[ID=35950] | 4352 || PgxVertex[ID=127317] | 5521 || PgxVertex[ID=7367] | 2068 || PgxVertex[ID=45172] | 1830 || PgxVertex[ID=218422] | 1510 || PgxVertex[ID=181060] | 1450 || PgxVertex[ID=197650] | 1398 |+---------------------------------+==>nullopg-nosql> pgxResultSet.getNumResults()==>322783opg-nosql> opg.getVertex(273095)==>Vertex ID 273095 {bt_addr:str:1PJnjo4n2Rt5jWTUrCRr4inK2XmFPXqFC7}


Are Bitcoin Addresses Totally Anonymous?

60

• Yes but it depends …


Some Observations

61

•A few very active addresses


Some Observations (2)

62

•Activities with an interesting pattern


Summary and Future Work

63

• This talk covered• Blockchain/Bitcoin transactions

• Oracle Spatial and Graph Property Graph database

• Techniques used for graph modeling, query, visualizations are easily applicable to other domains

• Banking, logistics network, asset network, social network, security, …

• Future work• Add temporal (timestamp)

• Merge public addresses

• Identify “true value”

• Maybe convert bitcoin amount into USD?

• Detect anomaly /fraud

• Decentralized system, no “bank” or “government” to complain about abnormal activities


Resources on Oracle‘s Property Graph Database• Oracle Spatial and Graph Property Graph

http://www.oracle.com/technetwork/database/options/spatialandgraph/overview/rdfsemantic-graph-1902016.html

• Oracle Spatial and Graph Blog

https://blogs.oracle.com/oraclespatial/

• Getting Started – Creating a Property Graph on Oracle Database by Arthur Dayton from Vlamis Software Solutions

https://blogs.oracle.com/oraclespatial/getting-started-creating-a-property-graph-on-oracle-database

• Oracle Big Data Spatial and Graph on Oracle.com:www.oracle.com/database/big-data-spatial-and-graph

• OTN product page (white papers, software downloads, documentation, tutorials): www.oracle.com/technetwork/database/database-technologies/bigdata-spatialandgraph

• Oracle Big Data Lite Virtual Machine - a free sandbox to get started: www.oracle.com/technetwork/database/bigdata-appliance/oracle-bigdatalite-2104726.html

• Hands On Lab for Big Data Spatial: tinyurl.com/BDSG-HOL

• Blog – examples, tips & tricks: blogs.oracle.com/bigdataspatialgraph

64

http://www.oracle.com/technetwork/database/options/spatialandgraph/overview/rdfsemantic-graph-1902016.html

https://blogs.oracle.com/oraclespatial/

https://blogs.oracle.com/oraclespatial/getting-started-creating-a-property-graph-on-oracle-database

https://www.oracle.com/database/big-data-spatial-and-graph

http://www.oracle.com/technetwork/database/database-technologies/bigdata-spatialandgraph

http://www.oracle.com/technetwork/database/bigdata-appliance/oracle-bigdatalite-2104726.html

http://tinyurl.com/BDSG-HOL

https://blogs.oracle.com/bigdataspatialgraph/


Resources

• Online communities

– Oracle Spatial & Graph SIG user groups (search “Oracle Spatial Community”)

– @SpatialHannes, @JeanIhm, @alanzwu - follow the S&G team

• Spatial and Graph AskTOM Office Hours (every month)– https://devgym.oracle.com/pls/apex/dg/office_hours/3084

65

https://devgym.oracle.com/pls/apex/dg/office_hours/3084

Documents

Analyzing Blockchain and Bitcoin Transaction Data as Graph...Financial Services •Model customer relationship to products, services, people, places. •Analyze money customer [s flow