Upload
dhaya
View
217
Download
0
Embed Size (px)
Citation preview
7/31/2019 Halloway EvidentCodeAtScale
1/98
evident
late 14c., from O.Fr. evident and directly from L.evidentem (nom. evidens) "perceptible, clear,
obvious, apparent" from ex- "fully, out of" (see ex-)
+ videntem (nom. videns), prp. of videre "to
see" (see vision).
http://www.etymonline.com/index.php?term=evident
1
http://www.etymonline.com/index.php?term=evidenthttp://www.etymonline.com/index.php?term=evidenthttp://www.etymonline.com/index.php?term=vision&allowed_in_frame=0http://www.etymonline.com/index.php?term=vision&allowed_in_frame=0http://www.etymonline.com/index.php?term=ex-&allowed_in_frame=0http://www.etymonline.com/index.php?term=ex-&allowed_in_frame=0http://www.etymonline.com/index.php?term=evident&allowed_in_frame=0http://www.etymonline.com/index.php?term=evident&allowed_in_frame=07/31/2019 Halloway EvidentCodeAtScale
2/98
7/31/2019 Halloway EvidentCodeAtScale
3/98
3
7/31/2019 Halloway EvidentCodeAtScale
4/98
declarativeprogramming
4
7/31/2019 Halloway EvidentCodeAtScale
5/98
In computer science, declarative programming is aprogramming paradigm that expresses the logic of a
computation without describing its control flow.[1] Many
languages applying this style attempt to minimize or
eliminate side effects by describing whatthe program should
accomplish, rather than describing how to go aboutaccomplishing it.
http://en.wikipedia.org/wiki/Declarative_programming
5
http://en.wikipedia.org/wiki/Declarative_programminghttp://en.wikipedia.org/wiki/Declarative_programminghttp://en.wikipedia.org/wiki/Declarative_programminghttp://en.wikipedia.org/wiki/Side_effect_%28computer_science%29http://en.wikipedia.org/wiki/Side_effect_%28computer_science%29http://en.wikipedia.org/wiki/Declarative_programming#cite_note-0http://en.wikipedia.org/wiki/Declarative_programming#cite_note-0http://en.wikipedia.org/wiki/Control_flowhttp://en.wikipedia.org/wiki/Control_flowhttp://en.wikipedia.org/wiki/Computationhttp://en.wikipedia.org/wiki/Computationhttp://en.wikipedia.org/wiki/Programming_paradigmhttp://en.wikipedia.org/wiki/Programming_paradigmhttp://en.wikipedia.org/wiki/Computer_sciencehttp://en.wikipedia.org/wiki/Computer_science7/31/2019 Halloway EvidentCodeAtScale
6/98
rock/paper/scissors/
lizard/Spock
http://en.wikipedia.org/wiki/Rock-paper-scissors-lizard-Spock
6
http://en.wikipedia.org/wiki/Rock-paper-scissors-lizard-Spockhttp://en.wikipedia.org/wiki/Rock-paper-scissors-lizard-Spock7/31/2019 Halloway EvidentCodeAtScale
7/98
(defdominates{:paper:rock:rock:scissors
:scissors:paper})
declarative
7
7/31/2019 Halloway EvidentCodeAtScale
8/98
(defdominates{:paper:rock:rock:scissors
:scissors:paper})
declarative, functional
(defnwinner [play-1 play-2] (cond (= play-1 play-2) nil (= (dominates play-1) play-2) play-1 :else play-2))
8
7/31/2019 Halloway EvidentCodeAtScale
9/98
so everything should bedeclarative, right?
9
7/31/2019 Halloway EvidentCodeAtScale
10/98
some reasons not tobe declarative
functional requirement
non-functional requirement
constraint
10
7/31/2019 Halloway EvidentCodeAtScale
11/98
abstraction
11
7/31/2019 Halloway EvidentCodeAtScale
12/98
ideal numberof methods
0
12
7/31/2019 Halloway EvidentCodeAtScale
13/98
publicinterfaceMap { intsize(); booleanisEmpty(); booleancontainsKey(java.lang.Objecto); booleancontainsValue(java.lang.Objecto);
Vget(java.lang.Objecto); Vput(Kk, Vv); Vremove(java.lang.Objecto); voidputAll(java.util.Map
7/31/2019 Halloway EvidentCodeAtScale
14/98
publicclassListTablesRequest { public ListTablesRequest(); publicjava.lang.StringgetExclusiveStartTableName(); publicvoidsetExclusiveStartTableName
(java.lang.StringexclusiveStartTableName); publicListTablesRequestwithExclusiveStartTableName (StringexclusiveStartTableName); publicjava.lang.IntegergetLimit(); publicvoidsetLimit(java.lang.Integerlimit); publicListTablesRequestwithLimit(Integerlimit); publicjava.lang.StringtoString();
publicinthashCode(); publicbooleanequals(Objectobj);}
concretion
14
7/31/2019 Halloway EvidentCodeAtScale
15/98
Abstractions may be formed by reducingthe information content of a concept or
an observable phenomenon...
http://en.wikipedia.org/wiki/Abstraction
15
http://en.wikipedia.org/wiki/Abstractionhttp://en.wikipedia.org/wiki/Abstractionhttp://en.wikipedia.org/wiki/Phenomenonhttp://en.wikipedia.org/wiki/Phenomenonhttp://en.wikipedia.org/wiki/Concepthttp://en.wikipedia.org/wiki/Concepthttp://en.wikipedia.org/wiki/Informationhttp://en.wikipedia.org/wiki/Information7/31/2019 Halloway EvidentCodeAtScale
16/98
v1
F
v2
F
v3
F
v4
epochal time model
16
7/31/2019 Halloway EvidentCodeAtScale
17/98
v1
F
v2
F
v3
F
v4
abstraction here reduces
the information I must provide
abstraction here reduces
the information I can get17
7/31/2019 Halloway EvidentCodeAtScale
18/98
abstractions in Datomic
about 50 protocols / interfaces
(inter)action
between services
between components
perception some new (generic!) data structures
no concretions
18
7/31/2019 Halloway EvidentCodeAtScale
19/98
publicinterfaceEntity {
Objectget(Objectkey);Set keySet();
}
perception example
19
7/31/2019 Halloway EvidentCodeAtScale
20/98
7/31/2019 Halloway EvidentCodeAtScale
21/98
number of db
update methods inDatomic
2
21
7/31/2019 Halloway EvidentCodeAtScale
22/98
abstraction guidelines
fine grained
separate perception and action
perception is generic
22
7/31/2019 Halloway EvidentCodeAtScale
23/98
values
23
7/31/2019 Halloway EvidentCodeAtScale
24/98
7/31/2019 Halloway EvidentCodeAtScale
25/98
[?customer :email ?email]
Constrains the results returned,binds variables
data pattern
25
7/31/2019 Halloway EvidentCodeAtScale
26/98
[?customer :email ?email]
Constrains the results returned,binds variables
data pattern
entity attribute value
26
7/31/2019 Halloway EvidentCodeAtScale
27/98
[?customer :email ?email]
Constrains the results returned,binds variables
data pattern
constant
27
7/31/2019 Halloway EvidentCodeAtScale
28/98
[?customer :email ?email]
Constrains the results returned,binds variables
data pattern
variablevariable
28
7/31/2019 Halloway EvidentCodeAtScale
29/98
[42:email ?email]
Find a particular customers email
constants anywhere
29
7/31/2019 Halloway EvidentCodeAtScale
30/98
[42 ?attribute]
What other attributes does
customer 42 have?
variables anywhere
30
7/31/2019 Halloway EvidentCodeAtScale
31/98
7/31/2019 Halloway EvidentCodeAtScale
32/98
[:find?customer:where [?customer :email]]
find clause
variable to
return
32
7/31/2019 Halloway EvidentCodeAtScale
33/98
[:find ?customer:where [?customer:email]
[?customer:orders]]
Find all the customers who
have placed orders.
implicit join
33
7/31/2019 Halloway EvidentCodeAtScale
34/98
declarative < evident
declarative evident
tables, documents datoms
ORM maps, entities
ambient db explicit db
34
7/31/2019 Halloway EvidentCodeAtScale
35/98
(use '[datomic.api :only (q) :as d])
(q '[:find ?customer :where [?customer :id]
[?customer :orders]]
db)
API
35
7/31/2019 Halloway EvidentCodeAtScale
36/98
(use '[datomic.api :only (q) :as d])
(q '[:find ?customer :where [?customer :id]
[?customer :orders]]
db)
q
36
7/31/2019 Halloway EvidentCodeAtScale
37/98
7/31/2019 Halloway EvidentCodeAtScale
38/98
(q '[:find ?customer
:where [?customer :id][?customer :orders]]
db)
input(s)
38
7/31/2019 Halloway EvidentCodeAtScale
39/98
:in $database ?email
in clause
Names inputs so you can refer to themelsewhere in the query
39
7/31/2019 Halloway EvidentCodeAtScale
40/98
Find a customer by email.
parameterized query
(q '[:find ?customer
:in $database ?email
:where [$database ?customer :email ?email]]
db
40
mailto:[email protected]:[email protected]7/31/2019 Halloway EvidentCodeAtScale
41/98
Find a customer by email.
first input
(q '[:find ?customer
:in$database ?email
:where [$database ?customer :email ?email]]
db
41
mailto:[email protected]:[email protected]7/31/2019 Halloway EvidentCodeAtScale
42/98
Find a customer by email.
second input
(q '[:find ?customer
:in $database ?email
:where [$database ?customer :email?email]]
db
42
mailto:[email protected]:[email protected]7/31/2019 Halloway EvidentCodeAtScale
43/98
Find a customer by email.
verbose?
(q '[:find ?customer
:in$database ?email
:where [$database ?customer :email ?email]]
db
43
mailto:[email protected]:[email protected]7/31/2019 Halloway EvidentCodeAtScale
44/98
Find a customer by email.
shortest name possible
(q '[:find ?customer
:in$ ?email
:where [$ ?customer :email ?email]]
db
44
mailto:[email protected]:[email protected]7/31/2019 Halloway EvidentCodeAtScale
45/98
Find a customer by email.
elide $ in where
(q '[:find ?customer
:in$ ?email
:where [ ?customer :email ?email]]
db
no need to
specify $
45
mailto:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]7/31/2019 Halloway EvidentCodeAtScale
46/98
[(< 50 ?price)]
predicates
Functional constraints that canappear in a :where clause
46
7/31/2019 Halloway EvidentCodeAtScale
47/98
[:find ?item:where [?item:item/price ?price]
[(< 50 ?price)]]
Find the expensive items
adding a predicate
47
7/31/2019 Halloway EvidentCodeAtScale
48/98
[(shipping ?zip ?weight) ?cost]
Take bound variables as inputsand bind variables with output
functions
48
7/31/2019 Halloway EvidentCodeAtScale
49/98
[(shipping ?zip ?weight) ?cost]
function args
function arguments
49
7/31/2019 Halloway EvidentCodeAtScale
50/98
[(shipping ?zip ?weight)?cost]
function returns
bind return
values
50
7/31/2019 Halloway EvidentCodeAtScale
51/98
[:find ?customer ?product:where [?customer :shipAddress ?addr]
[?addr :zip ?zip][?product :product/weight ?weight]
[?product :product/price ?price][(Shipping/estimate ?zip ?weight) ?shipCost][(
7/31/2019 Halloway EvidentCodeAtScale
52/98
publicclassShipping { publicstaticBigDecimalestimate(Stringzip1, intpounds);
}
Functions can be plainJVM code.
arbitrary functions
52
7/31/2019 Halloway EvidentCodeAtScale
53/98
(q '[:find ?op ?e ?a ?v
:in $ ?renamings :where[?person :customer/interests][(.entity $ ?person) ?entity][(datomize ?entity ?renamings)
[[?op ?e ?a ?v]]]](db customer-conn)renamings)
find people with interests
53
7/31/2019 Halloway EvidentCodeAtScale
54/98
(q '[:find ?op ?e ?a ?v
:in $ ?renamings :where[?person :customer/interests][(.entity $ ?person) ?entity][(datomize ?entity ?renamings)
[[?op ?e ?a ?v]]]](db customer-conn)renamings)
as entities
54
7/31/2019 Halloway EvidentCodeAtScale
55/98
(q '[:find?op ?e ?a ?v
:in $ ?renamings :where[?person :customer/interests][(.entity $ ?person) ?entity][(datomize ?entity ?renamings)
[[?op ?e ?a ?v]]]](db customer-conn)renamings)
and make ... datoms?
55
7/31/2019 Halloway EvidentCodeAtScale
56/98
(->>(q '[:find ?op ?e ?a ?v :in $ ?renamings :where
[?person :customer/interests][(.entity $ ?person) ?entity][(datomize ?entity ?renamings)[[?op ?e ?a ?v]]]]
(db customer-conn)renamings)
seq (d/transact hobbies-conn))
ETL job
56
7/31/2019 Halloway EvidentCodeAtScale
57/98
->>
the ETL macro
57
7/31/2019 Halloway EvidentCodeAtScale
58/98
schema is data
58
7/31/2019 Halloway EvidentCodeAtScale
59/98
{:db/id #db/id[:db.part/db]
:db/ident:part/name:db/valueType:db.type/string:db/cardinality:db.cardinality/one:db/fulltext true:db/doc"The name of the part"
:db.install/_attribute:db.part/db}
attribute schema
59
7/31/2019 Halloway EvidentCodeAtScale
60/98
(create-eav :part/business-key :db.type/string :db.cardinality/one "The name of the part" :db.part/db)
why not this?
60
7/31/2019 Halloway EvidentCodeAtScale
61/98
value propositions
powerful extant API
make from any language
read without evaluation
names, not positions
extend easily
build from programs
61
7/31/2019 Halloway EvidentCodeAtScale
62/98
navigation is data
62
7/31/2019 Halloway EvidentCodeAtScale
63/98
publicinterfaceEntity { Objectget(Objectkey);
Set keySet();// evil enhancementObject reverseGet(Object key);
}
Entity revisited
63
7/31/2019 Halloway EvidentCodeAtScale
64/98
(get-in jack [:town])
Jacks town
64
7/31/2019 Halloway EvidentCodeAtScale
65/98
(get-in jack [:town :_town])
who else is in Jacks town?
65
7/31/2019 Halloway EvidentCodeAtScale
66/98
number of methods inDatomics graph
navigation API
066
7/31/2019 Halloway EvidentCodeAtScale
67/98
evident code is not enough
specifications
doc strings
drawings (graffle)
outlines (org, Confluence)
tests (???)
67
7/31/2019 Halloway EvidentCodeAtScale
68/98
tests are executabledocumentation (?)
68
7/31/2019 Halloway EvidentCodeAtScale
69/98
what is a test?
create input
driver
capture output
validate
communicate
record
69
7/31/2019 Halloway EvidentCodeAtScale
70/98
(are [x y] (= x y) (+) 0 (+ 1) 1 (+ 1 2) 3 (+ 1 2 3) 6
(+ -1) -1 (+ -1 -2) -3 (+ -1 +2 -3) -2
(+ 1 -1) 0 (+ -1 1) 0
(+ 2/3) 2/3 (+ 2/3 1) 5/3 (+ 2/3 1/3) 1 )
test, complected
validation
hand-curated
inputs
driver
outputs
70
7/31/2019 Halloway EvidentCodeAtScale
71/98
generative testing
71
7/31/2019 Halloway EvidentCodeAtScale
72/98
http://en.wikipedia.org/wiki/Mastermind_(board_game)
72
http://en.wikipedia.org/wiki/Mastermind_(board_gamehttp://en.wikipedia.org/wiki/Mastermind_(board_game7/31/2019 Halloway EvidentCodeAtScale
73/98
(defnrandom-secret[]
(gen/vec #(gen/one-of :r:g:b:y) 4))
input generation
73
7/31/2019 Halloway EvidentCodeAtScale
74/98
(defnscore[c1 c2]
(let [exact (exact-matches c1 c2) ums (unordered-matches c1 c2)unordered (apply + (vals ums))]
{:exact exact:unordered(- unordered exact)}))
fn under test
74
7/31/2019 Halloway EvidentCodeAtScale
75/98
(defnscoring-is-symmetric[secret guess score]
(= score (game/score guess secret)))
(defnscoring-is-bounded-by-number-of-pegs[secret guess score]
(< 0 (matches score)(count secret)))
validation
75
7/31/2019 Halloway EvidentCodeAtScale
76/98
(defspec score-invariants
game/score[^{:tag `random-secret} secret^{:tag `random-secret} guess]
(assert (scoring-is-symmetric secret guess %)) (assert (scoring-is-bounded-by-number-of-pegs
secret guess %)))
driver
inputs
validations
fn under test
76
7/31/2019 Halloway EvidentCodeAtScale
77/98
========================================================:secret | :guess | :score========================================================
(:r :r :r :r) | (:r :r :r :r) | {:exact 4, :unordered 0}(:r :r :r :r) | (:r :r :r :g) | {:exact 3, :unordered 0}(:r :r :r :r) | (:r :r :r :b) | {:exact 3, :unordered 0}(:r :r :r :r) | (:r :r :r :y) | {:exact 3, :unordered 0}
...
...or just enumerate all cases
77
7/31/2019 Halloway EvidentCodeAtScale
78/98
test inputs and outputsare data
78
7/31/2019 Halloway EvidentCodeAtScale
79/98
:dying "... .O.
..."
:off "O.O...O.O"
:on "|||
O.O|||"
visual inputs
http://thinkrelevance.com/blog/2009/10/07/brians-functional-brain-take-1-5
79
http://thinkrelevance.com/blog/2009/10/07/brians-functional-brain-take-1-5http://thinkrelevance.com/blog/2009/10/07/brians-functional-brain-take-1-5http://thinkrelevance.com/blog/2009/10/07/brians-functional-brain-take-1-57/31/2019 Halloway EvidentCodeAtScale
80/98
==============================================================================:input | :bind-form | :bind-type | :count | ?a |==============================================================================42 | ?a | scalar | 1 | 42 |
[2 3 5 7] | ?a | scalar | 1 | [2 3 5 7] |42 | [?a ?b] | tuple | | |[2 3 5 7] | [?a ?b] | tuple | 1 | 2 |42 | [?a ...] | collection | | |[2 3 5 7] | [?a ...] | collection | 4 | 3 / 2 / 7 / 5 |42 | [[?a ?b ?c]] | relation | | |[2 3 5 7] | [[?a ?b ?c]] | relation | | |==============================================================================
tests write docs
80
7/31/2019 Halloway EvidentCodeAtScale
81/98
once you add time,stateful interactions are
81
7/31/2019 Halloway EvidentCodeAtScale
82/98
once you add time,
stateful interactions are
data!
82
7/31/2019 Halloway EvidentCodeAtScale
83/98
===============================:call | :result===============================(put q :one) | true
(offer q :two) | true(offer q :three) | false(offer q :three 10) | false(take q) | :one(poll q :missing) | :two(poll q :missing) | :missing
(poll q :missing 10) | :missing===============================
time series table
83
7/31/2019 Halloway EvidentCodeAtScale
84/98
---------------------------------------------------------------Legend: incrementing value on reconnect, or [T]ime[O]utreconnects: [A]ttempt [S]ucceed [F]ail [B]ackoff [Interrupt]
[T]throw [Z]sleep [C]leanup c[L]eanupFailed
---------------------------------------------------------------
0 A L Z B A I S1 C A S
interleaving
84
7/31/2019 Halloway EvidentCodeAtScale
85/98
...........................................
...........................................
..............
{:retries 9,:blocks-planned 100,:blocks-completed #,:storage-fail-rate 0.05}
simulating failure
85
7/31/2019 Halloway EvidentCodeAtScale
86/98
...........................................
...........................................
..............
{:retries 9,:blocks-planned 100,:blocks-completed #,:storage-fail-rate 0.05}
simulating failure
possible sources:
timing
logs
metrics86
7/31/2019 Halloway EvidentCodeAtScale
87/98
tests are programs
87
7/31/2019 Halloway EvidentCodeAtScale
88/98
(defnrun-with-transcript[forms]
(binding [*ns* *ns*] (let [temp (gensym)]
(in-ns temp) (clojure.core/use 'clojure.core) (doseq [f forms] (pprint f) (print "=> ") (pprint (eval f))
(println)) (remove-ns temp) :done)))
from REPL to regression
88
7/31/2019 Halloway EvidentCodeAtScale
89/98
the sim
89
Setup
7/31/2019 Halloway EvidentCodeAtScale
90/98
Execution
p
Sim DB
SimData
sim-schema.dtm
orders-schema.dtm
SImGenerator
OrdersGenerator
Runner
Feeder
Time
Logs
Metrics
Validation
Validator
Orders DB
OrdersData
90
Setup
7/31/2019 Halloway EvidentCodeAtScale
91/98
Execution
p
Sim DB
SimData
sim-schema.dtm
orders-schema.dtm
SImGenerator
OrdersGenerator
Runner
Feeder
Time
Logs
Metrics
Validation
Validator
Orders DB
OrdersData
91
Setup
7/31/2019 Halloway EvidentCodeAtScale
92/98
Execution
p
Sim DB
SimData
sim-schema.dtm
orders-schema.dtm
SImGenerator
OrdersGenerator
Runner
Feeder
Time
Logs
Metrics
Validation
Validator
Orders DB
OrdersData
92
Setup
7/31/2019 Halloway EvidentCodeAtScale
93/98
Execution
p
Sim DB
SimData
sim-schema.dtm
orders-schema.dtm
SImGenerator
OrdersGenerator
Runner
Feeder
Time
Logs
Metrics
Validation
Validator
Orders DB
OrdersData
93
l
7/31/2019 Halloway EvidentCodeAtScale
94/98
simulation
not a precious curated path
uses real input and output points
models time
separates testing activities in time and space
validates via logic programs
some effort to set up
94
7/31/2019 Halloway EvidentCodeAtScale
95/98
how Clojureenabled Datomic
95
Cl j f
7/31/2019 Halloway EvidentCodeAtScale
96/98
Clojure features
data
functions
references
protocols
deftype
96
Cl j hil h
7/31/2019 Halloway EvidentCodeAtScale
97/98
Clojure philosophy
architecture
simplicity
power
focus
97
7/31/2019 Halloway EvidentCodeAtScale
98/98