Event Sourcing as the Foundation of Traceability · Event Sourcing for about a year. @tbfangel...

IDA Driving IT Århus 2020Thomas Bøgh Fangel @tbfangel

Event Sourcing asthe Foundation of Traceability

@tbfangel

MeThomas Bøgh Fangel

@tbfangel

Architect at Lunar since 2016

Distributed systems since 2004

Java, Scala, Typescript, Go

Event Sourcing for about a year

@tbfangel

Agenda● Building a bank from scratch○ Tech vision

● Event Sourcing○ Why & How○ PatternsAgenda

@tbfangel

100+Employees

25%Engineers

1M+Tx pr month

~100μ services

150kUsers

3K8S clusters

LunarFounded

2015Live2016

Smartphone onlyChallenger Bank

@tbfangel

Building a bank from scratch

@tbfangel

CardProcessor

NationalClearing

What does it mean to be a bank?

Pay with card

Transfer money

Keep money safe

@tbfangel

Trustworthy Secure

Correct

Tech Vision

@tbfangel

Traceability at all levels

nothing should happen without us knowing and our system should never be in a state

we cannot explain

Tech Vision

@tbfangel

≠Traceability

Distributed Tracing

@tbfangel

func DoSomeBusiness(s *State) (Result, error) {

if s.InImpossibleState() {//this should never happen - what to do?panic(“current state is impossible”)

}//happy cases below

}

@tbfangel

@tbfangel

Production=

The place where the impossible happens

@tbfangel

Traceability=

Explain the impossible

@tbfangel

“Something really awful happened to your money - but

we know exactly where the money is and we will fix it”

@tbfangel

Event Sourcing

@tbfangel

Shift of Focus

From state…...to events

@tbfangel

Event Sourcing Components

Storage

Event stream

Projection

Side effects

Handler

Event stream

Projection

+

+Business logic

(pure functions)

Aggregate root

@tbfangel

Implementation● Apogee: ES & CQRS* library in Go

(Open Source in 2020)● Postgres as event storage● In-memory views for AR● SQL backed views

* CQRS: Command Query Responsibility Segregation

@tbfangel

Patterns

@tbfangel

Adapter NationalClearing

Public APIs

Bank Coreasync sync

@tbfangel

External Event Streams● Public API● Same guarantees and properties

as internal event stream● Derived from internal event stream● Essentially a handler writing to an

event stream ● May keep track of source events

@tbfangel

External Event Streams

Storage

InternalEvent stream

Projection

Handler External Event stream A

External Event stream B

@tbfangel

Distributed flows● Distributed transactions● Represented as aggregate roots

(state machines)● State determines handler action● Side effects in handlers● Public event streams as API

@tbfangel

TRANSFER

Bank Core

TRANSFER

DK Adapter

CLEARING

ACCOUNT

TRANSFER

Handler Aggregate

Request Account Bank Transfer Adapter Transfer

1. TransferInitiated FundsReserved

1.

@tbfangel

TRANSFER

Bank Core

TRANSFER

DK Adapter

CLEARING

ACCOUNT

TRANSFER

Handler Aggregate

Request Account Bank Transfer Adapter Transfer

1. TransferInitiated FundsReserved

2. Initiated

2.

@tbfangel

TRANSFER

Bank Core

TRANSFER

DK Adapter

CLEARING

ACCOUNT

TRANSFER

Handler Aggregate

Request Account Bank Transfer Adapter Transfer

1. TransferInitiated FundsReserved

2. Initiated

3.

@tbfangel

TRANSFER

Bank Core

TRANSFER

DK Adapter

CLEARING

ACCOUNT

TRANSFER

Handler Aggregate

Request Account Bank Transfer Adapter Transfer

1. TransferInitiated FundsReserved

2. Initiated 4. Initiated

4.

@tbfangel

TRANSFER

Bank Core

TRANSFER

DK Adapter

CLEARING

ACCOUNT

TRANSFER

Handler Aggregate

Request Account Bank Transfer Adapter Transfer

1. TransferInitiated FundsReserved

2. Initiated 4. Initiated

5. TimedOut

5.

@tbfangel

TRANSFER

Bank Core

TRANSFER

DK Adapter

CLEARING

ACCOUNT

TRANSFER

Handler Aggregate

Request Account Bank Transfer Adapter Transfer

1. TransferInitiated FundsReserved

2. Initiated 4. Initiated

5. TimedOut

6. Succeeded

6.

@tbfangel

TRANSFER

Bank Core

TRANSFER

DK Adapter

CLEARING

ACCOUNT

TRANSFER

Handler Aggregate

Request Account Bank Transfer Adapter Transfer

1. TransferInitiated FundsReserved

2. Initiated 4. Initiated

5. TimedOut

6. Succeeded

7.

@tbfangel

TRANSFER

Bank Core

TRANSFER

DK Adapter

CLEARING

ACCOUNT

TRANSFER

Handler Aggregate

Request Account Bank Transfer Adapter Transfer

1. TransferInitiated FundsReserved

2. Initiated 4. Initiated

8. Succeeded 5. TimedOut

6. Succeeded

8.

@tbfangel

TRANSFER

Bank Core

TRANSFER

DK Adapter

CLEARING

ACCOUNT

TRANSFER

Handler Aggregate

Request Account Bank Transfer Adapter Transfer

1. TransferInitiated FundsReserved

2. Initiated 4. Initiated

9. TransferCompleted ReservationRemoved TransactionPosted

8. Succeeded 5. TimedOut

6. Succeeded

9.

@tbfangel

TRANSFER

Bank Core

TRANSFER

DK Adapter

CLEARING

ACCOUNT

TRANSFER

Handler Aggregate

Request

9.

1.

2.

3. 4.5.+ 6.

7.8.

Account Bank Transfer Adapter Transfer

1. TransferInitiated FundsReserved

2. Initiated 4. Initiated

9. TransferCompleted ReservationRemoved TransactionPosted

8. Succeeded 5. TimedOut

6. Succeeded

@tbfangel

Error handling● Failures are 1st class domain citizens● Idempotency* crucial both internally and

externally ● Only measure against timeouts and

crashes

* Idempotency: system state remains the same after one or several calls

@tbfangel

Tech VisionRevisited

@tbfangel

What about Correctness...?● 100% correctness is impossible● Understand your errors, then fix● Traceability leads to correctness by

explaining errors● Error correction is just another event

@tbfangel

… and Consistency?● Ordered event streams● Empowers consumers ● Eventually is better than maybe

@tbfangel

Thank You!https://tech.lunarway.com/blog/

https://tech.lunarway.com/talks/

https://tech.lunarway.com/opensource/