Embed Size (px)
Citation preview
— IL I
Belastingdienst Master lnformaUon Hub
Operational Handbook for theMaster Information Hub
OOCIRT1InL 6LI MIN Op4adbdoc
Page I of 11
3oturnent Htory
Revision History
Revision Revision Date Summary of Changes ChangesNumber marked
vO.1 23-08-2013 Initial document draft (TOC) n/avO.2 28-08-2013 Added scheduling and batch job content, plus recovery N
olacehotdersvO.3 04-09-2013 Minor changes NvO.4 10-09-2013 CombIned chapters on batch procedures, added solution context N
and ACL. Version to be reviewed Informally before meeting onSept17. More official version 0,5 schedu]epj,j3.
vO,5 12-09-2013 Processed review comments NAdded initial set of shutdown/start commands (Appendix A andB)
vO,7 19-09-2013 Processed review comment of. NDetailed Chapter 7.
vO.8 26-09-2013 Detailed password expiry NAdditIonal detail in Chapter 7Processed review comments of
vO.9 07-10-2013 Processed review comments of NvO.91 13-12-2013 Processed review comments of Belastingdienst Nvl.0 05-06-2014 Adjusted to address Trac no. 327 N
Adjusted as TWS Is used to schedulingV1.1 20-06-2014 ETL updates to sections: N
- 4.5/4.2.5/4.3.2/ 4.3.3• New section 6 on ETI Functional Maintenance
V2.0 27-06-2014 MDM portion of operational manual updated and version raised Nh whole number
V2.1 30-06-2014 Audit trail enabling I disabling aspects are added NV2.2 17/07/2014 Document modified through out to fix trac # 327,373, 374, 358. N
MDM scheduling information provided by Art is Incorporated.MDM Deployment rollback is documented
V3.0 17/07/2014 Version uplift for UAT release NV3.1 31/07/2014 Review comments from -are processed NV3.2 21/09/2014 Peviewcommentsfmm •areproce d NV3.3 27/09/2014 MDM part of final set of review comment from am processed NV3.4 22110/2014 ntegrated chapter 8 from. • NV3.5 29110/2014 Updated section 6.1.5 step 6— login with user isadmin NV3.6 08111/2014 Added subsections 5.4.1.5 and 5.4.2.2 from an older version of N
Operational Handbook (v4.0 of 30/07/2014) statIng: ‘Includesection 5.4 for #346 and uplifted the version for 2014.04.C3release”
V3.7 07/11/2014 Reformatted Backup and Recovery procedures NV3.8 10111/2014 Removed Chapter Service Recovery because Chapter 7 is a full
replacemeni Removed Chapter about isRecavery. This chapteris no longer necessary due to the automatic generation of theisRecovery confrgurationfile.
V3.9 11/11/2014 ModIfied Exit Code Table In section 4.4.5 (Defect 501, 504 and N508)
Page 201 13
V4.0 13/11/2014 Chapter 4.4.5 second tabte on page 44. ND.i C laioi tlflj (J1 tODjChapter with name Rejected Records Reloading in MDM isntroduced
V4.1 14/11/2014 Added Appendix C: How to Identify defaults as a result of NREFDATA lookup (defect #460)
V4.2 112014 Updated Appendix C: Added some examples for clarification NV4.3 2/2014 ModIfied Exit Code Table In section 4.4.5 (Defect #592) NV4.4 r!21 04 Embedded entire PowerPoints of B&R scenarios to Chapter 7 NV4.5 0/ 2/2014 Update Chapter 7 with cleat instructions on how to use the N
embedded PowerPointsV4.6 2/2014 Update Chapter 7 a copy of the Execution Steps per scenario Is N
included so that the printed copy will also contain the execution_____ stes.
V4.7 1/2015 ad a note regarding reject files cleanup in section 4.6 NV4.8 1/2015 ad return code 4 in sections 4.5.3.1 and 4.5.3.5 NV4.9 /2015 wsection:6.7 N‘15.0 /2015 ited section 4.5 to mention housekeeping script 095 NV5.1 /2015 ctldn added for http_access.Iog NV6.0 812015 mplete revIew forMlH 2.0 NV6.1 9/2015 ted section 4.3.4 related to #695 NV6.2 1/2015 ascribed MANAGE_SRC_FILES.sh In 4.5.2 and 4.5.3 N
rLSçriptsreviw-•V64 19/0212016 STG OUT Efl. RUN ID PARAUS must contain a comma in N
Par 4.3.4.2.45iO4I2016 in par 43.4 24 and Reset DS iobs
- [Tabel met opmaak
Pago 3 of 131
Contentr
Met opmaak
Met opmaak
Met opmaak
Met opmaak
Met opmaak
Met apmaak
Met opmaak
Met opmaak
Met opmak
Met opmaak
Met opmaak
Met opmaak
Met opmaak
Met opmaak’
Met opmaak
Met opmaak
Met opmaak
Met opmaak
Met cpmaak
Met opmaak
Met opmaak
Met opmaak
Met opmaak
Met opmaak
Met opmaak
Met opmaak
Met opmaak
Met opmaak
c
cn
cm
cz
DocLunent: no rio
M.tcpmaak__
Met opmaak
Met opmaak rii1!tfPf_ £2iMet opmaak
Metopmaak
Met opmaak
- Introduction 7, — ._—fj4,Qpmaak
1.1, - Document Purpose..
7.1.2, - S
.. ................................................_........ .7,,
1.3, Glossary
2 MIH Overview 9,, -—
21, Introduction 9, —-
2.2, Functional Description— ?rpcve... ..
2.3, Functional Description — Architectura! Overview ii4O2.4 Operational Description — Logical Operational Model 122.5 Operational DescripUon — Physical Operational Model 132.5 Operational Description — which standard middleware runs where 142.7, Operational Description — which additional MIH building blocks wn where 15,2.8 Overview ot Required Desktop Tooling 16,
3. Access Control and User Management 184. Operations 19
4.1, MOM On-line Services 19,4.1.1 Introduction 19,4.7.2 Process Monitoring 194.1.3 HTTP Cog Setting 344.1.4, MOM Log file locations 35,4.1.5 Shutdown 364.1.6, Startup 36
4.2 CDC Continuous Processing 364.2.1, Introduction 36,
4.2.2 CDC Process Monitoring 37,4.2.3 Log file locations 38,4.2.4 Shutdown 38,4.2.5 Startup 39,
4.3 uS (EEL) Processing 39,4.3,1 Introduction 394.3.2 ETL Process Monitoring 394.3.3 Logfile locations 404 34. Reslart Ability 414.3.5 Shutdown 494.4.3.6, Startup
4.3.7 Run-Time Process Control 5D4
Page 4 at 131
4.4 MOM Batch loader Processing and Data Reconciliation 49
4.4.1, Introduction 5149,
4.4.2 Resuming Batch Loading Process
4.4.3 MOM Batch Load Process and Data Reconciliation 514.9,
4.4.4 MOM Batch Processing File Structure
4.45 MOM Batch Loading Scripts 2&4i4.4.6 MOM Batch Loading Reports
4.4.7 Rejected Records Reloading in MOM 5G5.4
4.4.8 Puppy 5654,
4.4.9 Firewall administration 5654,
4.4.10 MOM Batch Loader Process MDnitoring 56o4
4.4.11 Archive location content
4.4.12 Shutting Down MDM Batch Process 585
4.5 MIH Batch Processing 5856
4.5.1 Introduction 5556,
4.5.2 The Scheduled Processes 5866
4.5.3 Batch process monitoring
4.6, MIH Ejection Tray
4 6 1, Data Acquisilion Election Tray 64
4 6,2 Data Validation (source-oriented) Ejection Tray 64
4 63, Data Validation (target-oriented) Ejection Tray
5. MDM Data and AppHcation Maintenance 6766
5.1 MOM Data Stewardship User tnterface Z66.5.2, Configuration and Reference Data Modification In MDM Database 6968
5.2.1, General 6966,
5.2.2 Database Changes Steps ZP,€9.5.3 Enabling and Disabling MOM Audit Trail Information 7170,
5.3.1 General 7.1795.3.2 Configuration Steps for Enabling TAIL 7774,
5.3.3 Configurabon Steps for Disabling TAIL 7772,
5.4 ExternalizIng MOM Logging Configuration
5.4.1 Enablement of External Logging Configuration 7473
5.4.2 Revert Back External Logging Configuration 8688,
6. Data Integration Functional Maintenance i9O,6.1 Operations Console
6.1.1 General 9100
6.1.2 How to activate the Operations Console using XMETA database
6.1.3 How to configure the Operations Console
Dccument: En2LP t.l I Hapii-,
jlet opmaak
Met opmak
Met opmaak
Met opmaak
Met opmaak
Met opmaak
Met opmaak
____________
Met opmaak
________
Met opmaak
_________
Met opmak
Met opmaak
Met opmaak
Met npmaak
Met opmaak
Met opmaak__
Met opmaak
Met oprnaak
Met opmaak
Met opmaak
Met opmaak
Met opmaak
Met opmaak
________
Met opmaak
___________
Met cpmeak
________
Met opmask
_______
Met opmaak
Met opmaak
_____
Met opmaak
Met epmaak
Met apmk
Met opmaak
Met opmaak
Met opmaak
Met opmaak
Met opmaak
_____
Met opmaak
Met opmaak
Met opmaak -—
“.9
ED
“.4
Page 5 of 131
Metopmaak
Met opmaak
Met opmaak
Met opmaak
Met opmaak
I1ci. upmaak LJUJJ
Met opmaak C2iiMet opmaak CZs1JMet opmaak
Met opmaak ciDMet opmask
Met opmaak ctJieMet opmaak ciJMet opmaak
Metopmaak cnMet opmaak
Met opmaak
Met opmaak
Met oprnaak
Met opmaak
Met epmaak
Met opmaak
Met opmaak
Met opmaak
_________
CEI8Met opmaak
_________________
C]iJMet opmaak
________________
cIIMet opmask
_______
C.J1,OiLlMet opmaak
____________________
L._[iPZ1iMet opmaak
_________
L1iJJMet opmask [.1ij4UMet opmaak
Met opmaak
Met opmaak C... flail]‘ 1 Metopmaak F... [1081]
Met opmaak fj)JMet opmaak
Met opmaak LJiifl]Met opmaak Cfli2Metopmaak (... [113]]
Met opmaak
Met opmaak L..1ii.UMet opmaak L.Li16]]
Met opmaak C... till]]Met opmaak
Met opmaak
Met opmaak L1°Met opmaak C.JllIU
L.. [1221]
Metopmaak L. t 1231]
6.1.4 How to connect to the Operations Database using XMETA Database
6.1.5 Configuring user access to the Operations Console
6.1.6 Starting the Operations Console
6 1 7 Monitoring the ETC Server 9908
6.1.8 Reference information
6.2 MIH_ETL Database i2i4Of
6.3 Linux — ETL Ejection Trays
6.4, CDC Management Console 1O23-04
6.5, DataStage Operations Database
6.6 CDC Logging j,3f12
6.6.1 InfoSphere CDC for zIOS 1O34-O
6.6.2 InfoSphere CDC for other operating systems
6.7, Recovering CDC After a Source Table Restore i044O3
7. Backup and Recovery Scenarios 10540%
7.1, MIH Recovery from CDCIETL Failure
7.2 MIH Recovery from SFS Failure Mi
7.3 MIH Recovery from MDM Failure
7.4 MIH Fallover Active to Passive
MIII Takeover Active to Passive 114,
7.5. MIH Recovery from MDM Database Failure 117,
8., Deployment Rollback 118
8.1, MOM Deployment Rollback 118,
Appendix A., MIH Runtime processes 122
Appendix B. MDM Data Stewardship User Interface 124
Appendix C. How to identify defaults as a result of REFDATA lookup 130
Document
Page e of 131
1. !itrok’ctk,n
1.1 Document PurposeThis document was produced as part of the Dutch Tax and Customs Administration (DTCA) Master InformationHub (MIH) project. It defines the set of activities required to run and operationally manage and maintain the MIHapplication components in a target environment.
The reader should refer to the Release Notes documents for the Master Data Management (MOM) and DataIntegration (Dl) subsystems, as they will describe for any given release any known defects and associated work-around which may impact this handbook. Reader should also refer to corresponding deployment guide (bothMDM and Dl) for detail information on deployment.
1.2 ScopeThis is document is limited to operational aspects limited to MIH solution and limited to MDM and Dl solutionlayer layers only. Though this document specifies the database backup and recovery situations as part of overalloperation support process but it does not detail down the backup I recovery steps. It is expected that DTO’sstandard practice will be followed In case of 082 backup and recovery.
This document Is targeted to explain the operational aspects In production environment anything beyond thisenvironment including data analysis environment (used with Analyzer and Discovery tool) Is considered beyondthe scope
The main intension of this document Is to support DTO’s exploitation team who are responsible for productionenvironment and this team is considered as the main reader of this document
1.3 Glossary
Term DescrIptIon
ACL Access Control List
BLD Belastingdlenst
BSB Belastingdienst Sefvice Bus
CDC Change Data Capture.DB2 IBM relatIonal database management system
DI The Data Integration ‘layer of the MIH solution (the combination of CDC and IISDataStage (ETL) software).
DTCA Dutch Tax and Customs Administration
DTO Dutch Tax OfficeESS Enterprise Service Bus
ETL Extract, Transform, LoadFQDN Fully Qualified Domain Name. Example: ts03p002tst.belastingdienst.nl
GI Gebmiker’s Instructie (User Instructions)HP Hewlett Packard f IT vendor)
HPBAC Hewlett Packard’s Business Avaitability Center product
IIOP Internet Inter-ORB Protocol (CORBA)
IIS InfoSphere Information ServerInfoSphere A family of IBM middleware products, including MOM, COC and IIS.IPAS IBM Pure Application Systems
I Document E.E.Mx4 Qpeat H3rdb 4
Page lot 131
Dotument E--M 4-Op $P
Page 8 of 131
1
Document EBA*ecfa44adbc4
Page9ofl3l
S.
I Doujmont MW4—Opado4QG
Poge 10 of 131
Pagell of 131
Page 12 of 131
I’
Documcnt - - .--:£
Pags 13 f 131
Page 14o 131
4
I Dou.t
Page 15 of 131
I cment D-f?W
Page 16 of 131
Page l7of 131
I IJocumet
Pa9e 18 o 131
GpHok1c
Page 19 of 131
.
Page 20 of 131
r w -
+
Document E’4 k -
Page 21 at 131
I Document
Page 22 of 131
t
I - )Qml.lt E!4DM CWa1a
Page 23 of 131
II
Page 240! 131
I cWfler
Page 25 of 131
Dt
Page 26 of 131
Page 27 of 131
I Documn -
Page 2B of 131
•
Doctmleflt
Page29af 131
I Document
Page 30 of 131
I Document
Page 31 of 131
Document LnaI14abook,GoG
Page 32 of 131
I DoainentdL4oc
Pane 33 of 131
I Dcument
Page 34 131
I EHpaa
Pge35 of 131
-D
I’ll’
’’’
II’n’
IhPI
tII
1JII
I’III
I Doainent
Page 37 of 131
Page 35 ot 131
0
I
11111,11
IIuIIIlimil
__________________________________________________________________
11ii11;1
-I
I Document Hondbo-duc
Page 4Dat 131
‘I a
11i111
11
II’iiiJ
I
I Damient
Page 42 of 131
111111
11II
I’‘H
’1I
jill
iii’
III
I
I 000ument E[t M
Page 44 at 131
I rocument
Page45of 131
rrrrrttrtrtrtrrrzrrrttttrztz
Document
Page 46 of 131
____________________
Jt:
Page 47 at 131
Met opmaak: Standaard, lnspringfng: Links: 1,9 an, GeneLop5or11mioqtkens of nurrnnexlng
Met opmaak: tnsprlnging: Links: 1,27 an, Genengstnkenn of numniering
I IJocument Ei73M
Page 48 of 131
I Document fW4
Page4gatlal
I Doc*imeit
Pago SOot 131
I Document: Pi -pecalO4ul
Page 51 of 131
F
Document oc
Page 5 of 131
I’ll’
,’’
I’ll’’ ‘‘h;i’
i;II
ill
C”
I-•
:-.‘.S
5aaC
.aflt..d
I-.
-..
--
-
--:r--
-
-
I-:
.--
--
--
--
--.
-
——
V—
*-“-
Page 54 of 131
I c1Mneft QBD
Page 55 of 131
Document
Page 5601 131
I Domie
Page 57 of 131
Dent
Page 56 of 131
IPage 59 of 131
I —-
Met opmaak: Lettertype: 8 pt
Page 60 of 131
7
It’l
l’’’
1111
1111
!;;h
ii’’
III
111111
11
IIi:IJ
III’
Iii
-U
Met opmaak: Engels (Vererilgde Staten)
I Doaneit Q8 g44—efatenal
Page 63 of 131
r3
I
“it”!ii’
‘‘‘
‘I’ll’IIIII
__
I
-
‘1’
• .4
Pa9e 67 o 131
II”l
lII:H
:IJ1111
‘Ii
I
I Oocijiirt B M14 peqN3.db d,c
Page 69 ci 131
11111111
ii!I!Ii1’
ll”ll
aCa
Iii1:11
III h11I’llI’ll
‘I’ll’’’
a-
Pane 72 of 131
3-
1
3%.
.3
Pane 0! 131
i
Page 74 of 131
I 000Lmlent 4—col3L
Page 75 of 131
I Document:
Page 76 at 131
•0
I’ll’
’’’
I‘ii’
IIP
I,
!‘:i
i’ III
I Document
Page 7B ot 1l
Page 79 of 131
I Document EZc.-p
Page 80 01131
4
•
Page 81 of 131
I Ooanent
Page 82 of 131
I Document 2DLM-R dok
Page 83 of 131
I Document BUMIH4)p Hncthoc 4o
Page 84 of 131
Page 85 of 11
Page 06 of 131
Page 87 ci 131
Page 88 of 131
III”
III,”
‘I’ll’
Ihi
III
III
I
C,
aa.
II
C.
III;:’!1111111
liii
I‘‘ii
ZtD MH Qaadb4oG
Page 91 of 131
II
Page 92 at 131
Pane 93 of 131
Doalmefit
Pane 94 o 131
Iii:‘‘
ii’ “II
11111111
P89t 96 of 131
/
I cum.r
Pa9e 97 of 131
Docljmert EOBLDMIH r*
Page 98 of 131
I uoci,nin EZI
Page 99 of 131
Docunient
Pe 100 of 131
Page 101 of 131
Page 102 of 131
I’
I Document
Page 103 of 131
Document: Efl2L-fl ll eena4andboocdoc
Page 104 of 131
7. backup anü Recovery Scenarios
This chapter contains the Backup and Recovery scenarios defined for MIH. The scenarios are based on Rip andReplace. in this chapter you will not find troubleshooting instructions but a procedure to repair a brokenenvironment by rip and replace of the broken pattern(s).
We have defined the following scenarios:
1. MIH Recovery from CDC/ETL Failure
2. MIH Recovery from SFS Failure
3. MIH Recovery from MDM Failure
4. MIH Fallover Active to Passive
5. MItt Takeover Active to Passive
6. MIH Recovery from MOM Database Failure
As a general prerequisite the passive leg should be correctly deployed.
tn some scenarios there is also the possibility to restore the ItS server using the isrecover.sh script. Thisrecovery step should only be used as a last resort. This recovery can only be executed oncel So after you haverestored ItS once further recovery on the same server is not possible.
Only use this procedure If the normal recovery procedure fails because the passive server is not correctlydeployed. Using the isreeovery.sh script you will have a good chance to recover anyway. If this also fails you willhave to correctly deploy the passive system before executing the procedure again.
The following steps should be executed if you really need to restore uS:
Restore uS
Daaiu.t
Page 105 of 131
Ori
gina
lP
ow
erP
oin
t
The
orig
inal
Pow
erP
oint
reco
very
proc
edur
eca
nbe
open
edas
follo
ws:
Rig
htm
ouse
clic
kan
dse
lect
Pre
sent
atio
nO
bjec
t->
Ope
n.A
nych
ange
sm
ade
toth
eP
ower
Poi
ntw
illbe
stor
edw
ithin
the
Ope
rati
onal
Han
dboo
k.P
leas
ebe
awar
eth
atth
eex
ecut
ion
step
sar
ein
the
Pow
erP
oint
and
aco
pyof
the
step
sar
eon
the
next
page
.
7.1
MIH
Rec
over
yfr
omC
DC
IET
LF
ailu
re
MIH
Rec
over
yfr
omC
DC
/ETL
Failu
re
Step
1:D
tCO
miS
Oi
SnJC
DC
Pat
tern
Step
2:R
eatr
tN
FS
r]
tlO
MD
.If
l
Pag
e10
6of
131
Ste
p1:
Dec
omis
sion
ETLJ
CD
CP
atte
rn1.
Ad
just
scri
pt
MA
NA
GE_
CD
Can
En
terp
rise
Nod
e
ci”]
2.M
ake
cure
tijt
allsu
bsc
rip
tio
ns
are
inac
tive
a.O
poii
IUM
info
sphe
reCO
GM
anag
emen
tC
onso
leis.
Clic
kM
onit
orin
g,ta
bS
ubsc
ript
ions
c.P
utal
lsu
bscr
ipti
ons
onIn
acti
ve3.
Sh
utd
ow
nb
oth
nudes
a.S
hutd
own
Cam
puto
Nod
eis
.Shu
tdow
nE
nter
pris
eN
ode
2.e-
enaw
eN
xsh
arin
g
Mou
ntex
po
rton
Com
pute
Node
omau
inco
tthe
exis
nng,
chan
ge(s
tab
and
mou
nt-a
4.M
ount
exp
ort
onE
nte
rpri
seN
ode
um
ou
ntu
dih
eex
isti
ng,
chan
gets
talu
and
mount
-a
Ste
p3:
Res
tore
conf
igur
atio
n1.
Che
ckto
see
that
CU
Csu
bsc
rip
tio
ns
not
In(a
llure
stat
ea.
Log
inon
the
IBM
info
sphe
reC
flCM
anag
enm
erm
i Con
sole
b.C
lick
Mon
itor
ing,
tab
Sub
scri
ptio
ns2.
Sto
pth
eC
flcA
cces
sSer
ver
and
the
Cfl
Cfl
ataS
tag
eSec
rer
II
.7a,
3.M
ake
sure
tand
ingz
one
and
SRC
dire
ctor
yar
eem
pty
4.A
skto
over
wri
teth
epa
ssw
ord
for
the
exis
ting
TSM
node
a.W
rite
dow
nth
en
ame
ofth
een
itti
ngTS
Mn
od
ean
dth
eB&
Rse
rver
,ii.
Cal
lB&
RB
ehee
r(B
OBS
)5.
Cba
nge
the
serv
erna
me
whe
nne
eded
6.C
hang
eth
eiP
-adre
sse
and/o
rno
dena
me
whe
nne
eded
I 7.R
egis
ter
new
leg
usin
gol
dna
me
exis
ting
TSM
node
I B.R
esto
reD
B2
dat
abas
e(F
oIl
and
incr
emen
tal)
asuse
rdb2in
stl
Prov
ide
the
num
bers
ofth
eFu
llfil
elu
sual
tyth
ela
rges
t)an
dth
ein
crem
enta
lfil
ean
den
ter
0to
proc
eed.
ifon
eof
the
file
sis
mis
sing
add
opti
on-m
acto
dxm
cco
mm
and,
‘th
eda
tew
ithin
the
last
part
ofth
efi
lena
me.
You
9.R
esto
re11
5a.
Itis
reco
very
isne
edex
ecut
est
eps
here
(See
Ope
rati
onal
Han
dboo
k)
0.R
esto
rebo
okm
arkt
from
end
ofth
epr
evio
usda
yto
tmp
11.
Res
tore
CO
CI
Ste
p4:
Act
ivat
eL
Sta
rtth
eC
bCA
ccet
s5er
uer
and
the
CO
CO
atas
tages
erver
II
2.C
all
Mai
nfra
me
xu
pp
ort
and
ask
tore
conf
igur
ew
bsc
dp
tio
r
I—
——
-—
---
I3.
Com
mun
Icat
ebo
okm
arks
toM
ainf
ram
esu
ppor
t
ciiii
b.ca
llM
n:nf
rarn
eS
upport
4.Pu
tal
lsu
bscr
ipti
ons
out
cont
lnou
sm
irro
ring
a.lo
gin
IBM
bIos
pher
eC
IXM
tnugem
snt
Con
cole
b.Cl
ick
Mon
itori
ng,
tab
Subs
crip
tions
c.R
ight
mou
secl
ick
Subs
crip
tion,
choo
seS
tart
Mir
rori
ng.,.
d.S
elec
tC
ontin
uous
,cl
ick
OKR
epea
tth
isfo
rese
rysu
bscr
ipti
onS.
Sw
itch
and
star
tM
III
sch
edu
lea.
Mak
esu
reST
OP
isne
tre
olac
mna
S2in
:
=
Pag
e10701131
Ori
gina
lP
ow
erP
oin
t
The
orig
inal
Pow
erP
oint
reco
very
proc
edur
eca
nbe
open
edas
follo
ws:
Rig
htm
ouse
clic
kan
dse
lect
Pre
sent
atio
nO
bjec
t->
Ope
n.A
nych
ange
sm
ade
toth
eP
ower
Poi
ntw
illbe
stor
edw
ithin
the
Ope
rati
onal
Han
dboo
k.P
leas
ebe
awar
eth
atth
eex
ecut
ion
step
sar
ein
the
Pow
erP
oint
and
aco
pyof
the
step
sar
eon
the
next
page
.
MIH
Rec
over
yfr
omS
FS
Fai
lure
MIH
Rec
over
yfr
omSF
SFa
ilure
icE
pao
e
C:-!lr.
tr,r:::e
PA5
SF5p
tter
PAS
MO
Mp
att
Ste
p1
De
sSiO
4SF
SP
atte
rn
Step
2:D
epiG
ySF
5P
atte
n
Ste
p3
Run
Sol
Xpc
kage
s
MO
jfl
Mab
awa)
OS
Pag
e10
8of
131
C
E0
In
C
=
UI’;:!’E
Illihill
C)
—
mU
.•1
U
4a E
0
nN
C
-4
CC
E0
C
Cm
Lfl
U,
‘1•0.
ci,
I
4
0
0
C0ICa.
fi• .••
• •.
• •
•
7.3
MIH
Rec
over
yfr
omM
DM
Fai
lure
Ori
gina
lP
ower
Poi
nt
Ste
p1:
DeC
omis
sion
MO
MP
atte
ni
The
orig
inal
Pow
erP
oint
reco
very
proc
edur
eca
nbe
open
edas
follo
ws:
Rig
htm
ouse
clic
kan
dse
lect
Pre
sent
atio
nO
bjec
t->
Ope
n.A
nych
ange
sm
ade
toth
eP
ower
Poi
ntw
illbe
stor
edw
ithin
the
Ope
rati
onal
Han
dboo
k.P
leas
ebe
awar
eth
atth
eex
ecut
ion
step
sar
ein
the
Pow
erP
oint
and
aco
pyof
the
step
sar
eon
the
next
page
.
Doc
umen
t:E
O2
RL
D-M
IH—
Qp
amu
io4
lan
db
nek
4o
c
=M
IHR
ecov
ery
from
MD
MFa
ilure S
tep
2:R
edir
eCt
NFS
.1m
.10t.b
/0S
Pag
e1
10
01
13
1
ri.n•m
CD11111111
.11:11:11•
ammo
p.
agz2
am
a
aaa
n
a
C
C
t
aa
In
=aa
a
C
Ori
gina
lP
ow
erP
oin
t
The
orig
inal
Pow
erP
oint
reco
very
proc
edur
eca
nbe
open
edas
follo
ws:
Rig
htm
ouse
clic
kan
dse
lect
Pre
sent
atio
nO
bje
ct->
Ope
n.A
nych
ange
sm
ade
toth
eP
ower
Poi
ntw
illbe
stor
edw
ithin
the
Ope
rati
onal
Han
dboo
k.P
leas
ebe
awar
eth
atth
eex
ecut
ion
step
sar
ein
the
Pow
erP
oint
and
aco
pyof
the
step
sare
onth
ene
xtpa
ge.
Doc
umen
t:
7.4
MIH
Fai
lover
Act
ive
toP
assi
ve
NIH
Failo
ver A
ctiv
eto
Pass
ive
____________
pIern
-
IP
S5
pnor
Ste
p1:
Dec
ores
sion
Act
ive
Pat
tern
s
Step
2:R
esto
reco
aflu
ratl
on
_____________
MD
Mp
ttcrn
1- 1.
P’tt
r.IO
PAO
b,/O
s
Pag
e11
2ot
131
Step
1:D
ecom
issi
onA
ctiv
eP
atte
rns
1.S
uspe
ndM
IHsc
hedu
le
Z.M
oke
sure
that
all
subs
crIp
tIon
sar
eIn
acti
vea.
Ope
nIB
MIr
ifor
pher
sCO
CM
anag
emen
tC
onso
leb.
Clic
kM
onit
orin
g,ta
bSu
bscr
iptio
nst.
Putat
subs
crip
tion
son
Inac
tive
3.if
recc
esar
tsh
utd
ow
naR
node
s
Ste
p3:
Act
ivat
eET
L/C
DC
1.S
tart
the
CD
cAcc
eus5
prve
ran
dth
eC
DC
Dat
atta
ge5e
rver
2.C
all
MaI
nfra
me
supp
ort a
ndas
kto
reco
nfig
ure
subs
crip
tion
sa.
Ene
cute
the
‘wuc
elpr
ogra
mrn
aso
otT
WS’
onth
eM
Fit.
Cal
lO
pera
tion
s(t
st3.
Com
mun
Icat
ebo
okm
arks
toM
ainf
ram
esu
pport
b.co
lM
nnfr
amn
Sup
port
4.P
utal
lsu
bscr
ipti
ons
onco
ntln
ous
mir
rori
nga.
Leg
InIB
MIn
fosp
here
CCC
Man
agem
ent
Con
sole
b.C
lick
Mon
itor
ing,
tab
Sub
tcrI
ptio
nsc.
Rig
htm
oc’s
eclic
kSu
bscr
iptio
n,ch
oose
5500
Mir
rOri
ng..,
d.S&
ect
Con
tinuo
us,
tick
OKR
epea
tth
isfo
rev
ery
subs
crip
tion
IEE
EZ
EE
EE
’1
6.R
esto
re(*
2da
taba
sefr
ompr
evIo
usda
yen
d
I7.R
esto
re11
5a.
Ifis
reco
very
isne
edex
ecut
est
eps
here
(See
Ope
rari
onol
Han
dboo
k)
I11111111
15EE1111
1111E
2.A
skto
over
wri
teth
ep
assw
ord
for
the
exis
ting
TSM
node
n.W
rite
dow
nth
ena
me
ofth
eco
stin
gTS
Mno
dean
dth
eB&
Rse
rve b.
Call
8&R
Beh
eer
(608
8)3.
Cha
nge
the
nerv
erna
me
whe
nne
eded
9,ne
ssur
elL
A..
I10
.R
esto
rebo
okm
arks
from
end
ofth
epr
evio
usda
yto
tmp
K11
.R
esto
reCC
C
h.’
4.C
hang
eth
eIP
.ad
ress
esan
d/o
rnode
nam
ew
hen
nee
ded
5.R
e’ls
ter
new
leg
usin
gol
dn
ame
exis
ting
TSM
node
:1
-I
it.
Ste
p4:
Act
ivat
eB
atch
load
er1.
Sw
Itch
and
star
tM
IHsc
hed
ule
You
will
find
the
dat
ew
ithin
the
last
part
orth
efi
lena
me.
Page
113
of13
1
MIH
Tak
eov
erA
ctiv
eto
Pas
siv
e
NIH
Tak
eove
rA
ctiv
eto
Pas
sive
IPAS
ETLC
DC
patt
ern
P55
SF5
pati
arn
PAt
tTL/
U3C
pat
et!:
Ste
p1:
Dec
orni
5sIo
nE
UJC
DC
Patte
rn
Fte
rnre
ee*a5,
Ste
p2:
Dem
sste
nB
atch
load
er—
PAtS
F5pa
tern
t
IPAS
WM
patt
ern
rAtF
.FDM
pattern
_________________________
tep4:
-
L :JM
OM
Da,
at.s
e,IO
S
Ori
gina
lP
ower
Poi
nt
The
orig
inal
Pow
erP
oint
reco
very
proc
edur
eca
nbe
open
edas
follo
ws:
Rig
htm
ouse
clic
kan
dse
lect
Pre
sent
atio
nO
bjec
t->
Ope
n.A
nych
ange
sm
ade
toth
eP
ower
Poi
ntw
illbe
stor
edw
ithin
the
Ope
rati
onal
Han
dboo
k.P
leas
ebe
awar
eth
atth
eex
ecut
ion
step
sar
ein
the
Pow
erP
oint
and
aco
pyof
the
step
sar
eon
the
next
page
.
Doc
umen
t:E
O28D
-F.-
petn
ea4-{
andbuoF
oe
Pag
e1
14
of13
1
G)
at
C—
0--J
-53-,ru
U, -
z
(f) 4
C
I
a
a
C
•0em
a
t
t
m
Co
0CO
C
CI,a
I,
In
C,
2
jI
C
aCIC
CI In
C
o C
C
CC
C
• 0C, C-I
0)a)
0
UU)
CC
-I
(•n.3
I
0
0U
C
C
II’ iiii
III liii‘HI‘H,
Ilulill Ia
a
m
t
0
•0I
=C
It
In
CIn
CC,CC-
II, iiIIIII
II’ ‘‘iiI Iii I
‘‘‘III’’
C
II
step
3-2:
Res
tore
conf
igur
atio
nifl
. Sta
rtC
oCA
cces
sSer
ver
and
CD
CD
ataS
tage
serv
er
I i1llIfflI][[
11.R
esto
reIS
(Bac
kupp
edby
scri
pt02
0_Il
S_B
AC
KU
P.sh
)
C. d. C.
Step
4:A
ctiv
ate
CD
C/E
TLon
pass
ive
leg
1.R
econ
fIgu
reC
DC
from
acti
ve(o
ld)
topa
ssiv
e(n
ew)
targ
etd
atas
tore
Cal
lLo
glsc
hD
atab
aseb
ehee
r(t
st...)
toex
ecut
eth
efo
llow
Ing
Upd
ate
quer
y.O
rdo
ftV
Uou
rsel
fIfyo
uha
veD
atas
tudl
o:
2.C
heck
ifal
lsu
bscr
ipti
ons
are
avai
labl
e(m
ake
sure
they
are
not
doub
le) Use
IBM
Into
sphe
reC
Dt
Man
agem
ent
Con
sole
,ta
bM
onito
ring
,ta
bSu
bscr
Iptio
nsto
nth
ene
wta
rget
data
stor
e)3.
Exe
cute
the
‘wis
adpr
ogra
mm
10, S
MC
pCA
ccex
s5er
vsc
pnd
CD
CD
ata5
tize
Se,
ver
!iU
.R
esto
re11
5(B
acku
pped
bysc
ript
O2O
jlS_B
AO
WP.
shl
I7.5
MIH
Rec
over
yfr
omM
DM
Dat
abas
eF
ailu
re
Thi
sty
peof
reco
very
scen
ario
desc
ribe
sth
esi
tuat
ion
whe
re(p
art
of)
the
MD
Mda
taba
se(s
)co
nten
tbe
com
es(l
ogic
ally
)co
rrup
t(f
orex
ampl
ein
valid
cont
ent
has
been
load
ed).
Als
o,th
edat
abas
eho
stin
gen
viro
nmen
t(o
nsy
stem
zan
da
SAN
stor
age
fabr
ic)
isco
nsid
ered
byde
sign
tobe
high
lyav
aila
ble
and
resi
lient
agai
nst
(sin
gle-
site
)di
sast
ers.
The
refo
rete
chni
cal/
infr
astr
uctu
refa
ilur
esar
eno
tco
nsid
ered
here
The
norm
alpr
oced
ure
isto
rest
ore
the
MD
Md
atab
ase
onzI
OS
(up
toa
cert
ain
poin
t-in
-tim
ein
the
past
).T
hat
mea
nsth
atth
est
anda
rdz/
OS
dat
abas
ere
stor
ese
rvic
esh
ould
beus
ed.
Thi
sal
som
eans
that
the
MD
Mba
tche
sth
atw
ere
proc
esse
daf
ter
that
poin
t-in
-tim
ene
edto
beru
nag
ain
toap
ply
the
sour
cesy
stem
data
chan
ges
once
mor
e.T
hein
put
file
sar
est
illav
aila
ble
inth
e/o
ptlm
ihlB
atch
Loa
dln/
Arc
hive
/XX
Xdi
rect
ory.
Cop
yth
eX
ML
file
sth
atne
edto
bepr
oces
sed
agai
nto
the
/opt
lmih
/Bat
chL
oadl
n/X
XX
dire
ctor
y.
Thi
spr
oced
ure
isno
tde
scri
bed
inde
tail
bec
ause
for
the
zIO
SD
atab
ase
adm
inis
trat
ors
itis
aw
elt-
know
nan
dco
mm
onpr
oced
ure
also
refe
rred
toas
BA
U(B
usin
ess
As
Usu
al).
Pag
ell
7of
131
8. Deployment Rollback
8.1 MDM Deployment RollbackAt the time of deployment it may be necessary to rollback deployment components. If something goes wrong atany deployment step and it Is not possible to rectify the same, it is advisable to stop the current deployment andredeploy the previous package which was successfully deployed. Following critical Information should beavailable to the deployer before starting MDM Server deployment.
• The current state of application; this include MDM Server application, Data Stewardship application, database structure level (DDL script) and database meta data level (reference end configuration DML).These Information! artefacts are available in previously deployed package. Deployer should haveaccess the previous deployment package so that it can be redeployed if necessary
• Deployer should know what are the components need to be deployed as part of current release and allthe deployment related information should be available to him I her
In case of deployment rollback of on-going deployment it Is very much necessary to know the deployment stepwhere things went wrong so that rollback strategy can be strategized. The attached spreadsheet Is a run sheetfor deployment for acceptance and production environment where two data centers need to be deployed. Thisspreadsheet lists the deployment steps and if anything goes wrong at particular step, It provides details ofmitigation strategy. For details instruction of action(s) related to rollback refer MDM deployment manual.
Predecessor Deployment Action Success Criteria Check Point Failure Action
Primary Data Centen Stop MOM latch Pmcestor Loot proceed furthertili the timeSending 5chedoTer command to should be Copped CustomerJog wiling stops. Ifeoecute MOM batch loading Cutiomer.iog keep on wiling forscripts sometimes, please check
scheduler to make sure it Is noteoecotingtrylngto Invoke MOMI atch pmcessor
1 Make sure no web service Jo web service request I Cot proceed furthertlil the timrrequest is tent to Primary Data sent to all WAS custom Coutowor.log mlilrg Cops. ifCenter. Eoecote chfuilpurtblack nodes availabin in Customer.log keep on rolling forshell script in bolh the custom Primary Data Center sometimes, please check badtories balancer setting to make sum It is
configured correctly. in case IssueIs not Identified in reasonable time
lease Cop deployment Is makePrimary Data Center online toresume bath batch loading andmb somlce
2 Rollback reference data All the SOt, statement You may come across some wamlngs; Enecutethe scripts available withavailable iv MOM schema; should be eecoted please make sore wamings are not previous delivery package. Followesecutethe DM1 scripts successfully without mpacting deployment. in rose there is the instmctlon available atavailable In deployment any error, any enrorpiease first make sore you
- have right access to eoecste the scriptsto MOM schema. if you are having allthe right uccesv ond yes are still gettingthe errors please consIder ft as fallow these scripts ate successfullyand lake failure acfionlsl euecuted dont proceed with
deployment further but makePrimary Data Center Is availablefor batch ioad and web sntvlcerequest
I Doctmtenfl BMiklpecalstnaY4.4avndL.vlv%€
PageitSof 131
foamiest; fi078 ASH.-Qpevalronai4dosdtooh dot
H
mnnrt tnrtM “iwo, wrr,.e’e nil tb’ ‘fl! .tatemerr )Yr,’r mr’rnn,r lrrw’ rn’r’’”ei’r F,nrirr’ the wriyte ovailebre webdata, execute CML scripts should he eaccuted please make sure warnings are not prevtaes delivery packageavaIlable in delivery pacioge. successfully without tmpadiegdeptoyment. In case them is pacioge. Follow the lestmctlen
I
ese sands aoeavaaWe el any error, aey errerplease first make mm you ausitahie atham rightacceosto execute the scriptsto MOM schema. If you am having allthe right accent and yen are still gettingthe ermrs pleaae coesider It as failumand take failure action(s) hese siitrs am mccc&lly
eaecaied don pmceed witheptoyrnerrc furtirer bat makerheary Data Center Is available
or hatch lead and web serviceequest
4 hange roefigaratlon Au the to! stalemerr Van may come acmss same warnings; senate the coefigsmtieo schemaeformation in cenfiguvetior shosid be eaecutod please make sum wamlngs are not pecific script afterchaoging thechema accessed by primary svrceosfaily wthaui impacting deployment. In case there h hensa came w (yjpgg
data center, esecate any error any error please first make mm poacontigaratien schema related ham right acress to eaetste the scriptsscript (third and tovath letterof to coomopoeding schema. if yes amhbscrio CNJpaHab haaingallthe sight access and yea am fourth letteref that file wili be
still getting the ermn please consider it fiN); Uth will reseat back setting toas tatam and take fainm action(s) preeleos delivery inset. Once this
sceipt is soccesoinily executeddoe’t proceed wIth depioymentinrther hot make Primary DataCanterisasollahie forbatck badand web mrvice reqsest
S MOM Server application MOM services should tend an weh service mqaest from Seas Have a quich leek lets the fatomdepioytneot (follow deployment be esearted command prompt asing web service reasee and tryta mseive If it assteps in deployment manual). sarcessinlip toot checb like msposse to eemre be revolved quickly, lease ofDetall deployment steps are MOM service is working future sodeplep the MOMdocumented it section ‘MOM application and mdeplep pmvloastemer Deployment Steps’ in delivery pacbge)lscisding MOM‘lID M1H - MOM Deployment application, database scriptMansai.doc’ document riepioymeet etc.)
t MOM Data stewardship Log on to Data MOM Data stewardship screen sheaid Hove a qslcb iook tots the faitamapplication deployment )faliow Stewardship he displayed and parties can be reases and try to resolve if It candeployment steps iv apptcatioe and search mtdeved asing these screens he msrdoed quickly, locate stdeployment mamal). Detail party taliere sodepicy the MOMdeployment steps am application, Data Stewardshipdacuwented in section Data appiiatlsn and redeploy previousSteward Deployment Steps in delivery packege )iertadieg MOM
‘ ‘gb Mit - MOM Deployment applicatles, data stewardshipManaat.doc’ document application, database script
deployment etc.)7 Make sore web service mqoest Web service mqaest Log tote both first aed second WAS ocase ef faitam ondepley the
is sent to Primary Data Center sent to WAS custom custom code one after another Is MDM application, Datanodes Is Primary Data Prhnary Data Centeras mdmopee aed Stewardsklp application aedCenter should get tange dimctsryto redeploy prevleas detineryeaenated pmperip [acioge (inclodieg MOM
ppliallee, data stewardshipcommand ‘tall -tfisstomar.leg’. The file application, datahase scriptshasid sat be mtliog if mqaedio am depiopoent dc)‘end ta WAS custom code
s Prbnary Data Center Start MDM larch Pmremor Leg late firstWM custom code is case of Miam asdepioptheSending schedoicrcomnsaod to shoald stscflworldng Pehnary Data Ceeteres mdmsper and MOM application, DataexecateklDM batch leading thangedliedoryte ttewardship application andscrtpto • redeploy previous delivery
cole The cummaso lad-s packege (including MOMCaotomer.leg’. The file droutd be miling application, data stewardshipIc screen application, database script
rieploymest etc.)
Page ttfl of t31
Document fp28&.Ml—Operrtlmeai.ll41ta4o
QSrnn’’ It,’:’ rn,r’ S’cn (ConIc itstrh Prvrrnenr (I n’ mm Cret 1PM nrr’tnn’ nod’ In fl”t r,nc”J hehernitt the tim”tending scheduler command to should be stopped Secondary Data Center as mdmoper and Cuttamer.lug rolling stops. Itexecute MDM batch loading charge dlmctoryto — Costomee.tog keep on mfling forscripts (if hutch luador ts active ““‘j somettmos, please chockhere, others-also lgnnre this Issue che command ‘tall-f scheduler to make tore It ts notstep) Customer.loaj, The Ste should rot he eoecotlrg trying to Invoke MOM
rolling to screen botch pmceosortfi idaho sore no wok semIte No web semico request Log tnto hoth first and second WAS Don’t proceed torthertlil the time
request ts sent to Secondary sent to all WAS custom custom code one ufteranother to Customer.log miling stops. IfData Center. Eoecote nodes avulloble In Secondary Data Center as mdmoperand Customerlog keep on mltlng forchfollporthluck shell script In Secondary Data Center change directory to — sometimes, pleanechech toadboth the custom nodes ‘‘ alancnr setting to mahe turn It Is
. rravov’vavucz onfigurod correctly. In cove lssoecommand ‘tall 4 Customer.Ing’. Theme Is not IdentiSod in reasonable timeshould not he wiling In screen please stop deployment In make
tevoodary Data Contor online toresume both hatch loading andwok snmhe. Make sure PrimaryData Center is also wiled buck tonmulous level
It Change MDMCMZ schema All the SQl. statement you may come across some marnings, Execute the cuufgvratlon schemacootguratlun information, should ho executed nleam make sore wamlogs are not modification script after changingexecute DML scnlpt successfully without impartIng deptoyrnnnt In case there lv the schema eanm. This Is availablet2MCN001.tut availaklrat anyorror. anyerrorpleasn first make surepmu at
.
have right access to enecute the scriptsto MDMCM2 schema If you am having
._. all the right assess and you are mullPlease don’t forget to change getting the errors please consider S as third and fourth tetterwill be ‘CN’;the schema name fatum and take failum actioulsi this will revert hack setting to
previous delivery package level.Dnce this script is waccmshoiyexecuted don’t proceed withdeployment harsher hut make
. Secondary Data Center Is availablefor halch load and web sntnlco
, request. flolihack thn deployment, of PrImary Data Center as well
,(Including MDM application, data
. tewardshlp, database schema)U MD(ØkeHerapcaflbq MDMoeMcouakoua Send n web semite requ stir mU 0 H v qvl ktook tnt th f Ikir
lepla5meut(foNow.diploepets reeaeaatod command prompt using web service reason and try Ia resolve If it canri pslo d plomm$mard4 auccasfitliy t I hrcktk rnspons to boo I edqukkly In as ofDetail deployment stepsn MDM service Is working failure undrpioy the MDMdntumented In sectine ‘MOM . application and redeploy previousSemen Deployment Steps’ In dettuery packagn (Including MDM‘DID Mist - MDM Deployment application, database scriptManuol.doc’ document deployment etc.). fluilback the
deployment In Primary DataCeoter as well )inclodlng MOMapplication, data stewardship,database schema)
13 MOM Data stewardship Log onto Data MOM Data stewardsklp screen should Haee a quIck louk Into the tallowappficatlon deployment (follow Stewardship he displayed and parties tao he reason and try to resolse If It tandeployment steps In application and search retrieved using these screens ha resolved quickly. tncase ofdeployment manual). Detail pasty failure ondeploy the MOMdeployment steps are applitatlon, Duta Stewardshipduamenled In sectIon Data ayplbatloo and redeploy pmoloosSteward Deployment Steps In delivery package )lncloding MOMlID MIH - MOM Deployment application, data stewardshipf.tanvai doc’ document application, datakase script
deployment etc.). Rollback thedeployment In Primsry DataCeeter as well (Indoding MOMapplloatlon, data stewardship,database schema)
Pugn 120 of 131
I Doamnent li-C M;H çQ7aaniDh
l,’l h,,,;r- to Iwol, nr..,’oot Ii Into loth t’ni ‘rd •onod .tJlrt inr’ef f,fl,ro 0fenf0v tinmake sure web service request sent to VJA5 custom cusratn node one afteraeother In MOM application, Datais sent to Secondary Data nodes In Secondary Secondary Data Center as mdmoper and 5tewardsklp application andCenter. Open up the firewall by Data Center should get rbanaedrectoryto redeploy previous deliveryeoecutlng cbbalportblock shell enecuted property ‘‘ ackege (includIng MOM
ript pplicattoo, data stewardshIpcommand tail 4Customerlog The fite application, database scrIptshould not be rolling If requests are deployment etc.)vnd to WAS eustom nude
15 Secondary Data Centee Start MOM Batch Processor Log Into first WAS custom node In Incase of taliure andeploy thetending schedulercommand to should srart workIng Secondary Data Center as mdmopor and MDM application, Dataesecute MOM batch loading rhane dinectoryto Stewardship applicatIon andscrIpts )it batch loader is active , rdeploy previous deliveryhem; otherwise Ignore this toje the command tall -I package (includIng KWMstep) Customer.bg. The file should be rolling applicatIon, data stewardshIp
0 screen application, database scriptdeployment etc.)
Page 121 of 131
C’)C’)
C,
C,
111111!!III liii
11111111
IIr’’’IIIJIII liii
‘Iii’’III III I‘‘I’ll’,
0
C0,I,0
t
0
‘‘‘ t!!III liiiIIIII1111111I’’’’II
C’)
C’1
0)0a-
11111111Ii’• ClIlililli WI
II”tI
Iltilill11111111
.5“I
a-
II
11111111III. C1I
iIItuii El,
I’llC111111—
11111111 xci
Co
0(0C’4
U
0
Iii iiiiIII.III liii
I’ll
N’’’’II
a-
0
a
Ct
7a
II”—
11111 0
III’’’i’ CI
I CJiU
I,aI11111111111111
11111111IIJIIII liii
i’li’iIII Iii
‘‘‘‘I’ll
()
0,C”
0
lIj iiHIi’-
III liii
IItHIIII”II =
n
aC
0)
a-
.2
—
4i
IIii iiIIII liii
liii 8
______
Engels fVerenigde Staten)
Pagina 4: [1] Met opmaak
______
Engels (Verenigde Staten)
Pagina 4: [2] Met opmaak —
Engels fVerenigde Staten)
na4]Metopmaak
Engels fVerenigde Staten)
[pagina 4: [3] Met opmaak —
Engels (Verenigde Staten)
t? [33MetopmaakEngels (Verenigue Staten)
I Pagina 4: [4] Met opmaak
_____
Engels (Verenigde Staten)
I Pagina 4: [4] Met opmaak
____________
Engels (Verenigde Staten)
[agina 4: [5] Met opmaak
_____-
Engels (Verenigde Staten)
[riEngels (Verenigde Staten)
[Pagina 4: [6] Met opmaak
Engels (Verenigde Staten)
Pagina 4: [6] Met opmaak
Engels (Verenigue Staten)
[Pagi____ raaic -
_______
Engels (Verenigde Staten)
L
19-2-2016 11:48:00
19-2-201611:48:00 1
19-2-2016 11:48:00
19-2-2016 11:48:00
19-2-2016 11:48:001
19-2-2016 11:48:00
_________
19-2-201611:48:00 I
19-2-2016 11:48:00 I
19-2-2016 11:48:00
19-2-2016 11:48:00 I
19-2-2016 11:48:00!
—
19-2-2016 11:48:00
192-20161148001Engels (Verenigde Staten)
8!etm!!k 19 2-2016 11 48 00Engels (Verenigde Staten)
na4:[8Jtopmaa 19-2-2016 11:48:00JEngels (Verenigde Staten)
tPagina 4:[9] Met opmaak 29-1-201610:56:00 7Engels (Verenigde Staten)
tpagina 4: [9] Met opmaak— 2942016 10:56:ói]
Engels fVerenigde Staten)
I Pagina 4: [10] Met opmaak
Engels (Verenigde Staten)29-1-2O1610:56O7
