Embed Size (px)
Citation preview
2
Introduction:
Agenda:
• Lecture - Database Concepts
• Lunch
• Database Workshop (hands-on)
3
NxTrend & Progress:
• NxTrend & Progress
• NxTrend runs on products we sell.
4
What Is Progress?
• Not an O/S
• Database engine
• 4GL Programming language
5
Licensing
• Enterprise DB
• Appserver
• 4GL
• Provision (PC Client side)
• Client Networking
• Query/Results (being replaced by SX.explorer)
• Merant (ODBC)
6
Progress Database Concepts:
• Physical Parts of a Database– .db - database– .bi – before image file– .lg – log file– .lk – lock file (only there when Broker is up)
7
.db
.bi
.lg8
Starting The Database Broker
• Unix Process
• Shared memory– virtual tables
• lock file (.lk)
9
.db
_mprosrv
.bi
.lg10
_mprosrv
Shared Memory
.db
.bi
.lg10
Shared Memory Virtual tables
User id Lock TTY PID
11
u l t p
_mprosrv
Shared Memory Virtual tables
.db
.bi
.lg12
.lk.lg
u l t p
_mprosrv
Shared Memory Virtual tables
.db
.bi
12
User Login
• Unix Process
• Logs into virtual tables
• Logged in the log file (.lg)
13
u l t p
User 1
_mprosrv
Shared Memory Virtual tables
.db
.bi
.lk.lg14
u l t p
User 1
_mprosrv
Shared Memory Virtual tables_progres
.db
.lk.lg
.bi
14
u l t p
User 1
_mprosrv
Shared Memory Virtual tables_progres
u1
.db
.lk.lg
.bi
14
Shared Memory Virtual tables
User id Lock TTY PID
User 1 tty/01 12345
15
u l t p
User 1
_mprosrv
Shared Memory Virtual tables_progres
u1
.db
.lk.lg
.bi
16
u l t p
User 1
_mprosrv
Shared Memory Virtual tables_progres
u1
.db
.lk.lg
.bi
16
Data-Flow Discussion
• Data gets pulled into memory– User reads from memory– User never read from the database
17
.db
a b c
u l t p
User 1
_mprosrv
Shared Memory Virtual tables_progres
u1
.lk.lg
.bi
18
a b c u l t p
User 1
_mprosrv
Shared Memory Virtual tables_progres
u1
.db
a b c
.lk.lg
.bi
18
a b ca b c
u l t p
User 1
_mprosrv
Shared Memory Virtual tables_progres
u1
.db
a b c
.lk.lg
.bi
18
Data-Flow Discussion
• User Updates data– lock table
19
a b ca b d
u l t p
User 1
_mprosrv
c
Shared Memory Virtual tables_progres
u1
.db
a b c
.lk.lg
.bi
20
a b da b d
u l t p
User 1
_mprosrv
Shared Memory Virtual tables_progres
u1
.db
a b c
.lk.lg
.bi
20
Data-Flow Discussion
• Before-Image File– BIW– Data gets written to disk (.bi file)
21
.lk.lg
.bi
a b da b d
u l t p
User 1
_mprosrv
Shared Memory Virtual tables_progres
u1
.db
a b c
22
a b da b d
u l t p
User 1
biw
_mprosrv
Shared Memory Virtual tables_progres
u1
.db
a b c
.lk.lg
.bi
22
a b da b d
u l t p
User 1
biw
_mprosrv
bt
Shared Memory Virtual tables_progres
u1
.db
a b c
.lk.lg
.bi
22
a b da b d
c
u l t p
User 1
biw
_mprosrv
bt
Shared Memory Virtual tables_progres
u1
.db
a b c
.lk.lg
.bi
22
a b da b d
c
u l t p
User 1
biw
_mprosrv
bt
Shared Memory Virtual tables_progres
u1
.db
a b c
.lk.lg
.bi
22
a b da b d
c d
u l t p
User 1
biw
_mprosrv
bt
Shared Memory Virtual tables_progres
u1
.db
a b c
.lk.lg
.bi
22
a b da b d
c d
u l t p
User 1
biw
_mprosrv
bt et
Shared Memory Virtual tables_progres
u1
.db
a b c
.lk.lg
.bi
22
Data-Flow Discussion
• Review
23
a b da b d
c d
u l t p
User 1
biw
_mprosrv
bt et
Shared Memory Virtual tables_progres
u1
.db
a b c
.lk.lg
.bi
24
a b da b d
c d
u l t p
User 1
biw
_mprosrv
bt et
Shared Memory Virtual tables_progres
u1
.db
a b c
.lk.lg
.bi
24
a b da b d
c d
u l t p
User 1
biw
_mprosrv
bt et
Shared Memory Virtual tables_progres
u1
.db
a b c
.lk.lg
.bi
24
a b da b d
c d
u l t p
User 1
biw
_mprosrv
bt et
Shared Memory Virtual tables_progres
u1
.db
a b c
.lk.lg
.bi
24
a b da b d
c d
u l t p
User 1
biw
_mprosrv
bt et
Shared Memory Virtual tables_progres
u1
.db
a b c
.lk.lg
.bi
24
Data-Flow Discussion
• After-Imaging File– AIW
25
.ai
a b da b d
c d
u l t p
User 1
biw
_mprosrv
bt et
Shared Memory Virtual tables_progres
u1
.db
a b c
.lk.lg
.bi
26
a b da b d
c d
u l t p
User 1
biw
_mprosrv
aiw
bt et
Shared Memory Virtual tables_progres
u1
.db
a b c
.lk.lg
.bi
.ai
26
Data-Flow Discussion
• Asynchronous Page Writer– APW– Data gets written to .db
27
.db
a b d
a b da b d
c d
u l t p
User 1
biw
apw
_mprosrv
aiw
bt et
Shared Memory Virtual tables_progres
u1
.lk.lg
.bi
.ai
28
Data-Flow Discussion
• Final Review
29
a b da b d
c d
u l t p
User 1
biw
_mprosrv
bt et
Shared Memory Virtual tables_progres
u1
apw aiw
.db
a b d
.lk.lg
.bi
.ai
30
a b da b d
c d
u l t p
User 1
biw
_mprosrv
bt et
Shared Memory Virtual tables_progres
u1
apw aiw
.db
a b d
.lk.lg
.bi
.ai
30
a b da b d
c d
u l t p
User 1
biw
_mprosrv
bt et
Shared Memory Virtual tables_progres
u1
apw aiw
.db
a b d
.lk.lg
.bi
.ai
30
a b da b d
c d
u l t p
User 1
biw
_mprosrv
bt et
Shared Memory Virtual tables_progres
u1
apw aiw
.db
a b d
.lk.lg
.bi
.ai
30
a b da b d
c d
u l t p
User 1
biw
apw
_mprosrv
aiw
bt et
Shared Memory Virtual tables_progres
u1
.db
a b d
.lk.lg
.bi
.ai
30
a b da b d
c d
u l t p
User 1
biw
apw
_mprosrv
aiw
bt et
Shared Memory Virtual tables_progres
u1
.db
a b d
.lk.lg
.bi
.ai
30
a b da b d
c d
u l t p
User 1
biw
apw
_mprosrv
aiw
bt et
Shared Memory Virtual tables_progres
u1
.db
a b d
.lk.lg
.bi
.ai
30
SX.enterprise Concepts
• Client/server vs self serving clients
• Appserver
31
User 1
_mprosrv
Shared Memory
prowin32
.db
32
User 1
_mprosrv
Shared Memory
prowin32Server
.db
32
User 1
_mprosrv
Shared Memory
prowin32Server
.db
32
User 1
_mprosrv
Shared Memory
prowin32Server
AppServer
.db
32
Appserver Discussion
33
User 1
AppServer Database
Appserver Discussion
34
User requests customer address
34
User 1
AppServer Database
Appserver Discussion
34
User requests a price for a specific customer
34
User 1
AppServer Database
Appserver Discussion
34
User 1
AppServer Database
Appserver Discussion
34
User 1
AppServer Database
Appserver Discussion
34
Appserver Discussion - Summary
• Users are connected to Appserver and Database• Small requests do not use Appserver• Larger requests go through Appserver• Need to have large pipe between
Database Server and Appserver Server
35
Disconnecting Users - shutuser script
• Using shutuser
• Risks in shutting a user out
• Never use kill -9
36
User 1
_mprosrv
_progresu l t p
Shared Memory Virtual tables
u1
.db
37
User 1
_mprosrv
prowin32Server
u l t pShared Memory Virtual tables
u1
.db
38
Database Files & Crash Recovery
• .bi file
• when brokers start after crash, rebuilding shared memory not flushed to disk
• when truncate, flushes to disk
• .lg file
39
Power Outage Scenario #1
• Data Pulled into memory
• User changes data
• Data not written to .bi file
40
a b da b d
u l t p
User 1
biw
apw
_mprosrv
aiw
Shared Memory Virtual tables_progres
u1
.db
a b c
.lk.lg
.bi
.ai
41
Loss of Power #1
41
User 1
.db
a b c
.lk.lg
.bi
.ai
41
Power Restored #1
What happens to data?
41
u l t p
User 1
biw
apw
_mprosrv
aiw
Shared Memory Virtual tables_progres
u1
.db
a b c
.lk.lg
.bi
.ai
41
Power Outage Scenario #2
• Data pulled into Memory
• User changes data
• Data is being written to Before-Image file
• No “end transaction” written
42
a b da b d
c d
u l t p
User 1
biw
apw
_mprosrv
aiw
bt
Shared Memory Virtual tables_progres
u1
.db
a b c
.lk.lg
.bi
.ai
43
Loss of Power #2
43
User 1
c dbt
.db
a b c
.lk.lg
.bi
.ai
43
Power Restored #2
What happens to data?
43
u l t p
User 1
biw
apw
_mprosrv
aiw
Shared Memory Virtual tables_progres
u1
.db
a b c
.lk.lg
.bi
.ai
43
Power Outage Scenario #3
• Data pulled into Memory
• User changes data
• Data is being written to Before-Image file
• “end transaction” written
44
a b da b d
c d
u l t p
User 1
biw
apw
_mprosrv
aiw
bt
Shared Memory Virtual tables_progres
u1
et
.db
a b c
.lk.lg
.bi
.ai
45
Loss of Power #3
45
User 1
c dbt et
.db
a b c
.lk.lg
.bi
.ai
45
Power Restored #3
What happens to data?
45
u l t p
User 1
biw
apw
_mprosrv
aiw
Shared Memory Virtual tables_progres
u1
.db
a b d
.lk.lg
.bi
.ai
45
Crash Recovery - Summary
• Check the .lg file
• Truncate .bi file after crash
• Check last 3 minutes of work
46
NxTrend Installation Standards:
• File Systems
• /db (striped filesystem)
• /bi (separate disk)
• /rd (striped filesystem)
47
Directories: /db
• /db/nxt.db
• /db/nxt.bi
• /db/nxt.lg
• /db/nxt.lk
48
Directories: /db/sort
• /db/sort/*.DBI
• /db/sort/*.lbi
• /db/sort/*.srt
49
Directories: /bi
• /bi/nxt.bi
** Only if /bi can be put on it’s own dedicated spindle
50
Directories: /rd
• /rd/bin
• /rd/opsys/ (*.pf, *.sh)
• /rd/dlc
• /rd/dlc/bin
• /rd/src/ (*.p, *.i, *.h)
• /rd/cust/ (*.p, *.i, *.h)
• /rd/exec/ (*.r)
• /rd/lib/nxt.pl
51
Progress Libraries
52
Index .r files
Progress Libraries
53
Progress Libraries
• Replacing files
• Adding files
53
Index .r files
Progress Libraries
53
Index .r files
Progress Libraries
53
Index .r files
Progress Libraries
53
Index .r files
Progress Libraries
53
Review Directories and Libraries:
• /db
• /bi
• /rd
• nxt.pl
54
NxTrend’s Expectation Of A Database Admin:
• Maintenance
• Crash Recovery
• Troubleshooting
• Scheduled Work
55
Database Admin Maintenance:
• Disconnecting users
• Verify Backup Logs
• Monitor/maintain extent structure – prostats.log
• Purging Database log files
• Promon
• Maintaining Scripts
• Maintain .pf files
• Maintain library file
56
Database Admin Crash Recovery:
• Brokers – start & stop
• Removing .lk files
• Clearing shared memory
• Removing processes
• Disconnecting users
• Truncating BI files
• Error messages in log files
57
Database Admin Scheduled Work:
• Dump/loads
• Progress upgrades/patches
58
Progress Parameter
Discussion:• Database Broker Parameters
• Client Parameters
• BI file parameters for Truncate
59
Database Broker Parameters
/rd/opsys/nxtdb.pf
• –bibufs 30
• –spin 1
• –directio
• –B 3750
• –L 50000
• –n 120
• –db /db/nxt
• –g /bi/nxt.bi
• –H <hostname>
• –S <service name>
• –N TCP
• –minport
• –maxport
60
Client Parameters /rd/opsys/enterprise.pf
• –mmax 512
• –q
• –T /db/tmp
• –t (not needed on PC client)
• –h (not needed for single database)
• –db /db/nxt
• –g /bi/nxt.bi
61
BI File Parameters For Truncate
/rd/bin/truncate.bi
• –bi 1024 (Cluster)
• –biblocksize 16 (Block)
• –G 0
• –g /bi/nxt.bi
Cluster
Block
.bi file
62
Report Management:
• Report Manager– rptmgr– rptmgr– rptmgr
• Report Scheduler– rptsch.p
• rptrun.p
• rptrun.p
• rptrun.p
• New rptmgr hybrid (SX.enterprise)– rptmgr1– rptmgr2– rptmgr3
63
Report Management:
• Report Manager
• Start/Stop
• Troubleshooting
• rptlog in sasc.printdir– rptmgr.err in /usr/tmp– user definable parameters
64
Database Workshop
Working With Databases (hands-on)
65
Working with databases
• start broker
• identify database broker– shared memory– Unix process– lock file
• stop broker
66
Working with databases
• emergency shutdown
• log file
• progress editor (errors messages)
67
Working with databases
• start APW
• start BIW
68
Working with databases
• crash recovery
• truncate bi file
69
Single-Volume vs Multi-Volume
70
Single-Volume vs Multi-Volume
• .db, .bi • .db, .d1, .d2...
• .b1, .b2...
• no -g option needed
71
Progress File Size Limit
2GB72
Single-Volume vs Multi-Volume
• .db, .bi
• 2GB• .db, .d1, .d2...
• .b1, .b2...
• no -g option needed
• 256 @ 2GB
73
ExtentsVariable vs Fixed
74
Variable Extents
Variable Extents
New Record
75
Variable Extents
Variable Extents
New Record
OS
75
Variable Extents
Variable Extents
New Record
75
Variable Extents
Variable Extents
New Record
75
Variable Extents
Variable Extents
New Record
OS
75
Variable Extents
Variable Extents
New Record
75
Variable Extents
Variable Extents
75
Fixed Extents
Fixed Extents
76
Fixed Extents
Fixed Extents
New Record
76
Fixed Extents
Fixed Extents
76
Single-Volume vs Multi-Volume
• .db, .bi
• 2GB
• 2 step writes
• .db, .d1, .d2...
• .b1, .b2...
• No -g option needed
• 256 @ 2GB
• Single step writes
77
Inode Locking
78
Inode Locking
78
Single-Volume vs Multi-Volume
• .db, .bi
• 2GB
• 2 step writes
• One inode lock
• .db, .d1, .d2...
• .b1, .b2...
• No -g option needed
• 256 @ 2GB
• Single step writes
• Many inodes to lock
79
Single-Volume vs Multi-Volume - Summary
• NxTrend recommends using Multi-Volume• Multi-Volume is easier to administer• Multi-Volume supports larger databases• Multi-Volume has faster writes• Multi-Volume can have more than one
write at once
80
Record Management
• High Water Mark
• RM Blocks
• RM Chain
• Free Chain
• Empty Blocks
81
High Water Mark
Free Chain
Empty Blocks
RM ChainRM Blocks
Record Management
82
High Water Mark
Free Chain
Empty Blocks
RM ChainRM Blocks
Record ManagementIndexed
82
High Water Mark
Free Chain
Empty Blocks
RM ChainRM Blocks
Record Management
Indexed
82
High Water Mark
Free Chain
Empty Blocks
RM ChainRM Blocks
Record Management
82
High Water Mark
Free Chain
Empty Blocks
RM ChainRM Blocks
Record Management
82
Record Management Scenario #1
• New Record
• Checks first 3 RM Chain blocks
• Record fits
• New Record fills RM Chain block
• Block gets taken off RM Chain
83
Record Management
Free Chain
Empty Blocks
RM ChainRM Blocks
NewRecord
84
Record Management
Free Chain
Empty Blocks
RM ChainRM Blocks
84
Record Management
Free Chain
Empty Blocks
RM ChainRM Blocks
84
Record Management
Free Chain
Empty Blocks
RM ChainRM Blocks
84
Record Management
Free Chain
Empty Blocks
RM ChainRM Blocks
84
How full is full?
• Approx. 93% full
• So that record can grow in same block
• Database may grow after dump and load
85
Record Management Scenario #2
• New Record
• Checks first 3 RM Chain blocks
• Record does Not fit
• Block pulled from Free Chain
• New Record gets put on RM Chain
• First 3 RM blocks get moved to end of Chain
86
Record Management
Free Chain
Empty Blocks
RM ChainRM Blocks
NewRecord
87
Record Management
Free Chain
Empty Blocks
RM ChainRM Blocks
87
Record Management
Free Chain
Empty Blocks
RM ChainRM Blocks
87
Record Management
Free Chain
Empty Blocks
RM ChainRM Blocks
87
Record Management
Free Chain
Empty Blocks
RM ChainRM Blocks
87
Record Management
Free Chain
Empty Blocks
RM ChainRM Blocks
87
Record Management
Free Chain
Empty Blocks
RM ChainRM Blocks
1 2 3
87
Record Management
Free Chain
Empty Blocks
RM ChainRM Blocks
1 2 3
87
Record Management Scenario #3
• New Record
• Checks first 3 RM Chain blocks
• Record does Not fit
• No blocks left on Free Chain
• Block pulled from Empty Blocks
• New Record gets put on RM Chain
• First 3 RM blocks get moved to end of Chain
88
Record Management
Free Chain
Empty Blocks
RM ChainRM Blocks
NewRecord
89
Record Management
Free Chain
Empty Blocks
RM ChainRM Blocks
89
Record Management
Free Chain
Empty Blocks
RM ChainRM Blocks
89
Record Management
Free Chain
Empty Blocks
RM ChainRM Blocks
89
Record Management
Free Chain
Empty Blocks
RM ChainRM Blocks
89
Record Management
Free Chain
Empty Blocks
RM ChainRM Blocks
89
Record Management
Free Chain
Empty Blocks
RM ChainRM Blocks
1 2 3
89
Record Management
Free Chain
Empty Blocks
RM ChainRM Blocks
1 2 3
89
Record Management - Summary
• New Records only check first 3 RM Chain blocks• First 3 RM Chain blocks get moved to end of chain• If you delete data, you cannot guarantee space will
be reused• Full Blocks are approx. 93% full
90
Dump & Load Exercise: (hands on)
• What is a dump & load
• m&m analogy
• Why do a dump & load
• How often should you dump & load
• Enable VST
91
Dump & Load
92
Dump & Load
92
Dump & Load
93
Review:
• Question & Answer Session
94
Questions
?
?
?
?
?
?
?
?
?
?
?
?
95
Progress Data Flow Exercise
• DB
• Shared Memory
• Auto-Server
• User
• BIW
• BI
• APW
96
