Upload
logan-allen
View
217
Download
0
Tags:
Embed Size (px)
Citation preview
2006.02.01http://www.javaservice.com
WonYoung Lee
for Web based System
Performance Analysis Mathematical Approach on
2
Question: What is the definition of Performance?
A.jsp B.jsp
http://192.168.0.2/a.jsp http://192.168.0.2/b.jsp
Unit Response time
0.5 secondsUnit Response time
4.0 seconds
3
Application A: Response Time Grahp
5 10 15 20 25
5
10
Concurrent Users
Resp
on
se T
ime(s
ec)
1
A.jsp B.jsp
http://192.168.0.2/a.jsp http://192.168.0.2/b.jsp
Unit Response Time
0.5 secondsUnit Response Time
4.0 seconds
4
5 10 15 20 25
5
10
Concurrent Users
Resp
on
se T
ime(s
ec)
A.jsp B.jsp
http://192.168.0.2/a.jsp http://192.168.0.2/b.jsp
Unit Response Time0.5 seconds
Unit Response Time
4.0 seconds
Application B: Response Time Graph
5
Comparison of A and B
5 10 15 20 25
5
10
Concurrent Users
Resp
on
se T
ime(s
ec) A
B
6
1. Objective of Performance
1.How many Clients2.Reasonable response time3.How to test or measure
7
1.1 Visit time & Think time 1. Definition
Click ! Click !
Response Time Think Time
Request Interval
Request Interval(sec) = Response Time(sec) + Think Time(sec)
Visit Time
New User Visited
Click ! Click ! Click ! Click ! Click ! Click !
Leave
8
1.2 Think time
Click ! Click !
Response Time Think Time
Request Interval
ThinkTime look like a constant value for the specific business domain
1. Definition
9
1.3 Concurrent User
Concurrent Users : 6 Time
Request Interval( ≈Think Time)
D
iffere
nt
IP A
dd
r ess
es
Visit Time
Definition
Concurrent User = ActiveUser + Inactive User
Note: HTTP is connection-less
1. Definition
Max Thinktime
Note: How many active users?
10
Request Interval
Response Time
Think Time
Concurrent User
1.4 Active UserTim
e
Click
Click
Active User
Active Service
Worker-Thread
System
1. Definition
11
1.5 Throughput
Time
Throughput(tps) =number of request
measure time(sec)
Unit : tph, tpm, tps, (pps, rps, ops, hit/sec)3,600 tph = 60 tpm = 1 tps
Measure Time
Definition
1. Definition
NOTE: Arrival Rate, Service Rate
12
2. Request/Response System Model
1.Mathematical Approach2.Queuing Theory3.Quantitative Analysis4.Measuring
13
2.1 Request and Response2. Request/Response System Model
Average Response Time
Throughput
Assumption: No ThinkTime
Question:What is the ability of her ?
Right Answer:Maximum throughput
# of Test Users
# of Test Users#
per
min
ute
s
Ave. R
esp
. Tim
e
14
2.2 MeasuringThroughput2. Request/Response System Model
Response Time
Virtual User (Assumption: ThinkTime=0)
Incr
easi
ng
Virtual User(ThinkTime=0)
Throughput(tps)
ResponseTime(sec)
15
2.3 Little’s Law 2. Request/Response System Model
Response Time(sec)
ActiveUser(ThinkTime=0)
Number of ActiveUser = Throughput(tps) x Average Response Time(sec)
Number of ActiveUser
Average Response Time(sec)Throughput(tps) =
ActiveUser
Throughput(tps)
ResponseTime(sec)
N
R
T
N = T x R
16
Active Service Request Interval
Response Time
Think Time
Concurrent User
Worker-Thread
2.4 ActiveUser’s Law
System Click
Ave. ResponseTime(sec)
Ave. ResponseTime(sec) + ThinkTime(sec)
ActiveUser = ConcurrentUser x
2. Request/Response System Model
Active User
ResponseTime’s Law
17
2.4.1 Proff of ActiveUser’s Law2. Request/Response System Model
ActiveUser
Resp.Time(sec)Throughput(tps) =
ConcurrentUserRequest Interval(=Resp.Time+ThinkTime)
Throughput(tps) =
Little’s Law
Resp.Time(sec)
Request Interval(=Resp.Time+ThinkTime)ActiveUser = ConcurrentUser x
ActiveUser
Resp.Time(sec)ConcurrentUser
Request Interval(=Resp.Time+ThinkTime)=
18
2.4.2 Meaning of Active User -1-
Request Rate Service Rate
Active User
Ave. ResponseTime(sec)
Ave. ResponseTime(sec) + ThinkTime(sec)
ActiveUser = ConcurrentUser x
2. Request/Response System Model
19
2.4.3 Meaning of Active User -2-2. Request/Response System Model
20
2.5 Concurrent User Equation2. Request/Response System Model
Definition
Equations
ResponseTime
Resp.Time + ThinkTime
= ActiveUser x ( 1 +
ConcurrentUser = ActiveUser x
Resp.Time(sec)
Request Interval(=Resp.Time+ThinkTime)ActiveUser = ConcurrentUser x
= Throughtput x ( ResponseTime + ThinkTime)
ActiveUser
Resp.Time(sec)Throughput(tps) = Little’s Law
)
= ActiveUser + ( Throughput x ThinkTime)
①
②
③
(Note: Throughput(tps) : ArrivalRate or ServiceRate)
ResponseTimeThinkTime
21
-Permanent Cookie-Thinktime-Arrival Rate-Responsetime-Active User(Service)
-- Concurrent User
2.6 Example of Concurrent User Monitoring2. Request/Response System Model
22
2.7 SLA & ThinkTime
SLA(Service Level Agreement) - Concurrent User : 5,000 - Average Response Time: less than 3 sec
SLA(Service Level Agreement) - Concurrent User : 5,000 - Think Time = 30 (Example) - Average Response Time: less than 3 sec
ThinkTime Agreement- Tele-market : 10-15sec- MIS Intranet : 15-20 sec- Internet Banking : 25-35 sec- Online Shopping Mall : 30-40 sec- Community : more longer Real ThinkTime Data
+
2. Request/Response System Model
23
2.8 Performance Test ThinkTime = 30
Virtual User(ThinkTime=30 ConcurrentUser)
Thro
ughput(
tps
)
5,000
Resp
onse
Tim
e
1.5
235.3
8,000
4.0
Throughput Graph
AverageResponseTime Graph158.7
ConcurrentUser = Throughtput x { ResponseTime + ThinkTime(30) }
5000 = 158.7 x ( 1.5 + 30 ) 8000 = 235.3 x ( 4.0 + 30 )
2. Request/Response System Model
24
2.9 Performance Test ThinkTime=0
Virtual User(ThinkTime=0 ActiveUser)
Thro
ughput(
tps
)
238
Resp
onse
Tim
e
158.7
235.3
941
Throughput Graph
Average Response Time Graph
1.5
4.0
ConcurrentUser 5,000 8,000
ConcurrentUser = ActiveUser + Throughput(tps) x ThinkTime(sec)
ConcurrentUser = ActiveUser x { 1 + ThinkTime(sec)
Resp.Time(sec)}
ConcurrentUser = Throughput(tps) x { Resp.Time(sec)+ThinkTime(sec)}
2. Request/Response System Model
25
2.10 Queuing Theory - G/G/1
λ μ
Response TimeUtilizationActiveUser
Throughput (λ<μ)
(λ<μ)
Arrival Rate Max Throughput
2. Request/Response System Model
26
Concurrent User
req/s
ec
??
Concurrent User Concurrent User
Arrival Rate(λ)
Active User(N) Response Time(R)
Response TimeActiveUser
Arrival rateArrival rate
2.11 Maximum Concurrent User2. Request/Response System Model
27
2.12 Saturation Point, Buckle Zone2. Request/Response System Model
28
Tuning ? What’s mean?
2.13 Understanding of Throughput Graph2. Request/Response System Model
29
2.14 Throughput and Active User
Demo
2. Request/Response System Model
30
3. Multiple Applications
- Different Hit Ratio- Different Performance- Homogeneous/heterogeneous Bottleneck Condition
2. Request/Response System Model
31
3.1 2-Application model3. Multiple Applications
40
TPSmax
20
40
20
A B
40
20
10
40
30
20
A BA B 5
40
1015
A B
Active Threads
40TPS
Saturation Point
A application
40
20
TPS
Saturation Point
B application
Active Threads40
Active Threads
40TPS
4020
20
10
A
B
Active Threads
40TPS
4010
30
5
A
B
30 Active Threads
40TPS
40
1510
B
A
10 30
Graph 1 Graph 2 Graph 3
1 : 1 3 : 1 1 : 3
TPSmax TPSmax TPSmax
32
3.1.1 2-Applicatoin Model
40A
20
10
20
30
B10
graph 1
graph 3
graph 2
O
Different Arrival Rate
Limited by same resource bottleneck
Homogeneous bottleneck condition
3. Multiple Applications
33
xr1 r2
3.1.2 2-Applicatoin Model
T2
y
T1
x
xT1
yT2
+ = 1
=
Q
P
O
(r1,r2)r2
r1
A: r1 (req/sec)B: r2 (req/sec)
R2
R1
y
3. Multiple Applications
34
3.2 3-Application Model
Q
O T2
T3
y
z
P
(0, T2, 0)
(0, 0, T3)
(0, 0)
r1
r2
x
T1
r3
x
T1
y
T2
+ = 1z
T3
+
xr1
yr2
=zr3
=
(T1, 0, 0)
(r1, r2, r3)
3. Multiple Applications
35
3.3 n-Applicatoin Model
NOTE: Under Homogeneous Bottleneck Condition
3. Multiple Applications
( Critical Inequality Performance Equation )
36
3.4 Performance Utilization
T2
T3
y
z
x
T1
x
T1
y
T2
+ = 1z
T3
+
App. Ti
TPSmax
ri
req/sec
ri / Ti
x 400 48.2 0.12
y 9 1.2 0.13
z 50 22.7 0.45
0.12 + 0.13 + 0.45 = 0.70
∑ { ri / Ti } x 100 = 70 %
Q
P
∑ { ri / Ti } ≤ 1.0
3. Multiple Applications
Utilization of Critical Performance
37
3.5 Revision of Utilization of Critical Performance
App Ti
TPSmax
ri
req/sec
ri / Ti
a1 42 5.4 0.129a2 9 1.9 0.210a3 70 10.7 0.153a4 15 4.5 0.300
Sub ∑ri =22.5 ∑ { ri / Ti } = 0.79
… … … …
TOTAL 25(req/s)
a.jspa.jsp : 5.4 req/sec
b.jspb.jsp : 1.9 req/sec
c.jspc.jsp : 10.7 req/sec
d.jspd.jsp : 4.5 req/sec
…..
Arrival RateArrival Rate
∑ { ri / Ti } ≤ ρ* 1.0 (ρ is total hit ratio of ∀ai)
∑ { ri / Ti } / ρ = ∑ { ri / Ti } / (22.5/ 25) = 88%
3. Multiple Applications
38
Q
O
z
P
(0, 0)
r2
x
r3
xr1
yr2
=zr3
=
(r1, r2, r3)
Arrival rate line
bottleneck saturation flat
Bottleneck surface
∑ { ri / Ti } ≤ 1.0 +ε(ε>0)
3.6 Multiple Bottleneck Theory
Heterogeneous bottleneck condition
3. Multiple Applications
Critical Inequality Performance Equation (final)
39
3.7 Performance Matrix
App. Arrival Rate(tps) Arrival Ratio MaxThroughput
λi /Τi
a1 λ1 λ1 /λ=ρ1 Τ1 λ1/Τ1
a2 λ2 λ2 /λ=ρ2 Τ2 λ2/Τ2
a3 λ3 λ3 /λ=ρ3 Τ3 λ3/Τ3
… … … …
an λn λn /λ=ρn Τn λn/Τn
(Others) λothers 1-ρ(Ex.20%) N/A N/A
SUM ∑ λi + λothers = λ ∑ρi + 0.2 = 1.0 N/A ∑ λi/Τi
∑ λi/Τi ≤ ρ(1.0+ε) (ε≥0)
(ρ is total hit ratio of a∀ i)Critical Performance Utilization:
∑ λi/Τi
ρ(1.0+ε)(When ConcurrentUser = N)(ε≈ 0)
≤1.0
NOTE: 8:2 Rule
3. Multiple Applications
40
3.8 Objective of Performance Estimation
Arrival RateConcurrent Users
N λ
kN kλ
Current Performance Utilization :
∑ λi/Τi
ρ(1.0+ε)1.0 (100%)
x k
Current Current
FutureFuture
3. Multiple Applications
How many concurrent users can be accepted?
How many times than now?
41
3.9 Performance Test for Multiple Scenario
App. Ratioρi
a1 0.5
a2 0.2
a3 0.1
… …
(Others) 0.2(20%)
SUM 0.8(80%)
ThinkTime = 28.5 sec(Example)
Response Time Graph
Virtual User(ThinkTime = 28.5 sec)
942
Acceptable Response Time
3. Multiple Applications
42
테스트 명 Hit Count Arrival Rate R(Ratio) T(TPS)예상
Rmax(Req/ sec)
ArrivalRate/ T
R/ T 비고
1. (listItemInfo)공매공고 297 0.083 0.14 16.25 0.14Rmax 0.005 0.0086
2. (listitem)매각물건 288 0.080 0.13 1.5 0.13Rmax 0.053 0.0867 J AVA EXCEPTION ERROR
3. (viewitm_real)매각물건 180 0.050 0.08 18.5 0.08Rmax 0.003 0.0043
4. (index)홈페이지 116 0.032 0.05 2.75 0.05Rmax 0.012 0.0182 CPU 100% FULL
5. (searchItemByDetailInfo)파워검색 109 0.030 0.05 8 0.05Rmax 0.004 0.0063 8 DB 95% FULL명일때 서버6. (listImpendingBidItem)입찰임박물건 161 0.045 0.07 28 0.07Rmax 0.002 0.0025 CPU 70%웹 서버 사용율7. (searchZipAddress_01)주소검색 55 0.015 0.03 60 0.03Rmax 0.000 0.0005 CPU 100%웹 서버 사용율8. viewItem_bond물건정보 53 0.015 0.02 19.5 0.02Rmax 0.001 0.0010 CPU 90%웹 서버 사용율9. (listKamcoAucNotice)공매공고 48 0.013 0.02 2.5 0.02Rmax 0.005 0.0080 8 DB CPU 100%명일때 서버 사용율10. newAucNotice새로운공고 45 0.013 0.02 4 0.02Rmax 0.003 0.0050 6 DB CPU 100%명일때 서버 사용율11.FAQ ViewFAQ보기 44 0.012 0.02 24 0.02Rmax 0.001 0.0008 WEB CPU 80%서버 사용율12.LISTFAQListFAQ 43 0.012 0.02 19.5 0.02Rmax 0.001 0.0010 DB , WEB CPU 90%서버 서버 사용율13. -viewAucNotice새로운공고 43 0.012 0.02 24 0.02Rmax 0.000 0.0008 - FULL웹 서버 다운 세션14.ID checkDuplicateUserId중복확인 33 0.009 0.02 67 0.02Rmax 0.000 0.0003 CPU - 94%웹 서버 사용율15. listItemInfoBySched공매일정 60 0.017 0.01 22.5 0.01Rmax 0.001 0.0004 CPU - 90%웹 서버 사용율16. listAucNotice통합공고 26 0.007 0.01 9 0.01Rmax 0.001 0.0011 DB CPU 100%서버 사용율17. welcomeRegisterForm회원가입 24 0.007 0.01 22.5 0.01Rmax 0.000 0.0004 CPU 90%웹 서버 사용율18. createUserForm회원가입동의 22 0.006 0.01 23.5 0.01Rmax 0.000 0.0004 CPU 80%웹 서버 사용율19. listDocForm부가정보 21 0.006 0.01 32 0.01Rmax 0.000 0.0003 CPU 90%웹 서버 사용율20. searchItem파워검색 21 0.006 0.01 19 0.01Rmax 0.000 0.0005 CPU 90%웹 서버 사용율21. listNewItem새로운물건 19 0.005 0.01 4 0.01Rmax 0.001 0.0025 DB CPU 100%서버 사용율22. loginForm로그인폼 19 0.005 0.01 25.5 0.01Rmax 0.000 0.0004 CPU 90%웹 서버 사용율23. contractPersonalUser이용약관 17 0.005 0.01 24 0.01Rmax 0.000 0.0004 CPU - 70%웹 서버 사용율24. login로그인 16 0.004 0.01 3.5 0.01Rmax 0.001 0.0029 DB CPU 100%서버 사용율
Sum 0.489 0.79 0.79Rmax 0.095 0.1535
Maximum adaptable concurrent
user: 219
Expected maximum TPS : 6.515(tps)
Maximum adaptable concurrent user : 202
Expected maximum TPS : 6.719 (tps)
Result of Test using Multiple
Scenario at the same time
Result of Test using each application
test.
3.10 Example of Performance Test
1 + 1 = 2.5 ?
3. Multiple Applications
43
3.11 Termination about Performance Test
Load Test
Stress Test
Availability Test
Performance Test
44
Relative Performance Problem SQL Query bottleneck (DB index, Full Scan, Heavy Query)
Bottleneck on back-end transactions (CICS,TUXEDO, TCP/IP
Socket)
Relative bad performance on specific application
(synchronized, CPU time)
Relative bad performance on most of application caused by
too small H/W
Other side issue, for example, Network bottleneck.
Two Types of Performance Problem
Conditional Performance Problem JDBC Connection resource leakage
Memory leakage (need too large memory, Memory Leak, Native
Memory Leak)
Unbalanced WAS tuning (Pool Size, Number of Thread, Heap Size)
Caused by a bug on JVM/WAS/JDBC (Sybase JDBC, JVM Bug, WAS
Bug)
Thread Lock/Dead Lock (Application issue, Firewall issue)
45
Relative Performance Problem
Request Rate Service Rate
Active User
46
Relative Performance Problem: Ramp up Test
47
Relative Performance Problem: Ramp up Test
48
JDBC Connection resource leakage
Memory leakage (need too large memory, Memory Leak, Native Memory
Leak)
Unbalanced WAS tuning (Pool Size, Thread 개수 , Heap Size)
Caused by a bug on JVM/WAS/JDBC (Sybase JDBC, JVM Bug, WAS Bug)
Thread Lock/Dead Lock (Application/Framework issue, Firewall issue)
Database lock caused by uncommited nor unrollbacked
database issue (buffer full, unexpected Batch Job, ..)
upload or download a large file
unexpected infinite loop on an application: CPU 100%
Disk/memory Full
bad performance on specific application or specific usres
Conditional Performance Problem
49
S80(12-way,8GB, 78,126 tpmC)S80(6-way,6GB, 41,140 tpmC)
Visitors a day: 4,506 usersPeak Concurrent users: 275 usersPeak Arrival Rate : 18.3 tpsRequest Interval : 18 secVistors vs Concurrent users: 6.1%
CPU : ?(unkown, no issue)119,266tpmC/18.3tps = 6,517 tpmC/tps
3.1 Issue on Capacity Planning
3.1.1 K-university KMS System
50
3.1.2 B company
M80 (4-way, 3GB, 34,588 tpmC) x 2S80 (12-way, 8GB, 78,126 tpmC) x 26F1 (4-way, 4GB, 44,500 tpmC) x 1H80 (2-way, 1GB) x 8
[Seoul:M80(4-way)x2: 69,176 tpmC ]Visitors a day: 1,621(total 3,435)Peak Concurrent Users: 600 usersHits a day : 466,639 hitsPeak Arrival Rate : 20.0 tpsRequest Interval : 30 secAverage Visit time: 1:25:43 Visitors vs Concurrent Users: 31%CPU Utilization: 70-100%69,176tpmC/20tps= 3,459 tpmC/tps
6F1(4-way,4GB, 44,500 tpmC ) Added.CPU Utilization: 60%113,676tpmC/20tps= 5,684 tpmC/tps
51
3.1.3 N-bank Internet banking
M80(4-way, 4GB, 34,588 tpmC) x 3
Visitors a day: 96,753 usersPeak Concurrent Users: 1,500-2,000 usersHits a day : 1,795,867 hitsPeak Arrival Rate : 58.3 tpsRequest Interval : 25.7 secVisit time: 6min 25secAverage hits per visit time: 18.6 clicksVisitors vs Concurrent Users: 1.6-2.0%
CPU Utilization: 70%103,764 tpmC/58.3tps= 1,780tpmC/tps
52
3.1.4 J-bank CRM System
S85(12-way, 32GB, 124,818tpmC) X 2+ HOST DB
Visitors a day: 5,028 usersPeak Concurrent Users: 250 usersHits a day : 264,060 hitsPeak Arrival Rate : 10 tpsRequest Interval : 25.3 secVisit time: 4min 10secAverage hits per visit time: 9.9 clicksVisitors vs Concurrent Users: 4.97 %
CPU Utilization: 30-40% 249,636tpmC / 10 tps = 24,964 tpmC/tps
53
3.1.5 D-insurance e-Hanaro System
WAS 6H1(4-Way 4 GB, 40,763 tpmC)DB H70(4-Way 4GB, 17,134 tpmC)+HOST CICS
Visitors a day: 2,800 user(resistered 3,300)Peak Concurrent Users: 350 userHits a day : 301,190 hitsPeak Arrival Rate : 33 tpsRequest Interval : 9 secVisit time: 16 minAverage Hits per visit time: 120.5 clicksVisitors vs Concurrent Users: 12.5 %
CPU Utilization: 50%40,763 tpmC / 33 tps= 1,235 tpmC/tps
54
3.1.6 D company
WAS H80(2-way, 2GB, 14,756(?) tpmC)DB S80(12-way, 8GB, 67,908tpmC)
Visitors: 258 usersPeak Concurrent Users: 45 userHits a day : 65,192 hitsPeak Arrival Rate : 4.2 tpsRequest Interval : 11-18 secVisitors vs Concurrent Users: 17.4 %
CPU Utilization: 50-60% (DB CPU: 35%)14,756 tpmC / 4.2 tps= 3,513 tpmC/tps
55
3.1.7 K-bank eCRM System
WAS H80 (2-way, 4GB, 14,756(?) tpmC) x 2DB M80(2way-4GB, 18,647(?) tpmC)
Visitors a day: 37,951 usersPeak Concurrent Users: 230 usersHits a day : 235,527 hitsPeak Arrival Rate : 5.83 tpsRequest Interval : 24.7 secVisit time: 2 min 33 secAverage hits per visit time: 6.2 clicksVisitors vs Concurrent Users: 0.6 %
CPU Utilization: ?(unknown, no issued)29,512 tpmC / 5.83 tps= 5,062 tpmC/tps
56
3.1.8 K-Card
M80 (4-way, 4GB, 34,588 tpmC) x 9WSBCC , Servlet/JSP, CTG
Visitors a day: 5,323 usersVisitors an hour: 4,500 usersPeak Concurrent Users: 2,800 usersHits a day : 6,198,133 hitsPeak Arrival Rate : 217 tpsRequest Interval : 23 secVisitors vs Concurrent Users: 53 %
CPU Utilization: (see left bottom side grahp)311,292tpmC / 217 tps= 1,435 tpmC/tps
[2nd machine]Peak Arrival Rate : 31.7 tpsPeak Concurrent Users: 600 users34,588 tpmC / 31.7 tps= 1,091 tpmC/tps
[4th machine]Peak Arrival Rate : 50 tps34,588 tpmC / 50 tps= 692 tpmC/tps
CPU Utilization
57
3.1.9 Result of my statistics
tpmC / 1 tps(based on 70% cpu)
58
Analysis Methodology
3.2 Bad Process for Perf. Mgmt.
Proposal Pilot BMTAnalysis/
DevelopingFunction
estTesting with
Small real users
SystemOpen
Monitoring
Unit App.Perf.Test
Perf.Prediction
Perf. DataLogging
Additional App. Develop
ApplyingApp.
WorkloadAnalsys
Experienceddata
CapacityEstimation
BMT methodology
AppTuning
Perf Analysis
Unit App.Perf.Test
Perf.PredictionPerf Analysis
59