Upload
dynatrace
View
100
Download
3
Embed Size (px)
Citation preview
Confidential, Dynatrace LLC 1
5 Steps for Identifying Deficiencies and Fixing Problems FAST
Gary CarrSoftware ArchitectAmerican Fidelity
Asad AliProduct Specialist
Dynatrace
slide 2
1. Know what’s important to ALL key stakeholders• Who identifies deficiencies?• Who discovers problems?• Who wants resolution?
2. Create specific criteria for PoC• Review current systems, know your gaps• Stay open minded
3. Work on Deployment4. Always look for the metrics5. Share the results
Confidential, Dynatrace LLC
The 5 Steps
slide 3 Confidential, Dynatrace LLC
Nine times on Forbes “Top 100 Places to Work”
Focus on 24x7 service to Customers, Colleagues,
Community
Technology supports relationships, rather than replacing
Partner support is key to growth
1.9 million policies, 1,800 employees
Who is American Fidelity?
slide 4 Confidential, Dynatrace LLC
In-house tools provided the basics
Before Dynatrace
• API, ESB, F5 URL calls• DNS related
problems• No visibility into
exceptions
slide 5 Confidential, Dynatrace LLC
• Two kinds of problems• Visible, but lots of time and resources• Invisible
Before Dynatrace
“It was like being in a dark room”Traci Tabyanan, Applications Manager, American Fidelity
slide 6 Confidential, Dynatrace LLC
• Business• More accurately measure
user satisfaction• Provide excellent customer
experience• Release applications more
often• Reduce technical debt
Reasons for Enterprise-ready APM
“We’re heading toward being a technology company that supplies insurance”Diana Bittle, VP of Software Solutions, American Fidelity
43k Users
slide 7 Confidential, Dynatrace LLC
Reasons for Enterprise-ready APM• Technical
• Reproduce production problems accurately in dev
• Understand how 3rd party tools operate
• Excessive database calls• Visual topology of complex delivery
chain• Between tiers and technology
platforms• Inconsistent performance between
test environments for same build• Code-level diagnostics
slide 8 Confidential, Dynatrace LLC
• Use of guesswork to identify root cause• Lack of visibility into the end user response time• Visibility gaps between application tiers• Need to reproduce problems to identify root cause• Necessity for war rooms for major problems
How to identify deficiencies in your APM
slide 9 Confidential, Dynatrace LLC
Don’t be like Michael Scott…
“the office” is a trademark of NBC
slide 10 Confidential, Dynatrace LLC
Prepare for Successful PoC• Focused on enterprise wide applications• Invited developers, testers, DevOps, security, and project managers• Shared applications and issues with APM experts• Set up before PoC began with specialists from our company and Dynatrace• Very specific requirements for specific tests
• Found a need in our company and used the PoC to solve• Visibility from individual’s browser to deepest calls in the system• In-depth analysis of resource consumption per transaction• Dashboards specific to each stakeholder
• Showed enterprise level• AND provided easy drill-down
slide 11 Confidential, Dynatrace LLC
• Define concrete and specific PoC success criteria• Use most significant application(s)• Stress test your environment with the APM tool• Invite all stakeholders to the PoC• Look for value DURING• Ensure apples-to-apples
Lessons Learned from 500+
slide 12 Confidential, Dynatrace LLC
• Problem #1: Most of the execution time for main app was spent in .NET remoting and ADO• + ID where most of
time was spent• Focus on reducing
technical debt in those areas
AF’s PoC: Key Findings
slide 13 Confidential, Dynatrace LLC
• Problem #2: Able to isolate remoting issue to one particular server• Been looking for a month• Saw because capture all
transaction, including exceptions• Seamless fit: infrastructure
and processes• Helped id specific code
changes
AF’s PoC: Key Findings
slide 14 Confidential, Dynatrace LLC
• Simple view of complex delivery chain
• Automatically created
• Auto-discovery removes guesswork
PoC: Key Findings
Dynatrace
Art!
slide 15 Confidential, Dynatrace LLC
• Database connectivity issues• Configuration settings • Remoting calls => $40M savings• Architecture anti-patterns• Memory leaks• N+1 query patterns
Other Key Findings
slide 16 Confidential, Dynatrace LLC
• Involve DevOps early!• Share successes• Early example:
• 2-day MIRT right after deployment, so were not using Dynatrace much yet…
• Let’s check one thing easier than thought
• Solution in 20 minutes• Benefit grows exponentially
• Now Dynatrace involved early in MIRT
Deployment Success
slide 17 Confidential, Dynatrace LLC
Metrics help build better apps, faster
More enrollments. Yeah!
slide 18 Confidential, Dynatrace LLC
Metrics help build better apps, faster
More processing. Boo!
• From 20 minutes to 2 hours
• Errors and unexpected timeouts
• Tried increasing planned timeouts
slide 19 Confidential, Dynatrace LLC
Metrics help build better apps, faster
• Quickly saw exact problem• Database query
taking 7 minutes Called 443 times
• Down to 56 calls• Straightforward
code change
slide 20 Confidential, Dynatrace LLC
Gasps + Clapping = Performance HeroEven as enrollments continue to grow, processing time remains low
slide 21 Confidential, Dynatrace LLC
Take-aways from a developer• Look at System Profiles> YourApp> Diagnose
Events> Exceptions• Customize your timeframe (i.e. 7PM – 9PM)• Use PurePath™ = “Intense Tracing”
• Look for method names• Look for long execution times• Keep drilling down• Sampling would not have found this issue
• Don’t be afraid to ask for help
slide 22 Confidential, Dynatrace LLC
• Coincidences bring nightmares• Sometimes the state of the system is the problem• Gap-free APM provides
• All the data leading up to issue, from every tier• Plus inter-tier stitching• Prioritize and reduce technical debt
• Faster fixes• “Customer cannot log in, it happened at 8:45 a.m.”• Just grab all the data from 8:44 a.m. to 8:46 a.m.
It’s all about the visibility
slide 23 Confidential, Dynatrace LLC
• Ability to compare slow transactions with fast transactions• Reduces the need to
reproduce problems• Easily help identify root cause
of pool exhaustion issues• Enables conversion of
application data to business data• Eliminates guesswork
Benefits of gap-free data
slide 24 Confidential, Dynatrace LLC
User Satisfaction• Now developers have a
closer connection to customers• APM shows impact of coding
methods• Developers can see real-time
user satisfaction
slide 25 Confidential, Dynatrace LLC
• Promotes DevOps culture• Reduces Ops anxiety over newer
code release• Enables frequent code releases• Make Dev an equal partner in
problem resolution• Promotes innovation
Dev and Prod synch serves customers
slide 26 Confidential, Dynatrace LLC
• War room calls directly show the power• Town Hall meetings: App Development, Business, Marketing,
and more• TV and tablets with HTML5• Results• Challenges!
How AF spreads the use of APM
slide 27 Confidential, Dynatrace LLC
• Promote key improvements to upper management
• Use APM tool to make the developers the “hero”
• Translate APM success to business $$$
• Display APM dashboards in public areas
• Use APM tool to reduce schism between Dev and Ops
Insights from other customers on spreading APM
slide 28 Confidential, Dynatrace LLC
• Continue to improve relationship with the business• Using dashboards• See their applications
• Dynatrace Community• Sharing information
• Training• User Groups and Conferences• New release strategy
Extra notes
slide 29 Confidential, Dynatrace LLC
•Define concrete PoC criteria•Use significant application(s) for PoC• Look for value DURING• Share success of APM results• Promote DevOps culture•Make Dev an equal partner in problem resolution
Summary
slide 30 Confidential, Dynatrace LLC
1. Know what’s important to ALL key stakeholders• Who identifies deficiencies?• Who discovers problems?• Who wants resolution?
2. Create specific criteria for PoC• Review current systems, know your gaps• Stay open minded
3. Work on Deployment4. Always look for the metrics5. Share the results
In Review: The 5 Steps
Copyright 2016, Dynatrace LLCConfidential, Dynatrace LLC 31
THANK YOU!