View
99
Download
0
Category
Tags:
Preview:
Citation preview
A part of the Nordic IT group EVRY
Distributed Video Management and Security Systems
by Alexey Ostapov
Testing Tips and Tricks
Topics to Discuss
DIFFICULTIES IN TESTING OF COMPLEX SYSTEMS
DESIGNING TESTS FOR SECURITY SYSTEMS2
BUG HUNT …3
… AND REPRODUCTION SCENARIOS
AUTOMATED TESTING
TIPS AND TRICKS
1
5
6
4
What I am doing here?
Know how to start testingKnow how others avoid rakesKnow how getting things doneCompare with personal experience
Get some Fun
Why this presentation is interesting?
Who is this guy?
Alexey Ostapov5+ years in software testing5 years in security systems testing1 year as QA trainerCertified tester and so onTest leadJust nice guy
http://www.linkedin.com/in/ostapov I am here
Simple System Test Flow
1.Run Application2.Do Some Steps3.Get some error
4.Reproduce (do step 2)5.???6.PROFIT!
Video Security System Test Flow
1.Run Application2.Do Some Steps3.Get some error
4.Reproduce (do step 2)5.???6.Does not reproduced?
Big Bro SW
Complexity of “Big Brother Software”
Plugins
SDK
Plugins
Operating System
Drivers
Frameworks
DirectX
Updates
Environment
Network
Cameras
Sensors
Recorders
Monitors
Services
FirmwareEquipmentSettings
API
Big Bro From Inside
What Input for Test Design?
Specification
Specification
Specification
StandardsExisting systems
Common Sense
What Test Design Techniques We are Using?
Equivalence partitionsBitrate (256 Kbps, 1 Mbps, 5 Mbps, etc)Device type (SD, HD, Full / MPEG, .H264 cameras, etc)
Decision Tables and State TransitionDevice state (sees motion, in recording, not connected, etc)Network condition (high, low, pings / 100 Mbps, 1 Gbps, etc)Settings setup (Administrator, User, Another User / Schedules, Quality)
PairwiseOS (Windows family)Device typeNetwork condition
All known suits!
What Strategy Use?
What Strategy We are Using?
Risk based strategy?SW can be considered as safety-critical
Specification based strategy?Requirement are not completeRequirements changing frequently
Mixed strategy – YESWhy?
Our Testing Process
Prerelease System Test
Exploratory testing
Updating Test Cases
Smoke Testing
System Regression
Test
Duration testing
Risk based testing
Main Targets for Bug Hunt
OS and UI• Everything user can do must be tested:
• System Configuration• System Usage• System Breaking
Network• Normal flow• Heavy traffic• Disconnections
Devices• Default Setting• Reconfiguration• Interoperability
Hunting BUG
1.Keep Calm!2.Remember what you did3.Gather as much as possible info
SettingsDevices/System stateLogsScreenshotEtc
What to do if you have found new one 1st time?
Reproducing BUG
1. Try to reproduce in own environment2. Try to reproduce in another environment (ask your colleagues)3. Try to reproduce same conditions
1. Network2. Devices3. Events
4. Use additional software to reproduce1. Debugger2. Sniffer3. Performance Monitor
5. Prepare new environment from scratch6. Do NOT give up
CTQ? What a hell?
• Setup defined environment
• Run software in continuous video displaying mode
• Run Control Tools
• Wait defined time (72 hours minimum)
• Check software characteristics
• Perform smoke testing
How we do Critical-to-Quality duration testing
Why We are Doing CTQ?
Test stability over timeFind UI issuesFind memory leaksFind Crashes
Moments of Big Bro
GDI object crashApplication crashes randomly without any reasonsCause – GDI UI objects were never reused
.h264 decoder crashReproducible constantly but rare over some timeCause – camera sends bad frame
2.5 Gb RAM instant memory leakCause - parasite network traffic causes Win API alloc of huge memory
Kill process crashDoes not left any info about what happenedCause – internal crash in thread manager
Video recorder frequent rebootCause – FW used to work synchronously. Request were out of stack
Challenges make testing fun!
Questions?
Contact us!
Address:03056,24, Polyova Str., Kyiv, Ukraine
Phone:+38 044 457-88-56Email:info@infopulse.com.ua
Alexey Ostapov
Email:alexey.ostapov@ infopulse.com.ua
Thank youfor
Attention!
Recommended