Learning Design StandardDigital Testing

Version: 1.0

Table of contentsRevision history.....................................................................................................3Using this Learning Design Standard....................................................................3Overview of digital testing......................................................................................3Target learners.......................................................................................................4Pathways to digital testing.....................................................................................4Capabilities needed for digital testing....................................................................5Relevant SFIA skills...............................................................................................7References............................................................................................................8Key content areas..................................................................................................9

Unit 1. Testing fundamentals............................................................................9

Unit 2. Automation...........................................................................................11

Unit 3. Test management................................................................................12

Unit 4. Specialisation.......................................................................................13

29 November 2018 Building Digital Capability Program – Learning Design Standard – Digital Testing 2

Revision historyDate Version Contact Content6/11/2018 0.1 Grant Nicholson First exposure draft30/11/2018 1.0 Grant Nicholson Release version

Using this Learning Design StandardThe Australian Public Service Commission (APSC), in conjunction with the Digital Transformation Agency (DTA), has developed a collection of Learning Design Standards (LDSs) to assist with the development of capabilities needed by the Australian public service to progress the digital transformation of government services.

You should read this LDS in conjunction with the Digital Learning Design Standard Reference Guide, which contains guidance for providers who may wish to provide learning solutions to APS agencies and to agencies seeking digital capability uplift though learning solutions based on these standards. The reference guide also describes the context, range of uses and other relevant details about the collection of LDSs.

All queries relating to this standard should be directed to: LearningDesign@apsc.gov.au.

Overview of digital testingThe primary function of digital testing is to validate that the products produced by the multi-disciplinary digital delivery team function appropriately in meeting the needs of the users and other stakeholders. Digital testing also validates the environment in which products operate (to make sure test results are valid) and automates where possible to accelerate the product delivery timeframe and reduce the associated cost of implementing the products.

29 November 2018 Building Digital Capability Program – Learning Design Standard – Digital Testing 3

Target learners PrimaryManual testers wishing to stay relevant in the modern world of testing. User Researchers interested in developing their automation skills. SecondaryAnyone with an interest or rudimentary exposure to coding, and who has a keen eye for detail or a passion for solving problems.

Pathways to digital testingEverybody has a different work history and career path. The following are some of the more common fields people may have come from before becoming a digital tester:

Manual testers User researchers Quality assurance officers Business analysts Developers IT support officers Recent IT or Engineering graduates (i.e. STEM) System Analysts Business front line staff

Pre-requisite qualificationThis learning design standard (‘this standard’) is intended to work in parallel with International Software Testing Qualifications Board (ISTQB) Foundation Level 2018 qualifications.

The material covered in the ISTQB Foundation Level 2018 syllabus is not duplicated in this standard. Any training program intended to upskill non-testers should incorporate the ISTQB Foundation content. This may be achieved by replicating that content in a single comprehensive training package aligned with this standard or may involve sending the candidates on a separate training program based on the ISTQB Foundation syllabus in addition to a training package that contains only the material in this standard.

Other relevant qualifications & certifications The following qualifications are relevant to the capability described in this LDS:

All ISTQB (International Software Testing Qualifications Board) qualifications ISTQB Agile Testing Extension ISTQB Test Automation Engineer

Capabilities needed for digital testingThe skills, knowledge and attributes described in this section are the minimum needed for someone to be effective as a digital tester. A person undertaking the learning defined by this

29 November 2018 Building Digital Capability Program – Learning Design Standard – Digital Testing 4

standard should already have the listed pre-requisites, and by the completion of their learning, they should have all the knowledge and skills described below. They may need experience of these in a workplace environment to embed the learning and become effective.In this table, ISTQB refers to essential content that is part of the ISTQB Foundation Level syllabus (see pre-requisite qualification, previous section).

KnowledgeTopic

ReferenceOrganisational context   •      Enterprise architecture 1.1•      DTA Digital Service Standard 1.1•      Agency testing framework and policies 1.1•      Privacy and ethics policies 3.4 Methodologies, procedures and standards   •      Delivery methodologies 1.2, 1.5•      Agile delivery 1.2•      Types of testing ISTQB•      Testing techniques ISTQB•      Software Development Lifecycle (SDLC) ISTQB Tools   •      Automated testing tools 2.2•      Test management tools 3.3 Theory/theoretical   •      Design thinking 1.3 Concepts   •      Continuous delivery 4.8

Professional SkillsTopic

ReferenceTechnical   •      Scripting/coding 2.4•      Data management 3.2, 4.6, 4.10   Analysis, synthesis & evaluation  

29 November 2018 Building Digital Capability Program – Learning Design Standard – Digital Testing 5

   •     Investigation/research Prerequisite•      Problem decomposition Prerequisite Planning and organising    •      Test planning ISTQB•      System behaviour modelling 1.6•      Risk management ISTQB & 1.7 Relationships and interpersonal   •      Stakeholder management 3.4   

29 November 2018 Building Digital Capability Program – Learning Design Standard – Digital Testing 6

Attributes / Behavioral SkillsProfessional

• Innovation

• Continuous improvement

• Passion for quality

• Out-of-the-box thinking

• User empathy

• Outcome focused

• Risk management

• Attention to detail

• Advocate for testing values

• Cross-team collaboration

Digital

• Embrace change

Personal

• Ability to say no

• Resilient

• Pragmatic

• Curious

• Initiative

• Flexible

• Quickly acquire knowledge

Relevant SFIA skillsThe Skills Framework for the Information Age (SFIA) is a global standard that defines Digital and other ICT related skills. A person possessing the following SFIA skills at the levels indicated would be capable of performing the role described by this standard.Code Skill Applicable

LevelsCaveats*

TEST Testing 4USEV User experience evaluation 4PROG Programming/software development 3BPTS Business process testing 4METL Methods and tools 3*Caveats are identified components of a SFIA skill that are not explicitly required for the current role. For the purpose of this Learning Design Standard the SFIA description should be read as though the caveated components were not included in the SFIA skill description.

29 November 2018 Building Digital Capability Program – Learning Design Standard – Digital Testing 7

References International Software Testing Qualifications Board (ISTQB) Australia and New Zealand Testing Board (ANZTB) DTA Digital Service Standard What is Agile Testing ? Agile Testing and Quality Strategies: Discipline Over Rhetoric Independent Testing and Agile Teams Blog: Agile Methodology - The Complete Guide to Understanding Agile

Testing Blog: Agile Testing - The Role of QA in Agile Chaos Engineering: A cheat sheet GitHub list of Awesome Chaos Engineering Resources Test Automation Frameworks (SmartBear)

29 November 2018 Building Digital Capability Program – Learning Design Standard – Digital Testing 8

Key content areasThe following table outlines content areas that need to be covered in any learning program designed to deliver the digital testing capability. You should read this in conjunction with the ISTQB Foundation Level syllabus (see pre-requisite qualifications). Unit = area of learning. Topic = Component of area of learning.

Unit 1. Testing fundamentals  Learning objective: Describe testing concepts and methodologies and the environment/context in which they operate.

Topic title Topic learning objectives Critical content

1.1 Government & organisational context

Identify government and organisational policies, frameworks, application landscape, processes and standards

1 Digital Service Standard (DSS)2 WCAG standard3 Identify the established organisation

architectures, such as Business Information Data Technology Application Cloud Security

4 Identify the established organisation software development and testing framework and knowledgebases

1.2 Agile delivery (vs. alternatives)

Describe agile vs. waterfall delivery

1 How agile approaches are used in the creation of products and services

2 How agile differs from waterfall3 The roles and functioning of a

multidisciplinary agile digital delivery team

4 Agile testing relates to ‘early and targeted’, risk based and usually automated testing.

1.3 Design thinking Describe design thinking and user centred design

1 Design thinking focuses on defining the problem, researching, forming ideas, prototyping and testing

2 User centred design puts the user at the heart of the design process

3 In agile delivery with user centred design, iteration loops help refine the product design to meet user needs

29 November 2018 Building Digital Capability Program – Learning Design Standard – Digital Testing 9

1.4 Automated vs. manual testing

Describe automated testing

Identify common testing tools

1 The differences between automated and manual testing (including why automate)

2 The differences between automated testing development and software engineering

3 Common testing tools and their features4 Challenges involved with automated

testing in certain domains1.5 Testing in various development approaches

Describe the common development approaches and nature of testing in them

1 Test Driven Development (TDD) 2 Behavior Driven Development (BDD)3 Acceptance Test Driven Development

(ATDD)

1.6 Advanced testing approaches

Describe how testing differs in each of these testing approaches and with the above development approaches

Demonstrate system behaviour modelling

1 Exploratory Testing 2 Session Based Testing (SBT)3 Chaos Engineering4 System behaviour modelling5 Circumstances in which these

approaches are warranted

1.7 Risk management

Describe the application of risk management to testing

Identify critical elements of a testing scope risk assessment

Demonstrate risk assessment of a testing activity

1 Why use a risk management approach?2 Identifying risks involves understanding

the environment architecture and user psychology

3 Setting the testing scope appropriately4 Documenting and reporting on the risk

assessment

Unit 2. AutomationLearning objective: Demonstrate the execution, creation and tuning of an automated test.

Topic title Topic learning objectives Critical content

2.1 An automated testing framework

Identify the key requirements of an automated testing framework

1 Types and features of testing automation frameworks

2 Considerations in selecting the appropriate automation framework

2.2 Specialised Identify specialist 1 Specialised tools and what they are

29 November 2018 Building Digital Capability Program – Learning Design Standard – Digital Testing 10

automated testing tools

automated testing tools and sources of knowledge about their application.

used for2 Communities of practice3 Vendor specific bodies of knowledge

2.3 Designing automated tests

Design an automated test

1 Design considerations, such as; Focus on highest risks first Breadth and depth of scope Efficiency/performance reusability

2 Using design patterns

2.4 Constructing automated tests

Demonstrate the creation of automated tests

1 Explore options for implementing the test design using the tools available

2 Build a test using; Record, play-back and tune Clone and modify an existing test Code the test from scratch

3 Trial execution to verify test operates as expected

2.5 Executing automated tests

Demonstrate the execution and collation of results from a pre-existing automated test

1 Confirm dependencies of the test are met

2 Schedule and trigger the test 3 Verify test execution4 Assess and record results

2.6 Tuning automated tests

Describe the automated test tuning process

1 Considerations when tuning; Identify what needs tuning Identify variables that can or should

be changed Consider follow-on impact of all

changes Consider data implications (for this

and other tests)

Unit 3. Test managementLearning objective: Describe approaches to the management of a testing environment and the achievement of testing outcomes.

Topic title Topic learning objectives Critical content

3.1 The testing environment

Describe the requirements of a testing environment

1 Using the live environment vs. a dedicated test environment

2 Considerations in replication of part or full live environment

3 Appropriate test data is critical4 Constructing barriers to prevent tests

29 November 2018 Building Digital Capability Program – Learning Design Standard – Digital Testing 11

impacting on live users/data

3.2 Test data Define ‘appropriate test data’

Demonstrate the anonymisation of data

1 Test data should enable all tests to give meaningful results

2 Most useful if test data replicates live data patterns (including partial state transactions)

3 The importance of anonymising cloned live data used for testing purposes

3.3 Selection of testing tools

Describe the considerations when selecting appropriate testing tools

1 Assess suitability of existing (and potential) corporate tools for; compatibility with testing environment

& organisation applications capability & features performance re-usability of tests automation

3.4 Managing testers (including end-users as testers)

Describe the management of testers, including end-user testers

Demonstrate appropriate selection of testers

1 Privacy and ethics issues and policies2 Estimating the size and complexity of

testing for a given project3 Selection of appropriate testers4 Orientation and support of testers5 Coordinating testing across multiple

(possibly dispersed) testers6 Collating results and providing feedback

to testers

Unit 4. SpecialisationLearning objective: Demonstrate an in-depth familiarity with the concepts, methodologies, bodies of knowledge and tools of one or more of the following testing specialisations.

Topic title Topic learning objectives Critical content

4.1 Security testing Define security testing

Describe the approaches to security testing

Identify the main considerations in security testing

1 Security standards and the Information Security Manual

2 The sorts of vulnerabilities that affect hardware, software and networks

3 The role of encryption4 Types of security testing, such as;

Verification against security architecture

Penetration testing Ethical hacking

29 November 2018 Building Digital Capability Program – Learning Design Standard – Digital Testing 12

5 Human behaviour/psychology considerations

6 Tools and knowledgebases4.2 Infrastructure testing

Define infrastructure testing

Describe the approaches to infrastructure testing

Identify the main considerations in infrastructure testing

1 Understanding the infrastructure architecture & infrastructure components (including cloud services)

2 Determining the appropriate scope of testing

3 Specialised infrastructure testing equipment

4.3 Mobility testing Define mobility testing

Describe the approaches to mobility testing

Identify the main considerations in mobility testing

1 The differences in application behaviour and user experience between mobile devices and non-mobile devices

2 Designing test approaches specifically for mobile platforms

3 Determining which mobile platforms/devices to include in testing

4 Special considerations in testing on mobile platforms (e.g. difficulty in automation)

5 Determining relevance of test results to mobile devices/platforms no included in testing

4.4 Service virtualisation testing

Define service virtualisation testing

Describe the approaches to service virtualisation testing

Identify the main considerations in service virtualisation testing

1 What is service virtualisation and how does it benefit a project?

2 Identifying priority services to be virtualised

3 Knowing the limits of the virtualised service

4.5 Performance & volume testing

Define performance & volume testing

Describe the approaches to performance & volume testing

Identify the main

1 Identify the various aspects of performance and types of volume/loads that can be tested

2 Methods for determining the appropriate performance tests to conduct Previous performance behaviour Infrastructure pinch points Changes/forecasts of user behaviour

29 November 2018 Building Digital Capability Program – Learning Design Standard – Digital Testing 13

considerations in performance & volume testing

New system components3 Considerations in automating

performance/load tests4 Projecting live performance where test

environment doesn’t match live environment

5 Assisting in root cause analysis when performance issues are identified

4.6 Data warehouse testing

Define data warehouse testing

Describe the approaches to data warehouse testing

Identify the main considerations in data warehouse testing

Demonstrate application of data lineage tracing

1 What is data warehouse testing?2 Tracing data representations back to

their source (data lineage)3 Formulating expected results from

source data 4 Sample testing5 How does automation fit in?

4.7 Usability & accessibility testing

Define usability & accessibility testing

Describe the approaches to usability & accessibility testing

Identify the main considerations in usability & accessibility testing

1 What are usability and accessibility issues?

2 Identify usability and accessibility guidelines and legislation

3 How should testing be structured to address usability and accessibility?

4.8 Continuous delivery testing

Define continuous delivery testing

Describe the approaches to continuous delivery testing

Identify the main considerations in continuous delivery testing

1 What is a continuous delivery/continuous integration environment

2 The role of automated testing and entry/exit criteria

3 Working closely with developers and system integrators

4.9 User acceptance testing

Define user acceptance testing

1 Why and when should user acceptance testing be conducted?

29 November 2018 Building Digital Capability Program – Learning Design Standard – Digital Testing 14

Describe the approaches to user acceptance testing

Identify the main considerations in user acceptance testing

2 Structuring testing activities for nominated user testers

3 Monitoring and recording user testing4 Analysing and reporting test results

4.10 Testing environment configuration and validation

Describe testing considerations in a cloud vs virtualised vs hybrid environment

Apply data de-identification

Describe the testing environment validation process

1 The differences between physical (usually in-house), cloud, virtualised and hybrid testing environments

2 Identifying data requirements3 Creating and maintaining test data4 Data de-identification considerations &

techniques5 Validation of test data6 Validation of test environment as a

whole

29 November 2018 Building Digital Capability Program – Learning Design Standard – Digital Testing 15