View
217
Download
2
Category
Tags:
Preview:
Citation preview
Testing ofXBRL files for the supervisory reporting
Eurofiling Workshop in Rome
Katrin Heinze
5 May2014
Rubric
www.ecb.europa.eu ©
The objective of this presentation is
to provide an overview on the testing strategy of the SUBA project
to explain the test plan and sample test cases
to inform about the next steps
Page 2Eurofiling Workshop in Rome, 5th of May 2014
Objectives
Rubric
www.ecb.europa.eu ©
1
2
3
Sample XBRL test cases
Next steps
SUBA* test strategy
Overview
Page 3Eurofiling Workshop in Rome, 5th of May 2014
* SUpervisory BAnking Data System
Agenda
Rubric
www.ecb.europa.eu ©
1
2
3
Sample XBRL test cases
Next steps
SUBA Test Strategy
Overview
Page 4
Agenda
Eurofiling Workshop in Rome, 5th of May 2014
Rubric
www.ecb.europa.eu ©
175
28
33
114
Total No run Failed Passed
Functional Tests Non-Functional Tests Regression Tests
Based on User requirements Use cases Experience of users Technical requirements Source code (unit tests) …
Categorised following Load tests (number of users) Performance tests (processing time) Volume test (quantity of data) Stress test Test of the (data) security Usability tests …
Done after changes on Components
o XBRL taxonomyo Validation rules
Interfaceso EXDIo RIAD
Infrastructure …
Technical Test Execution Status17.04.2014
SUBA Test Strategy
Eurofiling Workshop in Rome, 5th of May 2014 Page 5
Rubric
www.ecb.europa.eu ©
HP ALM usage within the SUBA project
Scope
Planning of releases and test cycles Enable traceability between requirements, tests, defects and
according code changes Increase visibility with dashboards on analysis and KPIs
Achievements
Definition of test cases linked to the defined requirements Execution of test cases based on defined test data Defect tracking and documentation of bug fixes
Way forward
Increase the level of automation on test execution Customise reports and analysis based on the project needs Preparation of test cases for the next release
SUBA Test Strategy
Eurofiling Workshop in Rome, 5th of May 2014 Page 6
Rubric
www.ecb.europa.eu ©
1. level: Categories
2. level: Use cases
3. level: Test cases
Conventio
ns on numberin
g
Structure in the Test Lab
Test plan structure
Eurofiling Workshop in Rome, 5th of May 2014 Page 7
Rubric
www.ecb.europa.eu ©
Tracking of the test progress in HP ALM
Eurofiling Workshop in Rome, 5th of May 2014 Page 8
Rubric
www.ecb.europa.eu ©
1
2
3
Sample XBRL test cases
Next steps
SUBA Test Strategy
Overview
Page 9
Agenda
Eurofiling Workshop in Rome, 5th of May 2014
Rubric
www.ecb.europa.eu ©
Positive functional test cases
Creation of test data for each module of the COREP taxonomy which is a valid XBRL instance document.
Valid according to
XML specification XBRL 2.1 specification XBRL Dimension 1.0 specification XBRL Formula 1.0 specification EBA disabled validation rules EBA Filing Rules for COREP and FINREP
By using different file sizes
small medium large
Sample XBRL test cases
Eurofiling Workshop in Rome, 5th of May 2014 Page 10
Rubric
www.ecb.europa.eu ©
Negative functional test cases
Tests that ensure that the application shows a proper behaviour when invalid files are received.
Creation of test data where the system is expected to provide correct error messages.
Possible test cases : A reporting entity sends a file which is not a well-formed XML document. has not the correct format, i.e. jpg, txt, xls etc. has not the expected encoding. includes neither facts nor contexts. misses a namespace in the xbrli root element includes letters where numbers are expected as fact value misses an explicit member in the scenario element Includes a fact with no value but also no nil attribute is provided Refers to a non-existing module Refers to two different modules.
Sample XBRL test cases
Eurofiling Workshop in Rome, 5th of May 2014 Page 11
Rubric
www.ecb.europa.eu ©
1
2
3
Sample XBRL test cases
Next steps
SUBA Test Strategy
Overview
Page 12
Agenda
Eurofiling Workshop in Rome, 5th of May 2014
Rubric
www.ecb.europa.eu ©
Test scope
Test period: 13th to 15th of May
Test activities:• Acquisition
NCA sends data to EXDI via WebUI or standard adapter. EXDI system sends a delivery message. SUBA system sends an acknowledgement message to NCA with the XBRL
validation result NCA receives the messages in the scheduled time periods.
Eurofiling Workshop in Rome, 5th of May 2014 Page 13
Next steps
Voluntary functional tests with NCAs
Rubric
Load tests
IT infrastructure for establishing a supervisory function at the ECB
14
Next steps
First estimation about the load expected for the first remittance date
Number of reported entities 128
First remittance date NCAs to EBA/ECB 31.07.2014
Modules reported Number of files (per institute)
Max. file size in MB
Max. number of observations
COREP consolidated
Own Funds 1 115 177.536Large Exposures 1 264 355.930NSFR 1 9 7.510
Total 512 49.664 69.244.928
*) Estimations based on information of the Deutsche Bundesbank for one big German bank
Rubric
Organisation and approach – Test case preparation
Specification workshops
Specification sign-off Implementation
• Development of software• Definition of test cases (HP ALM)
Tests execution
• Tests runs (HP ALM)• Defects workflow (HP ALM)• Bug fixing
Control point
Metric Target (by priority)
Average High Medium Low
A % reviewed requirements 100% 100% 100% 100%
B % requirements test coverage 75% 100% 100% 50%
C % tests run 90% 100% 100% 75%
% tests succeeded 100% 100% 100% 100%
Defect resolution time (working days) 3 3 5
Defects reopened ratio 5% 5% 10%
Control point A Control point B Control point C
Eurofiling Workshop in Rome, 5th of May 2014 Page 15
Next steps
Rubric
www.ecb.europa.eu ©
Key issues and challenges
Load tests Defining scenarios for load tests on XBRL processing
via automated test, simulating also peaks
Preparation for next releases
Defining test cases based on more precise user requirements
EBA testing Participating in tests scheduled by EBA
Functional tests Supporting the voluntary functional tests with
the NCAs, organized by WG SIM
Eurofiling Workshop in Rome, 5th of May 2014 Page 16
Next steps
Recommended