1

TG MTSModel Test Suite v2.0

Wim Devos, Pavel Milenov, Slavko

Lemajic, Paolo Isoardi

Control and management of agricultural land in IACS workshop

Baveno, 23-25 May 2016

1

Outline

1.Origins: from ATS over Model Based Testing to MTSv2.0

2.Parts: the scope and content

3.Operation: how do the tests work?

4.Delivery

2

2

ORIGINSPart1

3

LPIS Workshop : Applications and Quality, 6-8 th October, 2009, Tallinn 4

Objective

Model Conformance TestConformance Statement (ICS)

Abstract Test Suite (ATS)

Data Conformance TestExecutable Test Suite

(ETS)Additional Information for Testing

Conformance Test Report

Analysis of results

Application Schema or

Feature Catalogue of the implementation

under test

ISO 19105 Conformance and testing

conformancefulfilment of specified requirements

conforming implementationimplementation which satisfies the requirement

3

LPIS Workshop : Applications and Quality, 6-8 th October, 2009, Tallinn 5

ATS Structure

Abstract test suite structureModules and tests: e.g.

Module A.1.1 can be assigned ‘Conformant’ value if one of the tests A.1.1.1.1 OR A.1.1.1.2 OR A.1.1.1.3 OR A.1.1.2.1 is

‘Conformant’

Test purpose: verify definition of reference parcel: reference parcel demarcated by the farmer, who

cultivates it (manage/execute his tenure rights: ownership, rent etc.) on multi-annual basis

Test method: Inspect the documentation of the application schema or feature catalogue, verify

consistency with ICS

ATS NOTE: Conformant with Farmer’s block definition

Test reference: LCM specification

Test type: Basic test

A.1.1Basic test

A.1.1.1 A.1.1.2

A.1.1.1.1 A.1.1.1.2 A.1.1.1.3 A.1.1.2.1

Baveno 2015

ABSTRACT & EXECUTABLE TEST CASES act Prepare MTS executable test cases within ETS su ite

MS Testing labaratory

Integrated

administration and

control system :

Perform MTS

Prepare schema definition tests

Prepare extended code list tests

Prepare: encoding tests

Prepare feature type completeness tests

Prepare: attribute / association

completeness tests

Prepare: multiplicity tests

Prepare: value type tests

Prepare: constraint tests

Startpreparingexecutabletest cases

Finishpreparingexecutabletest cases

«FeatureType»Base types::ExecutableTestCase

«FeatureType»Base types::AbstractTestCase

Retrieve all abstract test cases

Retrieve: eligibility profile

documentation

«FeatureType»Quality assessment framework::

MtsItemUnderInspection

Prepare schema mapping test

«trace»

«trace»

«trace»

«trace»

«trace» «trace»

«trace»

«trace»

«trace»

«trace»

BASIS GML...

6

4

Baveno 2015

MODEL BASED TESTING MESSAGES

MTS outputs:LOG of conformance assessment

FAIL: The source element (x) do not have corresponding target element.

FAIL: The type of target element (x) do not match the type of source element (x).

FAIL: The target element (x) do not have documentation.

FAIL: The multiplicity of target element (x) do not match the multiplicity of source element (x).

....

TESTING REPORT of conformance testing.... Overall testing report:

Application schema: (FAIL/PASS)Leaf: (FAIL/PASS)Feature type: (FAIL/PASS)Type: (FAIL/PASS)Data type: (FAIL/PASS)CodeList: (FAIL/PASS)Union: (FAIL/PASS)Enumeration: (FAIL/PASS)Property (classes): (FAIL/PASS)Relationships: (FAIL/PASS)Generalization: (FAIL/PASS)Assosciation: (FAIL/PASS)Aggregation: (FAIL/PASS)Composition: (FAIL/PASS)Property (Relationships): (FAIL/PASS)

7

Baveno 2015 WS and later

1. during the WS:� lot’s of feasibility concerns expressed by MS

� three volunteers: DE, HU and ES

2. meeting in ISPRA (April 2015):� Result of tests by the volunteers

� Main issues:

• Eligibility profile: catalogue of LC types > set of all system choices,

• MTS scope too broad: MMU, EFA

• LCM bases tests might require database redesign or SW recoding

• Implementation: software, user interface, cost-efficiency,

multilingual support

3. formal request to downsize UML on TG ETS (June 2015)4. ….and then everybody became busy (Autumn 2015+).

8

5

PARTSPart2

9

Information on implementation choices: TG IXITimplementation extra information for testing

Why TG IXIT?

The ETS inspection procedure was conceptually quite simple:1. Detect critical defects

2. Map agricultural land of the RP that is physically visible

3. Compare observations with SUT records

4. Perform pass/fail test for 5 quality expectations

But over time, upon request of the MS, a number of design dependencies were introduced in the procedure for:

• CD application• waiver use• contamination

The module A111 was no longer sufficient� structured approach needed

10

6

Data model test

Why data model tests?Support correct mapping of SUT to the TG ETS documentation1. the scoping query of the RP population

� “reference parcel”

� “non-zero MEA”

� “declared last year for direct payment”

2. The comparison of ETS observations� “recorded referenceArea”

3. The correct reporting and data exchange� lpisPointZerostate.gml

� lpisPolygonZerostate.gml

� updateEvidence.xml

If not correctly done: � comparing apples to pears

11

MetadataWhy metadata?

� Not new, first initiative in 2009 Sofia WS

(start LPIS QA portal)

� Annual questionnaire of DGAgri

� ECoA questionnaire

What is asked?� Not the INSPIRE metadata record for

cataloguing, but selected elements thereof.

� What is needed for understanding SUT

during screening

� “LPIS” + supporting sources (ortho/DEM)

This is the more dynamic part of TG MTS � standalone report: metadata.xsd

12

7

OPERATIONPart3

13

Performing the IXIT

What are the key Regulation choices (features) of an LPIS design?� A Who authors the primary RP boundary, the perimeter of the

identified unit of land? � defines the reference parcel type

� B What process delineates the physical borders of the agricultural

land (generate the MEA/referenceArea)?

� C How are eligible landscape feature (under GAEC) adjudicated?

� D How are RPs assembled (optional further processing choices)?

� E Is there pro rata reduction on PG with scattered ineligible features?

� F Is there (Y/N) validation evidence of the required positional

accuracy requirements for each individual reference parcel perimeter?

by 6 poly(cho)tomy questionnaires A to F

� Each SUT is described by 6 adjectives

14

8

Data model tests 1

Based on model testing methodology with 3 abstract test cases

1.featureTypeCompletness : Is this spatial object represented?

2.attributeTypeCompleteness: Does it have this information element?

3.codeListEnumerationCompleteness: Does it respect conventional naming?

Is the list complete?

“this spatial object”, ”this information element”, “naming” are abstract parameters. To obtain executable tests they need to appropriately rephrased in terms of both

� ETS data element (e.g. referenceParcel, referenceArea, CP)

� SUT (is “it” a feature type, an attribute, a method (combination, query)?)

The description of the 41 ETS data elements is stored in � UML (online)

� An xls log file15

Data model tests 2

To generate the SUT contribution to generate your 41 executable

tests, revise your 2010 ATS archive package, in terms of either� The SUT feature catalogue

� The SUT application schema

DG JRC already proposes 41 “starter” executable tests.

But then comes the tricky part:� Each starter’s may work “as is” for your SUT

� But “it” may not work for your SUT, in which case you will need to

rephrase that executable test to obtain the correct result

If not:

� you could test pears disguised as apples > papple!

� your ETS results become unreliable, i.e. non-conforming!

16

9

Data model tests: an exampleLCM refers to1. an area value2. in hectares3. as maximum for DP-payment≡ input for IACS process

LPIS 2012 answer describes 1. an area value √2. in hectares √3. polygon of eligible land Х≡ output of a mapping process

This is a partial match� Potentially incorrect as it

misses the reference to DP application/payment

Example (LCM)

referencedAbstractTestCaseId 14

executableTestCaseId 1412

executableTestName checkReferenceArea

referencedAbstractTestCaseName attributeAndAssociationCompleteness

executableTestResultDescription

Feature class "referenceParcel" within

LCM has attribute field "ReferenceArea".

Value in this field corresponds to the

maximum area, for which an application

can be made and it is reported in

hectares

Example (LPIS 2012)

referencedAbstractTestCaseId 14

executableTestCaseId 1412

executableTestName checkReferenceArea

referencedAbstractTestCaseName attributeAndAssociationCompleteness

executableTestResultDescription

Feature class "ELG_AREAS" within SUT

has attribute field "GR_AREA". Value in

this field corresponds to the total

geometric area of all polygons

representing eligible lands (GAC

compliant) for a given reference parcel

and it is reported in hectares

consequence of incomplete matchfree adaptation from various real experiences

1. How to scope (non-zero MEA)?� No problem if there are no other mechanisms between this eligible area

and pre-printed form/crosscheck/payment

� But what if “eligible area” includes to 2nd pillar conditions?

� JRC: “wrong scoping”

� MS: “no problem, because not available for payment”

� JRC: sample size not respected (650 + 150 ≠ 800 )

2. How to assess observed area conformity?� No problem if “eligible area” relates to all 1st pillar conditions

� But in this case:

• MS: “Pre-printed form populated with last year application area”

• MS: “crosscheck done with intersection of AP and RP”

� MS: “no problem because no impact on payment”

� JRC: LPIS is not performing its basic roles at all! 18

10

Metadata collection setup

This is a data model test, but reference descriptions located in:• Not LCM• Inspire Implementing Rules on metadata• Inspire Common Data Specification for orthoimagery (CP, DEM)• Ad hoc: ixitQualifier, sensorPlatform, demPlatform

Why different reference models and templates?The LPIS (perimeter1 and/or border2) vector data

supported by orho-imagery3

supported by digital elevation model4 (DEM/DSM)

Expectation: 2-4 or more metadata records in 2 data formats:� Vector

� Raster/grid

19

Metadata content

For each named dataset (from IR MD):temporalExtent/phenomenonTime: to which time period it refers to?

resourceLocator: where does it reside?

identifier: what’s its content/role?

lineage: what is the production line?

spatialResolution: what scale or GSD?

responsibleParty: who’s behind?

metadataPointOfContact: who to contact?

The regulations imply at least two metadata records1.LPIS: the vector dataset of polygons that hold MEA > 0

2.orthoimagery: the background(s) used during the assessment year

But applicable third party geospatial data (cadastre / land cover / dem) also need to be reported.

20

11

MTS dynamics

IXIT, data model and LPIS vector metadata relate to a single SUT (≡lot) that is subject the TG ETS

� TO DO for 2016� Until upgrade: only temporalExtent will be dynamic through update

Aerial imagery is often subject to a 3 year update cycle � one record by separate specification/lineage/contractor� annual deliveries (and geographic extent) are not requested

Supporting and Source data are independent� Monitor the update of these external sources

21

2016 DELIVERYPart4

22

12

Typical MTS data volumes

TG IXIT: 6 values (outcomes)

Data model test 41 values (local references/descriptions)41 x 4 codes (internal date/pass)

Metadata 10 values/codes x 3 (vector/ortho/source)

GRAND TOTAL 6 + 41 + 30 ≈ 80 values

23

MTS package and update deliveryDeadlines:

MTS package is synchronous with ETS reporting package.

Data exchange:

For 2016 (@31 Dec 2017) 1st complete delivery: 2 options• Either 3 xsd’s package upload (benefit: immediate validation by portal)

• Or 1 xls followed by manual entry in LPIS QA portal (disadvantage: 2 edits)

After upgrade of SUT (@31 Dec 2018,2019,2020): package re-upload

After update of dynamic data values (@31 Dec 2018,2019,2020) for� temporalExtent of SUT� change of image specification/lineage/contractor� change of third party metadataUpdate of affected values via LPIS QA portal application

24

13

Questions?

25