8/17/2019 Mapping Tool User Manual 2012
1/19
Supporting Publications 2012:EN-XXX
© European Food Safety Authority, 2012
TECHNICAL REPORT
User Manual for Reporting Officers and Reportersfor Mapping Member State Standard Terminology to EFSA
Standard Terminology
European Food Safety Authority1
European Food Safety Authority (EFSA), Parma, Italy
ABSTRACTEuropean Food Safety Authority (EFSA) is charged with coordinating the reporting of
zoonoses, zoonotic agents, animal population, antimicrobial resistance and foodborne
outbreaks in the European Union under the Directive 2003/99/EC as well as analysing
and summarising the data collected. For data collection purposes EFSA has created a
sample Excel based mapping tool to allow Member States to manage the mapping of
their standard terminology to that published by EFSA. This technical report is a user
manual document that describes how to map terms and generate XML for submission of
2012 data, via the published BIOMO data models.
© European Food Safety Authority, 2013.
KEY WORDSZoonoses, XML, BIOMO Data Models ,XSD, DCF, Excel.
1 Correspondence: [email protected]
mailto:[email protected]:[email protected]:[email protected]:[email protected]
8/17/2019 Mapping Tool User Manual 2012
2/19
User Manual for the 2012 Zoonoses Mapping Tool
Supporting publications 2012:EN-XXX 2
SUMMARY
This user manual document provides guidance on use of the BIOMO mapping tool for the
submission of 2012 data by the Member States and other reporting countries.
The manual provides a general introduction to the mapping tool. The report covers allaspects related to the management and use of the mapping tool. In particular detailed
guidelines are provided for the mapping of standard terminology and the creating of a
locally validated XML file for submission of data to EFSA via the DCF (Data CollectionFramework)
Instructions are also given on the updating of EFSA standard terminology dictionaries,and how to incorporate these updates in the Mapping tool.
This manual is specifically aimed to guide the mapping of member specific standard
terminology to those published by EFSA, thus creating a much needed documented and
centralized mapping of controlled terminology.
8/17/2019 Mapping Tool User Manual 2012
3/19
User Manual for the 2012 Zoonoses Mapping Tool
Supporting publications 2012:EN-XXX 3
TABLE OF CONTENTS
ABSTRACT 1
Summary 2
1
Introducing the Mapping Tool 4
1.1
Getting Started by Profiling Your Data ...................................................................6
1.2 Mapping Your Profiled Terms ...................................................................................9 2
The coding of your data 11
2.1.1 The LookUpDicTerm Function............................................................11
3
Generating your XML 13
3.1 Copy, pasting and preparing the data for XML export....................................... 13 3.2 Activating the XML Developer tab ......................................................................... 15 3.3 Adding your data model XSD as an Excel XML Map ........................................... 16 3.4 Mapping the XML Map to the data in the Coded_XML worksheet .................... 17
3.5 Exporting our XML Map table to an XML File ....................................................... 18
8/17/2019 Mapping Tool User Manual 2012
4/19
User Manual for the 2012 Zoonoses Mapping Tool
Supporting publications 2012:EN-XXX 4
1 Introducing the Mapping Tool
The Mapping tools main goal is to provide a simple yet useable platform for member
states to map their country specific standard terminology to those published by EFSA,
producing an XML file for the submission of sample or aggregated based zoonoses
monitoring data via the DCF.
This User Manual introduces the basic functionalities available in the Mapping Tool, such
as:
Mapping terms explicitly
Providing lookup and encoding of explicitly mapped terms
Generating and validating XML via published EFSA data mode XSD
This manual will include some suggestions on how to proceed when you want to profile
your data, these are simply suggestions on our behalf and do not need to be followed to
the letter.
The mapping tool is essentially an Excel workbook which has three default worksheets,
which are colour coded. In the diagram below you will see these default worksheets, the
red coloured tab worksheet (number 1 in diagram) which contains the text
ZOO_FACT_(MODEL_NAME) is where you will put your own data, this is the first step,
whereby you must fit your current data into the structure of the data model in question.
This worksheet is the format in which Member States must organise their data for
successful mapping and submission of data to EFSA.
The next worksheet of importance is coloured green and is called CODED (number 2 in
diagram), this worksheet reads the data inserted into the red ZOO_FACT worksheet, and
then searches for the correct mapped term in the various pick list worksheets included in
the Excel Workbook (more on this later).
The last coloured worksheet (number 3 in diagram) is called DCF_FORMATTED_DATA
(XML in the old mapping tool), and this is where you will eventually paste your finisheddata, either to export as XML via the published BIOMO XSD schema or to leave as is in
Excel format, remember the DCF supports both Excel and XML format. It is very
important that the Excel workbook is saved in Excel 97-2003 Workbook (*.xls) format as
the DCF only supports Excel files of this type, also when submitted to the DCF if the
workbook contains more than one worksheet it is only the first worksheet in the
workbook that is read, this is the reason why DCF_FORMATTED_DATA is the f ist sheet in
the series.
8/17/2019 Mapping Tool User Manual 2012
5/19
User Manual for the 2012 Zoonoses Mapping Tool
Supporting publications 2012:EN-XXX 5
Figure 1
The CODED worksheet contains the LookUpDicTerm function that will query the
mapping you will create in each of the pick list worksheets. The function will return a
mapped EFSA code, it is important to make sure that the mappings you have created
are correct. In figure 2 below, highlighted by number two, there is a cell under the
ZOO_FACT_AMR_ISOLATE_AST data model element “zoonosis” selected, as you cansee in number three this cell contains a formula called LookUpDicTerm.
The LookUpDicTerm excepts as its first value the name of a pick list to search, in this
case “ZOO_CAT_PARAM_AMR ” and a value, the value will be selected from the red
worksheet ZOO_FACT_AMR_ISOLATE_AST, this value must have been mapped to a
pick list term in the ZOO_CAT_PARAM_AMR pick list, if not the function will return
NOT MAPPED, which mean the value you are trying to use has no valid mapping. All
NOT MAPPED values must be fixed before a valid submission can be made.
Number four in figure 2 is highlighting the fact that ZOO_FACT_(MODEL_NAME)
elements (like repCountry, lang, zoonosis .etc) that require a EFSA standard term(pick
list), are linked, you need to simply click on the link and you will be moved to the
correct pick list/catalogue in the workbook, where you can easily create your mappings.By default the CODED worksheet only has 200 pre-filled cells with formula’ s, if you
have more than 200 data entries (rows) in the red worksheet
ZOO_FACT_(MODEL_NAME), you will need to extend the number of rows with
formula’s to match. For example I have 560 rows in ZOO_FACT_(MODEL_NAME),
only the first 200 will be visible in the CODED worksheet, you will need to select row
number 200 in the CODED worksheet and drag it to the corresponding row number in
ZOO_FACT_(MODEL_NAME), which is 560, this way all values will be copied across.
Figure 2Each ZOO_FACT_(MODEL_NAME) data model contains an up to date list of BIOMO
standard terms; these are the pick lists that are also available on the online Zoonoses
Web reporting tool. Each model will contain only the pick lists that it needs to reference .
General Points on Excel worksheets
It is worth mentioning here that because each cell in the CODED worksheet contains a
formula, the more data you put in, the slower the use of the Excel worksheet will
become. I personally would recommend that you profile all your data, mapping all the
terms existent, and then create multiple copies of the excel workbook. You can then split
the data into lots of say five to ten thousand rows, thus keeping the excel worksheets
lighter and quicker to use, of course this is just a recommendation.
8/17/2019 Mapping Tool User Manual 2012
6/19
User Manual for the 2012 Zoonoses Mapping Tool
Supporting publications 2012:EN-XXX 6
1.1 Getting Started by Profiling Your Data
Figure 3
When you open any of the data models ZOO_FACT Files you must “ enable”
Marcros so that the custom function LookUpDicTerm can run. The figurebelow shows you the sequence of steps to follow in doing this.
8/17/2019 Mapping Tool User Manual 2012
7/19
User Manual for the 2012 Zoonoses Mapping Tool
Supporting publications 2012:EN-XXX 7
If you do not already have a list of all the terms that need to be mapped per data model
to EFSA terms, one simple way to profile your data is to use an Excel pivot table. To
profile quickly your data follow the following steps:
1.
Before inserting a Pivot table into a new worksheet, first select all data in the redZOO_FACT_(MODEL_NAME) worksheet by c licking the select all table feature
indicated in the figure below.
Figure 4
2.
Then select the Insert Tab from the control Ribbon, and choose Pivot table.
Figure 5
3. The table range is already set by virtue of the “select all table feature “we
performed in step 1, now we want to create a pivot table in a new sheet.
8/17/2019 Mapping Tool User Manual 2012
8/19
User Manual for the 2012 Zoonoses Mapping Tool
Supporting publications 2012:EN-XXX 8
Figure 6
4. We now have a pivot table that has all the data values that are in the
ZOO_FACT_(MODEL_NAME) red worksheet, so we can now start profiling the
elements like zoonosis and matrix which need to have there terms matched with
the BIOMO standard term pick lists ZOO_CAT_PARAM_ZOO and
ZOO_CAT_MATRIX. The below figure shows the pivot table that we created,
where we have selected Zoonosis, as you can see on the left of the figure, all the
unique terms in the Zoonosis column are displayed. These are the terms that we
need to insert into the ZOO_CAT_PARAM_ZOO pick list worksheet . We now
copy the unique terms under the zoonosis column on displayed on the left, and
paste them somewhere on in the profiling worksheet to the right of the pivot table
so that we can reference them later.
Figure 7We repeat the above step for all data model elements that need to be mapped to
the EFSA standard terms pick lists, copy and pasting the resultant unique terms
into columns to the right the pivot table.
8/17/2019 Mapping Tool User Manual 2012
9/19
User Manual for the 2012 Zoonoses Mapping Tool
Supporting publications 2012:EN-XXX 9
Figure 8
5. Eventually the list to the right of the pivot table will have all the unique terms that
need to be mapped like in the figure below.
Figure 9
1.2
Mapping Your Profiled TermsAt this point you have profiled and saved the l ists of unique terms that need to be
mapped from your data set to those of EFSA’s. Step five in “profiling your data” lists the
profiled terms that we will use in our example mapping. The following sequence of steps
will map a these values to their corresponding EFSA standard term.
The figure below highlights the sect ions where mapping is performed in the
ZOO_CAT_PARAM_AMR pick list; all pick lists have these mapping sections. A mapping
section is linked to a specific standard term pick list. Area one is where your Member
State specific terms are inserted and area two where the BIOMO standard term that
matches the Member State term is selected from the dropdown box highlighted in area
figure 10 below.
8/17/2019 Mapping Tool User Manual 2012
10/19
User Manual for the 2012 Zoonoses Mapping Tool
Supporting publications 2012:EN-XXX 10
Figure 10
The following steps describe the mapping process.
1. Open the CODED worksheet in the workbook. As we know from figure 2 on page
5, the Zoonosis element name is linked to the ZOO_CAT_PARAM standard term
pick list. By using the hyperlinked index in the element “zoonosis” we can move to
the ZOO_CAT_PARAM _AMR pick list.
Once we have selected the ZOO_CAT_PARAM _AMR standards terms pick list we
paste in our profiled terms under column A (MemberState_TERMS.
Figure 11
2. Once you have pasted your profiled terms into the A (MemberState_TERMS
column, it’s time to start looking for their matching term in the Zoonoses data
models standard terms pick list. In column B, EFSA_TERM, there is a dropdownlist of all the standard terms in the ZOO_CAT_PARAM _AMR pick list for the
element Zoonosis, you have to now search through this list until you find a
8/17/2019 Mapping Tool User Manual 2012
11/19
User Manual for the 2012 Zoonoses Mapping Tool
Supporting publications 2012:EN-XXX 11
standard term that matches your Member State specific standard term. When you
find a match, select it, and now your standard term is physically matched
to EFSA’ s term. The LookUpDicTerm function(figure 2, page 5) in the
CODED worksheet will now return for every occurrence of your standard term, in
this example “Salmonella - S. Abortusovis ” the code value“RF-00000372-
MCG”.
Figure 12
3. We repeat the above step until all your terms are matched for all you profiled
elements
2 The coding of your data
The CODED worksheet is an exact copy of the ZOO_FACT_(MODEL_NAME)
red worksheet, every cell in the CODED worksheet references its opposite cell
in the ZOO_FACT_(MODEL_NAME) red worksheet. The default CODED
worksheets for all the published BIOMO data models have two hundred pre-
filled rows containing all the correct formula references.
Only data model elements that need to be mapped to BIOMO standard term
pick list need to have the LookUpDicTerm funct ion reference, all the rest of
the elements have a simple formula that select s its cell counter parts value inthe ZOO_FACT_(MODEL_NAME) red worksheet and copies it. When an empty
string or value is found, the formula replaces this with a string NULL. The NULL
string values will be replaced by a true null value when we generate XML via an
XSD.
2.1.1
The LookUpDicTerm Function
The LookUpTerm Function is the glue between the CODED worksheet and the
EFSA standard terminology pick list worksheets. In figure 13 below, you can see
that the zoonosis element in the CODED worksheet has a LookUpDicTerm
function reference; this is because all zoonosis values must come from BIOMO’s
standard term pick list ZOO_CAT_PARAM.
8/17/2019 Mapping Tool User Manual 2012
12/19
User Manual for the 2012 Zoonoses Mapping Tool
Supporting publications 2012:EN-XXX 12
The interactions one through to four in figure 13 outline how the LookUpDicTerm
interacts with the pick lists linked with data model elements, in this case the zoonosis
element in the ZOO_FACT_AMR_ISOLATE_AST data model, to return a mapped term
code.
The LookUpDicTerm excepts as its first value the name of a pick list to search, in this
case “ZOO_CAT_PARAM_AMR ” and a value, step one in figure 13, the value will beselected from the red worksheet ZOO_FACT_AMR_ISOLATE_AST, step two in figure
13. If you look again at step one, you will see that under the element zoonosis “Not
Mapped” is displayed. This means that the value you are trying to use has no valid
mapping. The value we are looking for in ZOO_FACT_AMR_ISOLATE_AST can be seen
in step two, which is “Salmonella - S. Abortusovis” .
In step 3 you can see that in the pick list ZOO_CAT_PARAM_AMR we mapped the term
“Salmonella - S. Abortusovis” under column A (MemberState_TERMS) to column B
(EFSA_TERM) “RF-00000372-MCG | Salmonella - S. Abortusovis” .
The LookUpDicTerm function can now use the value found in
ZOO_FACT_AMR_ISOLATE_AST!E2 ( “Salmonella - S. Abortusovis” ) to search in
column A (MemberState_TERMS) for a text match for “Salmonella - S. Abortusovis” .When a match is found, and it will be because we just mapped it, the function will ret urn
the code “RF-00000372-MCG” from the “RF-00000372-MCG | Salmonella - S.
Abortusovis” ZOO_CAT_PARAM_AMR pick list.
In step four you can now see that under the zoonosis element the correct coded term is
now displayed for the member state term “Salmonella - S. Abortusovis” . All these step
need to be repeated for all data model elements that require a mapping in an EFSA
standard control terminology pick list.
Figure 13
8/17/2019 Mapping Tool User Manual 2012
13/19
User Manual for the 2012 Zoonoses Mapping Tool
Supporting publications 2012:EN-XXX 13
3 Generating your XML
With the release of Office Excel 2003, the importing and exporting of XML via a
valid XSD has been supported. Under the Developer tab, an XML tool exists forthe creating of what Excel calls XML Maps. When BIOMO publishes its data
models, it also publishes an XML schema or XSD, which supports the strict
structure and validation of any data model, and as such can be used to generate
from our CODED worksheet a valid XML file containing all our coded values
The following sequence of steps will produce and exportable validated data set
in XML.
3.1 Copy, pasting and preparing the data for XML/Excel export
Copy all the data from the CODED worksheet. The quick way to copy all cells
containing data is to use the hot key combination of Control + Shift +Space as
indicated in the figure below, or select the select all square in the top left had
corner of the worksheet, as indicated in figure 4 page 7.
Figure 14
1. In step two we want to paste all the values we copied from the CODED worksheet
in step one, into the Excel worksheet DCF_FORMATTED_DATA. The worksheet
CODED uses cell formulas to generate its cell contents, by referencing data in the
ZOO_FACT_AMR_ISOLATE_AST red worksheet. In the DCF_FORMATTED_DATA
worksheet we only want to paste the actual cell values and not the formulas contained in the CODED worksheet. To do this, first we select all the cells in the
active DCF_FORMATTED_DATA worksheet, and then under Paste we choose
“Paste Values” , you can also do the same under Paste Special, selecting the
values option in the dialog box.
8/17/2019 Mapping Tool User Manual 2012
14/19
User Manual for the 2012 Zoonoses Mapping Tool
Supporting publications 2012:EN-XXX 14
Figure 15
2. This next step will replace all occurrences of the string “NULL” in cells, with an
actual true value of null, this is to prevent errors in generating and validating XML
via the XSD. Empty string values in Excel cells cause problems for the XML parser
used by Excel, that is why we replace empty strings first with a string “NULL” and
later with a real null value. To replace the string “NULL” , choose the Find and
Select tab, then select Replace, when the Find and replace dialog appears type
NULL into the Find what: search box, leave the Replace with: text box empty
(indicates null to Excel). Finally select the Replace All button which will now
update all instances of NULL with a true null value.
Figure 16
If you do not want to submit XML validated data you can stop here at this point, you
must only insure that the DCF_FORMATTED_DATA worksheet is the first worksheet in
the workbook. It is very important that the Excel workbook is saved in Excel 97-2003
Workbook (*.xls) format as the DCF only supports Excel files of this type, also whensubmitted to the DCF if the workbook contains more than one worksheet, it is only the
8/17/2019 Mapping Tool User Manual 2012
15/19
User Manual for the 2012 Zoonoses Mapping Tool
Supporting publications 2012:EN-XXX 15
first worksheet in the workbook that is used, this is the reason why
DCF_FORMATTED_DAT A has the fist sheet in the series.
3.2 Activating the XML Developer tab
The Developer tab allows you to import and create XML files, normally it is notimmediately available as a default option and so needs to be activated direct ly by
you. The following figure indicates the sequence of selections you need to undergo
to activate the Developer tab.
Figure 17
8/17/2019 Mapping Tool User Manual 2012
16/19
User Manual for the 2012 Zoonoses Mapping Tool
Supporting publications 2012:EN-XXX 16
3.3 Adding your data model XSD as an Excel XML Map
To import the XSD that you want to use to validate the data model you must first go
to the Developer tab and select the Source icon. The Source icon will activate the
XML Source window, at the bottom of this window there is a XML Maps button
which you need to select. Once the XML Maps button is selected the XML Maps
dialog box will appear, you now need to select the Add button, which will activate a
File d ialog. In the file dialog you now navigate to the location of the data model XSD
that you wish to import as an Excel XML Map.
Figure 18
Once you have selected the XSD, the Multiple Roots dialog will appear, please
select dataset as indicated in the figure below and then select OK.
Figure 19
8/17/2019 Mapping Tool User Manual 2012
17/19
User Manual for the 2012 Zoonoses Mapping Tool
Supporting publications 2012:EN-XXX 17
3.4 Mapping the XML Map to the data in the Coded_XMLworksheet
1. After we have imported our data models XSD file in as an XML Map, you can now
see the structure of the XSDs elements in the XML Source window. The section ofthe XML Map that we are interested in is under the dataset branch. To map the
XML Map to your data you will need to left click on your mouse button on the
Result branch, with the mouse button still held down you will need to drag the
Result branch across to the Top Left most cell (in this case resultCode), containing
the first element column name in the Coded_XML worksheet. Let go now of the
mouse button when the Top Left cell becomes active. Excel will now create a
Table with our XML Map as its basis. We have now successfully mapped our XML
Map to the elements in our data Model.
Figure 20
8/17/2019 Mapping Tool User Manual 2012
18/19
User Manual for the 2012 Zoonoses Mapping Tool
Supporting publications 2012:EN-XXX 18
2. Setting the Validate against Schema for Export Feature. To activate this feature
we need to first select the Developer tab followed by the Map Properties
button. The XML Map Properties dialog box will now be displayed, under XML
Schema va lidation, select the Validated data against Schema option followed
by the Ok button. Now when we export the data, the XML file c reated will
automatically be validated by the data model XSD that we imported.
Figure 21
3.5 Exporting our XML Map table to an XML File
To Export our data now we first select the Developer tab followed by the Export
button. The Export button will bring up the Export XML File dialog box. Here you
can name and navigate to where you want to store your XML file. After naming your
file simply select the export button to initiate the export procedure.
Figure 22
8/17/2019 Mapping Tool User Manual 2012
19/19
User Manual for the 2012 Zoonoses Mapping Tool
Supporting publications 2012:EN XXX 19