TED CSV open datadata.europa.eu/euodp/repository/ec/dg-grow/mapps/TED_advanced_… · This is a background document for the TED CSV open data and is intended for advanced users who

European Commission, DG GROW G4, 2020-06-04

1

TED CSV open data

Advanced notes on methodology Version 0.91

Contents 1. Introduction ................................................................................................................................. 2

Country indicators ............................................................................................................... 2 1.1.

Acronyms ............................................................................................................................. 2 1.2.

2. General methodological issues ................................................................................................... 4

Value of notices vs. notice count......................................................................................... 4 2.1.

Calls for competition vs. contract award notices ................................................................ 5 2.2.

Mixing of above- and below- threshold procurement ........................................................ 6 2.3.

Excluding cancelled notices ................................................................................................. 7 2.4.

Including CANs informing about a non-award .................................................................... 8 2.5.

Date of publication of a notice vs. date of award of a contract .......................................... 9 2.6.

Notices under different directives ....................................................................................... 9 2.7.

Notices under different legal regimes within a directive .................................................... 9 2.8.

Different types of calls for competition ............................................................................ 10 2.9.

Notices from EU and other international institutions ................................................... 10 2.10.

Fields moving from notice level to lot level .................................................................. 10 2.11.

Notices with buyers coming from multiple countries ................................................... 11 2.12.

Definitions of fields ........................................................................................................ 11 2.13.

3. Methodological issues for individual indicators ........................................................................ 12

One bidder ......................................................................................................................... 12 3.1.

No calls for bids ................................................................................................................. 12 3.2.

Publication rate ................................................................................................................. 13 3.3.

Cooperative procurement ................................................................................................. 14 3.4.

Award criteria, lowest price .............................................................................................. 14 3.5.

Decision speed ................................................................................................................... 14 3.6.

Missing values ................................................................................................................... 15 3.7.

Missing registration numbers ............................................................................................ 15 3.8.

Missing calls for competition............................................................................................. 15 3.9.


2

1. Introduction This is a background document for the TED CSV open data and is intended for advanced users who know the basic documentation and have significant experience with the data. This document has three parts: an introduction, a discussion of methodological issues concerning more than one indicator and a discussion of methodological issues for the individual indicators of the Single Market Scoreboard. The main purpose of this document is to use the running example of country-level indicators (in particular those of the Single Market Scoreboard) to explain advanced issues of working with TED data, methodological decisions on how to resolve them, and technical solutions planned by the Commission for the future. While illustrated on country-level indicators, these issues may be relevant for all sorts of TED data analyses. By going through the issues (potentially using this document as a checklist), a researcher can explain unintuitive results as well seeming noise in the data. Finally, this document also tracks the evolution of issues across recent changes in the TED standard forms. The document covers a wide range of methodological issues. The impact they have on results depends the analysis in question – and in particular its level of detail. In our experience, for most issues, the impact on results at the level of countries and years is small. We also expect that it would be small in more detailed levels such as regions. Nevertheless, empirical examination of robustness of results to the issues below is a welcome venue for future research. Moreover, these issues also serve as a reminder that studies needs good robustness checks as well as careful interpretation. We note that for many, if not most, issues below, there may not be a solution which can be empirically proven to be the best. This can be the case for example because the choice is between two theoretical biases of unmeasurable magnitude.

Country indicators 1.1. The country indicators, used as a running example in the text below, have two to three interlinked goals. Which goal is being measured influences which notices should be included in an indicator.

1. Measuring procurement practices of public buyers. This means describing how procurement is done, how are buyers trying to get the best value for money.

2. Measuring the results of their efforts. Results of public procurement are "value-for-money" of what is being bought and then other policy goals such as the impact on environment. Since these are hard to measure directly, proxies are commonly used instead – for example, if buyers managed to attract bidders and what prices they paid.

3. (A third, tentative and hopefully temporary function of the indicators is discovering where the data quality is low, i.e. where EU Member States (MS) are not fulfilling their publishing obligations, which undermines the measurement of the two goals above.)

Country-level indicators of course give only a very high-level view of procurement in the EU. For more actionable results, analysis must be done by comparing narrower categories. These segments include types of authorities (e.g. "Regional or local authorities" vs. "Ministry or any other national authorities"); regions (e.g. Wallonia vs. Flanders); and sectors, defined by CPV code or main activity (e.g. health vs. education; hospitals vs. schools).

https://data.europa.eu/euodp/en/data/dataset/ted-csv

http://ec.europa.eu/internal_market/scoreboard/performance_per_policy_area/public_procurement/index_en.htm



3

Acronyms 1.2.

Current forms ("2.0.9") are the standard forms based on the 2014 public procurement directives, specified in the 2015/1986 implementing regulation. These are the forms in the right column of the table at http://simap.ted.europa.eu/en/web/simap/standard-forms-for-public-procurement.

Older forms ("pre-2.0.9"), most importantly the "2.0.8" forms, are forms based on the 2004 and 2009 public procurement directives. "2.0.8" forms continue to be used for the defense directive. These are the forms in the left column of the table at http://simap.ted.europa.eu/en/web/simap/standard-forms-for-public-procurement.

CFC – a call for competition. This means a contract notice, a prior information notice used as a call for competition, or a qualification system with a call for competition.

CAN – contract award notice

VEATs – voluntary ex-ante transparency notice

FA – framework agreement

DPS – dynamic purchasing system

MS – EU Member States

http://simap.ted.europa.eu/en/web/simap/standard-forms-for-public-procurement




4

2. General methodological issues This chapter explains methodological issues related to more than one indicator.

Value of notices vs. notice count 2.1. Issue All indicators can be calculated in two versions: based on their value and based on their count1. The results of the two versions are different, because public procurement values are known to have a power distribution, i.e. the majority of value comes from only a minority of notices2. This means a few high-value notices can change results dramatically. In general, value-based indicators are probably better for most political messages as well as understanding economic (e.g. budgetary, market-shaping) impacts of procurement. On the other hand, count-based indicators give more relevant information on how widespread a particular procurement practice is. While professionally done procurement is not a goal in itself, knowing that bad practices are widespread, even if the economic impact is small, can be relevant by itself. For example, it can mean that the many small municipalities in a country are failing to navigate the procurement rules and fail in buying the small things their citizens need. Concerning data quality, values in notices have many issues: some MS have a large proportion of values missing (e.g. Sweden), while some notices include nonsensical values (e.g. 123456789). Furthermore, a particular problem concerns framework agreements, with some countries publishing on TED both the maximum value of an awarded framework agreement and the individual contracts awarded within framework agreements (e.g. Czech Republic, Slovakia). The current TED forms do not allow distinguishing between these two types of CANs. This means that both the value and the count of CANs can be inflated (see 2.2.3). Decision Because of low quality of value data, our country indicators are based on the number of notices, not their value3. We try to minimize the impact of multiple CAN publication by removing framework agreement and dynamic purchasing system CANs for some indicators as well as using CFCs rather than CANs (see 2.2). Future In the 2.0.9 standard forms, notices must contain information on value, otherwise they are not published. Furthermore, the Commission is working with the MS on increasing the quality of their data by giving feedback on the information they are providing, as well as trying to improve the standard forms. This should enable the use of value as a basis for most indicators in the future. The issue of two types of framework agreement data should be resolved in a future version of the standard forms.

1 Count refers to the “number of notices”, their “frequency”. We use the word “count” to avoid confusion with

the notice publication number. 2 The 80/20 Pareto principle can possibly be used as a heuristic.

3 Value is used only for the calculation of total procurement value per country and year in the annual indicators

report. For this statistic, the results are not split according to a third variable (e.g. type of procedure) and so inaccuracies in individual notices have a smaller overall impact.


5

Calls for competition vs. contract award notices 2.2. Many indicators can be calculated at the level of CFCs as well as CANs (because some information, e.g. whether cooperative procurement was involved, is contained in both). While the results are likely to be similar, they will not be the same. Furthermore, it can be argued that the interpretation is different: while CFCs give information on business opportunities, CANs on procurement which actually took place4.

Value availability according to the law 2.2.1.Issue CFCs contain only the estimated value of a procurement, and submitting this value is not mandatory. Practices in the MS differ strongly. While in some countries, e.g. Czech Republic, submitting the estimated value is mandatory, in other countries, e.g. Poland, it is strongly discouraged or even forbidden. CAN, on the other hand, must5 contain either the value of the awarded contract or the lowest and highest bids.

Non-publication of CANs 2.2.2.Issue Publication of all notices is legally mandatory. However, while the non-publication of a CFC (or a voluntary ex-ante transparency notice) is legally considered as a serious flaw in a procurement procedure and may lead to annulment of a contract by a review body, the non-publication of a CAN does not pose the same threat. Based on preliminary comparisons between national databases and TED, as well as indirect evidence from comparing the number of voluntary ex-ante transparency notices and contract award notices, it seems that in some MS, buyers are not publishing the CANs they are legally obliged to. Future We are in contact with the MS where non-publication seems to be an issue. More concrete information should be available from the www.digiwhist.eu project, which will allow better comparisons between national and EU databases.

Framework agreements and dynamic purchasing systems used 2.2.3.together with count-indicators

Issue For each framework agreement (FA) or a dynamic purchasing system (DPS) announced in a CFC, there may be multiple CANs. For DPS, this will almost always be the case, while for FA, this will be the case if national law prescribes publication of contracts based on framework agreements in TED (this is the case e.g. for Czech Republic and Slovenia).

If count-based CAN indicators (see subsection 2.1) are used, these multiple CANs (possibly resulting from just a few FAs or DPSs) can influence the results in a way that is not a valid measure of “procurement practices” nor “procurement results”. This can happen in two ways: either there are many peculiar CANs (e.g. in Portugal a single DPS influences the country results on the usage of accelerated restricted procedures); or, on the contrary, peculiar patterns which are usually not combined nor with FA nor DPS (e.g. direct awards) can be drowned out by the many CANs coming out of the FAs and DPSs. Furthermore, since the publication of contracts in FA is obligatory only in some countries, this can also introduce bias for cross-country comparisons.

4 For countries which have good quality data on links between notices (see section 3.9), this difference in

interpretation can be limited by taking in account only those CFCs which resulted in a CAN. 5 De-jure in all version of forms, de-facto since 2.0.9.

http://www.digiwhist.eu/


6

Decision For a couple of CAN-based indicators whose impact would be particularly heavy (e.g. One bidder contracts, see 3.1) we exclude FAs. However, for most notices, we leave them in, because in some countries a large portion of public procurement is done through FAs and excluding them would mean ignoring an important part of the country’s procurement practices. Future This will be a smaller problem once value-based indicators are used, because the weight of contracts within framework agreements will be smaller. The problem can be fully resolved by adding an option to clearly mark contracts in framework agreements to a future version of the standard forms. Decision If an indicator can be based on CANs as well as CFCs, the default option is CFCs. This is because for now we are not using the values included in the CANs anyway, CFCs are likely to be missing less often, and there are no multiple notices per one FA and DPS. Future We are working with MS on improving the reliability of value data as well as ensuring publication of all CANs. When the situation improves, some indicators are likely to be calculated on the basis of CANs, so as to benefit from the higher relevance of value-based indicators.

Mixing of above- and below- threshold procurement 2.3.Issue Above-threshold contracts6 in TED are mixed with some below-threshold ones (MS have a right to advertise them on TED, which we consider a good practice) and these two cannot be easily distinguished (see example of an unused method for distinguishing them in Box 1). Assuming that procurement practices and results can be on average different for below- and above- threshold procurement, and that these differences are different country by country, this could lead to biased results. Decision All notices are included for the indicators, without trying to estimate which ones are below-threshold and exclude them. The main reason is that it is not clear than any of the available estimation approaches removes more bias than it adds. (Neither of these biases is likely to have noticeable impact on the results, however, as even if, for example, the method below was implemented, the number of notices which would have been excluded is only around 2% of the total number.) Box 1: Unused approach for removing below-threshold notices

Threshold rules are too complex to be followed precisely just on the basis of TED data (for example, the legal distinction between central and non-central buyers, based on the annexes of the directives, does not correspond to the "type of contracting authority" option in the standard forms). Furthermore, even these complex rules have “exceptions”, such as a threshold increased by 20% in case of small lots (see 2014/24/EU Art. 5(10)). Thus, some kind of simplified approach needs to be taken.

6 More precisely, "above-threshold contracts" stand for all contracts falling within the procurement directives,

as compared to those which do not. Besides thresholds, this may also be a question of exceptions to the directives. As shorthand, we refer only to the most important factor – thresholds.


7

We considered an extremely simplified estimation: we would differentiate thresholds only by works/supplies/services, because this information is never missing and is reliable7; and, to ensure that we don’t exclude a valid procedure by accident, we use the minimal threshold8 existing in the rules for this type of purchase. Afterwards, CFCs which are below this value, or CANs whose CFCs were below this value9, would be excluded from the calculation. Minimal thresholds per year and per works / supplies & services

1.1.2006-31.1.2007

1.1.2008-31.1.2009

1.1.2010-31.12.2011

1.1.2012-31.12.2013

1.1.2014-31.12.2015

1.1.2016-31.12.2017

Works EUR 5 278 000

EUR 5 150 000

EUR 4 845 000

EUR 5 000 000

EUR 5 186 000

EUR 5 225 000

Supplies &

Services

EUR 137 000

EUR 133 000

EUR 125 000

EUR 130 000

EUR 134 000

EUR 135 000

However, even this approach runs into problems. Most importantly, a frequent error seems to be that values of notices are missing an order of magnitude, e.g. the notice contains "3" instead of "3 mil". As a result, these notices get excluded incorrectly. Furthermore, it is not clear what to do with missing values. Since dropping up to 80% of notices (in case of Sweden, or, for CFCs, of France) is not an option, notices with missing values need to be included by default. However, this in effect means that in countries which provide values some notices will be excluded, while in countries which do not provide values all notices will stay. Similarly, if we wanted to exclude CANs we would need to have a link with the CFC, but the frequency of how often these links are missing differs country by country – from 0% to 73%. This would again add bias. (Finally, notices just above the threshold may have the wrong price estimated in Euro, because we use the average exchange rate for the given year, which may push them just across the threshold)

Future This will be solved with future standard forms, which will contain information on whether the procurement is covered by an EU regulation or national law.

Excluding cancelled notices 2.4.Issue It is not always clear which notices were cancelled. Before 2.0.9, a cancellation could have been published using a standard form 14, but this was not a legal obligation and seems to be used

7 In the annual indicators report, we use only the overall minimal threshold. This is a similar approach, but even

more simplified and thus even less accurate. 8 Note that choosing the "minimal threshold existing in the rules" is, perhaps counterintuitively, very likely

better than choosing the "most commonly applicable" threshold (i.e. the one applicable for sub-central authorities, e.g. around 209 thousand for goods and services instead of around 135 thousand). The reason is that while some above-threshold procurement procedures would definitely get excluded if the usually applicable threshold was used (i.e. those by central authorities purchasing between 135 and 209 thousand), the number of the newly excluded below-threshold notices (e.g. those by sub-central authorities between 135 and 209 thousand) is unknown, and may be very low, because their number depends on voluntary publication. 9 The estimated value of the whole procedure is what decides whether a procedure is above- or below-

threshold, not the award value.


8

differently in different Member States. For example, since 2011, Latvia has sent 322 cancellations of CANs, while Germany and Poland around 150 each, and Spain and Italy around 90 each. This is, first, much smaller than would be expected, and, second, does not correspond to the size of the countries. Concerning CFCs, unsuccessful procedures have been signalled by some MS by using a standard form 14 to cancel the CFC, while in others by publishing a CAN on “non-award” (see point 2.5). Thus, when a CFC is cancelled, it is not clear whether the procedure was unsuccessful (i.e. it ran its course, but no bids were received) or that the CFC was cancelled by the buyer (e.g. because the planned budget disappeared, because the notice was published by accident in the first place, etc.). Decision Cancelled or unsuccessful CANs are excluded from CAN-based indicators, cancelled CNs are excluded from CN-based indicators. This is mainly because, besides being part of an unsuccessful procedure, notices may be cancelled because they were published twice or published by accident, in which case they are definitely not relevant. Furthermore, cancelled CANs are not relevant from the "procurement results" point of view, because no procurement is taking place. We remove these notices even though cancellation patterns may introduce bias across MS. Furthermore, we are aware that it can be argued that cancelled notices reflect practices and results in MS, as they reflect behaviour that buyers have at least tried to execute. Except for VEATs (see 3.2), we do not cancel notices just because their "following" or "preceding" notice has been cancelled, because this would be biased thanks to different proportions of missing links between notices in different MS. Future In 2.0.9, unsuccessful procedures should clearly be signalled using a CAN and this is a legal obligation. On the other hand, notices can no longer be cancelled, only corrected. This does not cover all user needs, so a possibility to clearly cancel notices should be introduced in future forms.

Including CANs informing about a non-award 2.5.Issue In 2.0.9 forms, a CAN could either inform about the award of a contract or the non-award of a contract (i.e. about an unsuccessful procedure). Before 2.0.9, it was not clear how to inform about unsuccessful procedures. Some buyers have simply not published anything, some have published a cancellation notice of the CFC, and some published CANs with no winner and no value (e.g. HU) or put "cancelled" or "unsuccessful" (in national language, seen e.g. in FI) in the winner name field. Looking superficially at the 2.0.9 data, where information about unsuccessful procedures is available, it seems that for the countries which have been using 2.0.9, in 2016 roughly 10% of lots were not awarded. Decision Since it seems very hard to distinguish for example between errors and intention, and the extent of the issue is small, we do not exclude any CANs (e.g. those with missing winner name). Future This issue should be resolved in 2.0.9, because non-award must be clearly marked in a CAN.

Date of publication of a notice vs. date of award of a contract 2.6.Issue


9

When calculating CAN indicators per year, there are two ways of deciding which notices to include: either the notices published in a given year, or contracts awarded in a given year. Since, according to the directives, a CAN may be published up to 30 days after the act of awarding a contract, this can influence results, because different notices will be included in the results for different years. This can play a role when comparing data from different sources, e.g. budget, invoices, and procurement notices. (Furthermore, since in many public administrations there is a budgetary pressure to award contracts in a given calendar year, which often gets done in December, the proportion of contracts which would be moved from year to year may be a bit higher than “30 days maximum” would suggest10.) Decision We use the notice publication date because it is never missing in the data and it is not human-entered, but automatically generated by the publication office, which means there can be no typos.

Notices under different directives 2.7.Issue There are several procurement directives, and each one is to a certain degree different (e.g. different thresholds, different legal obligations). In other words, "procurement" means something a bit different under, for example, the general procurement directive and the concession directive. Also, slightly different data gets collected under each directive (e.g. there is no information about the procedure in the concession directive, beyond whether there was a call for competition or not). Decision We include all the directives (including concessions) in our indicators, because we believe that the concepts are similar enough for answering policy-relevant questions. We note that authorities also sometimes publish notices falling under the wrong directive. However, we do exclude directly awarded notices11 which, in Annex D, specify that "The procurement falls outside of the scope of application of the Directive".

Notices under different legal regimes within a directive 2.8.Issue Within a procurement directive, some types of procurement are regulated differently than the rest (e.g. different thresholds, different legal obligations). In particular, this is the case for design contests, under the new directives12 for "social and other specific services", and in the old directives13 for "services listed in Annex II B of 2004/18/EC” (and corresponding annexes in 2014/17/EC and 2009/81/EC). Decision We do not include design contests, as they do not lead to the award of a contract, and thus are not in our opinion comparable to other types of procurement. We include the other light regimes: ""social services" and "Annex II B", because we consider them to be sufficiently similar. In particular, the old directives' "Annex IIB services" are included, because their basic regulation - "Contracts which have as their object services listed in Annex II B shall be subject solely to Article 23

10

Also, December is a busy part of the year so a non-urgent thing like publishing notices may be more often pushed to January. 11

Procedure codes AWP, NOC and NOP. 12

Directives 2004/17/EC, 2004/18/EC and 2009/81/EC. 13

Directives 2014/23/EU, 2014/24/EU, 2014/25/EU.


10

and Article 35(4))14" - still means that technical specifications should be non-discriminatory and publication of notices is mandatory (even though they can be marked as "not for publication"). Furthermore, since Annex IIB is a normal part of procurement under the new procurement directives, including such services also for the past makes the data more comparable across years. However, "Annex II B services" are excluded for the indicator on the use of negotiated procedures, because in their case this procedure is not a "bad practice" – the lighter regime was put in place precisely because publishing calls for competition may be difficult for these types of services.

Different types of calls for competition 2.9.Issue There are slight differences in the regulation of different types of CFCs: the contract notices, the concession notice, the prior information notices used as a call for competition, and the qualification system. Decision For simplicity, we include all of them in CFC-based indicators. We note this does not bias the calculation of the time necessary between receiving bids and awarding contracts. It is true that, assuming the data is available, this period is clearly different for qualification systems and possibly also prior information notices with a call for competition. However, these notices will always be excluded from the calculation of the indicator, because it is based only on open procedures, and PIN CFC cannot be used for an open procedure, while QS does not contain information about which procedures was used.

Notices from EU and other international institutions 2.10.Issue TED data does contain only procurement by Member States, but also by EU and other international institutions. Decision For the purpose of country indicators, these notices are excluded, because they are not national procurement (e.g. they can bias the description of Belgian procurement).

Fields moving from notice level to lot level 2.11.Issue An indicator could be based on field at the CAN level, but then got moved to lot level or contract award level. This happened for many fields between the standard forms under the 2004 and 2014 directives, e.g. award criteria, duration of the contract. This means that the level of analysis may differ from year to year and in some years (in particular 2016) also country to country, because some countries are using the 2.0.9 forms, some older forms. Decision We calculate the indicators at the most granular level possible. For example, award criteria have been published at notice level before 2.0.9 and at lot level beginning with 2.0.9. Thus, we calculate them at the lot level. If a notice does not contain information at the lot level, we take the information at the notice level as being part of every lot. (This is equivalent to weighting the notice by the number of lots.)

14

2004/18/EC Art. 21


11

Notices with buyers coming from multiple countries 2.12.Issue In 2.0.9 forms, notices can have multiple buyers, which raises the question of which country’s indicators should they be attributed to. Decision We follow the approach taken by the Publication Office, which chooses the first one on the assumption that it is the lead buyer.15 A more complicated approach can be to take the country from which the highest number of bidders comes. We do not go for the more complicated option for the sake of consistency, simplicity, and the very low number of these cases. Future In future forms, it should be possible to explicitly mark a lead buyer.

Definitions of fields 2.13.Issue

Some fields are not clearly defined for all situations, so there is imperfect comparability. This may be because of a lack of definition in the standard forms (e.g. for the number of bids, see 3.1) or different procurement legislation (e.g. possibly for negotiated procedures, see 3.2). For all definitional issues, we note that in larger or less centralized countries definitions could cause discrepancies even at the level of federal states, individual authorities, or different providers of e-procurement platforms. Decision

Since no information is available to correct this information, we use it as it is.

Future Definitional ambiguities should be resolved in a future version of the form.

15

This is the case e.g. for https://ted.europa.eu/udl?uri=TED:NOTICE:19573-2016:TEXT:EN:HTML .

https://ted.europa.eu/udl?uri=TED:NOTICE:19573-2016:TEXT:EN:HTML


12

3. Methodological issues for individual indicators This chapter explains methodological issues linked to only one indicator. We do not repeat the implication of the issues explained above on individual indicators (occasionally, we include references). The order of the indicators and their name corresponds to those used in the Single Market Scoreboard. We do not repeat the definitions given in the scoreboard.

One bidder 3.1.Issue This indicator relies on the field "Number of offers received" (pre-2.0.9) or "Number of tenders received" (2.0.9). However, the meaning of this field is likely not perfectly harmonized across the EU. In fact, at least three interpretations are possible:

All tenders received (anecdotally, this is the case e.g. in the Czech Republic).

All tenders received, which were not excluded for technical reasons (e.g. wrong format of the tender).

All tenders received, which were not excluded for technical reasons, passed the exclusion grounds and the selection criteria (anecdotally, this is the case e.g. in Hungary).

Furthermore, before 2.0.9, the "Number of offers" might have also been interpreted differently in case of two-stage procedures, where instead of being understood as "number of tenders" (as requested by the directives) it might also be understood as the first sign of interest by companies, i.e. the "number of or requests to participate". Besides the definition, we also note that alternative indicators with a similar message could be calculated: not only the "proportion of contracts awarded where there was just one bidder", but also the "median number of bids" or the "average number of bids". Finally, as discussed in point 2.2.3, in some countries, contracts within framework agreements are published on TED. In these cases, there is no clear consensus on how to inform about the number of bids. In particular, whether for framework agreements without the reopening of competition the number of bids should be 1 (because there was no competition) or the original number of bids; and whether for framework agreements with reopening of competition, the number of bids should correspond to the number of bids for this purchase, or the original number of bids. Decision We do not include any corrections, as information on different definitions is not easily accessible. We use the percentage of one-bidder contracts because it is most robust to outliers and is frequently used in the academic literature16. Furthermore, it can also be argued that the difference between one and two bids (i.e. no competition and some competition) should be given more importance than the median and average bid indicators give. We exclude framework agreements and direct awards from the calculation of the One-bidder indicator.

No calls for bids 3.2.Issue

16

See e.g. https://link.springer.com/article/10.1007%2Fs10610-016-9308-z.



https://link.springer.com/article/10.1007%2Fs10610-016-9308-z


13

We are interested in how pervasive the use of negotiated procedures without prior publication of a call for competition17 is. This means they need to be counted, and then compared to the overall number of procurement in the MS. However, in some countries, it may be the case that non-publication of CANs is falsely lowering the number of NOCs (see 2.2.2). Also, national legislation may influence what is categorised as a negotiated procedure without prior publication of a call for competition. For example, anecdotally, in Czech Republic and Bulgaria many contract modifications are perceived as new procedures requiring a new notice, while in other countries they would only be perceived as a modification of an existing contract. Decision We use all available CANs, because it is not possible to easily correct for CAN non-publication18. For reasons stated in 2.2, we use the number of CFCs as the denominator, as the proxy for the overall number of procedures in a MS. We do not know the extent of differences linked to modifications in national legislation, thus we do not take this into account for the indicators either. This could be partially addressed by choosing only direct awards with a given reasoning (e.g. including only “extreme urgency” or excluding “additional deliveries”). However, quality of data may limit this possibility.

Future Under the 2014 procurement directives the approach to modifications has been harmonized in the EU and a modification notice has been introduced. This means that the data should be more comparable across MS, because modifications in some MS will not be published as CANs.

Publication rate 3.3.Issue The annual report on public procurement indicators contains two types of publication rates: one where the denominator is the gross domestic product (GDP), and another where the denominator are a part of government expenditures. The main shortcoming of the former is that it does not reflect the different weight that government spending has in the economy of a particular MS. The main shortcoming of the latter is that expenditure statistics are less harmonized that GDP statistics, and thus may be less comparable than GDP.

17 These correspond to the NOC and NOP procedure codes. For an explanation why AWP are not policy

relevant, see section 2.8. 18 We have tried to estimate the lower boundary of CANs, at least for some countries, on the basis of published

VEATs. This means estimating the number of NOCs as the higher of the following two numbers: "number of NOCs" and "number of VEATs". This leads to more accurate results for Finland, as the number of comparable VEATs is around 200, while the number of NOCs around 30, and possibly slight more accurate results for Denmark. However, for all the other countries, this does not change the results. Thus, we do not include it in our country indicators, as it adds significant complexity for only a marginal gain. We are in contact with Denmark and Finland concerning this anomaly. For results to be reliable, note that it is necessary to compare apples with apples. In this case, this means comparing NOC informing about an award with VEATs under the NOC procedure (i.e. not under the AWP “procedure”) and were not followed by a NOC CAN informing about a non-award.


14

Solution In the Scoreboard we use only the GDP-based expenditure indicator for simplicity, as GDP is a more widely known concept.

Cooperative procurement 3.4.Issue Before 2.0.9 forms, the only information on cooperative procurement was a tick box indicating that "The contracting authority is purchasing on behalf of other contracting authorities". This was intended to mean procurement in one of two cases: "The contract is awarded by a central purchasing body" and "The contract involves joint procurement". This has been made explicit in the 2.0.9 forms, where these two options are listed instead of the option on joint procurement. However, as always, there are exceptions to how uniformly this definition has been accepted across the EU. Anecdotally, in Belgium, this field has been interpreted as meaning that the management of the procurement procedure has been outsources (e.g. to a legal company) - which explains the high values of this indicator for Belgium. Solution Since we do not have sufficient information for correcting the data, we take it data as it has been submitted by buyers. To compare data between pre-2.0.9 and 2.0.9 forms, we interpret "The contract involves joint procurement" as being ticked when at least one of its two more detailed equivalents in 2.0.9 has been ticked.

Award criteria, lowest price 3.5.Issue Two simple indicators can be calculated using award criteria:

a) average weight (in %) given to price compared to other criteria, across all notices/lots b) average % of notices/lots where lowest price was the only criterion.

While the first indicator is more accurate for knowing the importance of price, it can be calculated only with 2.0.9 forms. On the contrary, the second indicator is less accurate, but can be calculated from 2.0.8 as well as 2.0.9 forms. Also, this is one of the indicators which has been moved from notice level to lot level between 2.0.8 and 2.0.9. Decision Since in 2016 we have both 2.0.8 and 2.0.9 data, we use the second indicator. The lowest available granularity in a given year is used, in line with point 2.11. Future Once an overwhelming majority of data comes from 2.0.9, we should start using the first indicator.

Decision speed 3.6.Issue This indicator relies on information from a CN (deadline for submission of tenders) and CAN (award of contract). Since one CN can have multiple CANs, it is necessary to choose which CAN will be used for the calculation on the indicator.


15

Furthermore, we note that missing links between CNs and CANs (see results of indicator 2.9) can bias the results. For example, in one country, all links between CNs and CANs may be automatically included by the e-procurement system. In another country, links may need to be added manually, and thus buyers who are well-staffed and well-trained may be more likely to include these links (and thus be included in the calculation of the indicator) than buyers with less resources. Since better trained authorities will also evaluate tenders faster, this may lead to positively biased results for the latter country. In some cases, there are outliers in the data – from negative speeds to speeds of many years. Solution In case of multiple CANs per CN, we take the last CAN. We remove observations with a negative decision speed as well as decision speeds above the, somewhat arbitrary, threshold of two years.

Missing values 3.7.Issue As discussed in point 2.5, if a pre-2.0.9 CAN was used to inform about a non-award, this can slightly bias the results of this indicator – the value field will be empty, because there was no contract awarded. As many other indicators, this indicator can be calculated at the level of a CAN or at the level of a contract award. While for most countries the difference in results is under 20%, in some countries it can be very large – for example, in Finland, there are almost 0% missing values at the contract award notice level, but more than 75% missing at the contract award level. Solution Similarly to other cases, we calculate the indicator at the lowest possible level of granularity, that is at the level of contract awards. We do not do other corrections of data provided by buyers, as we do not have sufficient information about that.

Missing registration numbers 3.8.Issue In pre-2.0.9, only buyer registration numbers were required, while in 2.0.9, also the company registration numbers are required. Solution In 2017, for comparability purposes, we calculated missing registration numbers only on the basis of buyer registration numbers. From 2018, we include both. It should be noted that higher participation of SMEs (esp. sole providers, e.g. physical people providing a service) can have a negative effect on the indicator, because they are less likely to have a registration number (which is not considered personal data).

Missing calls for competition 3.9.Issues Besides procedures which do not have a call for competition, there is also an ambiguity about how contracts within framework agreements should be reported. Specifically, what is the previous notice for a contract within a framework agreement – is it the CAN announcing the framework agreement, or the CN? Solution Since there has not been a clear policy on this issue, we exclude framework agreements from the indicator.

Documents

TED CSV open datadata.europa.eu/euodp/repository/ec/dg-grow/mapps/TED_advanced_… · This is a background document for the TED CSV open data and is intended for advanced users who