NIA Evaluation Report - GOV.UKrandd.defra.gov.uk/Document.aspx?Document=13506... · NIA periods and there are key changes in agri-environment policy and delivery over this time period

Annex 1:

Counterfactual

Report

Final

November

2015

Collingwood

Environmental Planning Limited

in partnership with

GeoData Institute

Defra project code: WC1061

Monitoring and Evaluation of Nature

Improvement Areas: Final Report (2012-15)

Annex 1: Counterfactual Report November 2015

Monitoring and Evaluation of NIAs: Collingwood Environmental Planning Final Report (2012-15) i

Project title: Monitoring and Evaluation of Nature Improvement Areas: Phase 2

Contracting organisation:

The Department for Environment, Food and Rural Affairs (Defra)

Defra project code: WC1061

Lead contractor: Collingwood Environmental Planning Limited

Address: 1E The Chandlery, 50 Westminster Bridge Road, London, SE1 7QY, UK

Contacts: Ric Eales (Project Director) [email protected] Owen White (Project Manager) [email protected]

Tel. 020 7407 8700

Website: www.cep.co.uk

Partner organisations:

GeoData Institute:

Contact: Chris Hill

Email: [email protected]

Cascade Consulting:

Contact: David Kingsley-Rowe

Email: [email protected]

Report details: Report title: Monitoring and Evaluation of Nature Improvement Areas: Final Report (2012-15). Annex 1: Counterfactual Report

Work Package: WP4: Undertaking analysis and evaluation

Date issued: 27 November 2015

Purpose: To detail the work undertaken to explore the counterfactual as part of the Monitoring and evaluation the Nature Improvement Areas (NIA) Phase 2 project.

Version: Final

Author(s): Sheate, W., Baker, J., Papadopoulou, L., Hill, C., Hornby, D., White, O., Eales, R.

Citation: Collingwood Environmental Planning (2015) Monitoring and Evaluation of Nature Improvement Areas: Final Report (2012-15). Annex 1: Counterfactual Report. Defra Research Project WC1061.

Acknowledgements: The research project was commissioned by Defra, working in partnership with Natural England and guided by a Project Steering Group which included members of both organisations, representatives of the NIA partnerships and an independent expert ecologist.

The project team are grateful for to the national stakeholders, NIA Partnership Chairs and representatives from the NIA partners who participated in the interviews and survey, and the Natural England staff who supplied the data used as part of the counterfactual work.

Photograph Credits: Cover photograph:

Cannon Hill Park after woodland improvements, Birmingham and Black Country NIA (September 2013 © Su James)

mailto:[email protected]


http://www.cep.co.uk/




Monitoring and Evaluation of NIAs: Collingwood Environmental Planning Final Report (2012-15) ii

Contents

Executive Summary ......................................................................................................... iii 1. Introduction to the Work ........................................................................................... 1

1.1 Context and report structure ................................................................................................................ 1 1.2 Audience for this report ........................................................................................................................ 1 1.3 Aim and objectives ................................................................................................................................ 1 1.4 Summary of the approaches ................................................................................................................. 2 1.5 Overall logic model and its role in shaping research collection ............................................................ 3 1.6 How results were used within the NIA M&E Reporting ........................................................................ 4

2. Methodological Development and Analysis ................................................................ 5 2.1 Approach 1 methodology ...................................................................................................................... 5 2.2 Approaches 2 and 3 - environmental stewardship data ....................................................................... 8

3. Results - What Difference Have the NIAs Made? ...................................................... 10 3.1 Introduction ........................................................................................................................................ 10 3.2 Inputs and processes ........................................................................................................................... 10 3.3 Biodiversity .......................................................................................................................................... 21 3.4 Ecosystem services .............................................................................................................................. 23 3.5 Social and economic ............................................................................................................................ 28

4. Methodological Analysis and Implications ................................................................ 30 4.1 Methodological Limitations ................................................................................................................ 30 4.2 Reflections ........................................................................................................................................... 32 4.3 Implications for future monitoring and evaluation ............................................................................. 33 4.4 Conclusions ......................................................................................................................................... 36

Appendix 1: NIA logic Model and Attribution Routes ...................................................... 37

Appendix 2: NIA Partners' Survey Questionnaire ............................................................. 39

Appendix 3: Summary of Non-Parametric Statistical Results from the Survey .................. 47

Appendix 4: Partnership Chairs Interview Schedule ........................................................ 51

Appendix 5: National Stakeholders Interview Schedule ................................................... 52

Appendix 6: Approaches 2 and 3 - Detailed Methodology ............................................... 53

Appendix 7: Summary of Results from Approach 1 .......................................................... 58

Appendix 8: Graphs and Tables for Individual NIAs (Approach 2 trajectory analysis) ........ 60

Appendix 9: Graphs and Tables for Individual NIAs (Approach 3 comparative Analysis) ... 66


Monitoring and Evaluation of NIAs: Collingwood Environmental Planning Final Report (2012-15) iii

Executive Summary

Introduction and summary of the methods

One key challenge for the monitoring and evaluation (M&E) of the Nature Improvement Areas (NIAs) has been the assessment of the counterfactual, i.e. to provide an assessment of what would have occurred in the absence of the NIA initiative. This is necessary to better understand the difference the NIA initiative has made. The work presented in this report was developed to address this challenge.

The overall aim of the counterfactual work was:

To support and supplement the overall evaluation of the NIA initiative to be reported at the end of the three year funding period of the 12 initial NIA partnerships – by providing evidence related to the counterfactual.

The objectives were:

To provide an assessment of the difference the NIA initiative has made compared to what would have happened without the NIA initiative in place.

To implement an evaluation of the counterfactual focusing on selected outcomes and impacts of the NIA initiative.

To develop and adopt an approach that is practical, proportionate and cost effective within the budget available.

To innovate and test approaches for evaluating the counterfactual as part of natural environmental policy development and implementation.

The NIA M&E Steering Group worked with the M&E team to develop a range of approaches that were suited to the available resources and the needs of the NIA M&E project. Three separate but complementary approaches were developed and implemented:

Approach 1: Qualitative ‘counterfactual scenario’ based on: semi-structured telephone interviews with national stakeholders and NIAs’ Partnership Chairs; online survey with NIA partners, and analysis of NIA Funding Agreements.

Approach 2: Trajectory analysis of environmental stewardship data1 comparing trends in non-Entry Level Stewardship (ELS) option applications before the NIA initiative (2005 – 2011) with the data during the NIA initiative (2012 – 2014).

Approach 3: Matched comparison of environmental stewardship data in NIAs and non-NIA areas and the rest of England.

The research was designed to be exploratory with respect to Approaches 2 and 3 in particular, to test whether existing datasets could be used to support a quantitative approach to the counterfactual.

Results from the evidence collection

Although the results from the three approaches are presented here separately (Table A and Figure A), the results were analysed using triangulation across the three approaches and with the data from the online tool used in the main NIA M&E project; the outcomes from the three approaches are summarised here. Table A reports on the outcomes from the interviews and survey with respect to interviewees views in relation to the key themes of the overall evaluation.

1 Environmental stewardship refers to funding for farmers and land managers provided by the UK Government. More information is available at gov.uk https://www.gov.uk/guidance/environmental-stewardship

https://www.gov.uk/guidance/environmental-stewardship


Monitoring and Evaluation of NIAs: Collingwood Environmental Planning Final Report (2012-15) iv

Table A: Summary of Approach 1 results

Theme NIA partners (survey)

NIA partnership chairs (interviews)

National stakeholders (interviews)

Biodiversity The majority of respondents considered that biodiversity benefits had been delivered over and above what would have happened anyway.

The majority of partnership chairs considered biodiversity benefits to have been delivered over and above what would have happened anyway.

Some national stakeholders felt that biodiversity activities funded through environmental stewardship grants might have happened anyway, but most national stakeholders felt that NIAs sped up delivery and improved coordination of these activities.

Ecosystem services

Significant variation in responses about the extent that the NIA initiative has led to additional ecosystem service outcomes across NIAs depending on objectives and nature of NIAs.

The majority of partnership chairs felt that there was a greater focus on ecosystem service outcomes from habitat management than would have happened otherwise.

Specific benefits noted included flood/water management, woodland products and carbon storage and sequestration.

The majority of national stakeholders felt that the NIAs raised the profile of ecosystem services and some felt that improved coordination between Water Framework Directive (WFD) and biodiversity activities was achieved.

Social and economic wellbeing

Respondents felt that community relations were most improved by the NIA partnerships among these areas of activity.

The majority of partnership chairs felt that the NIA government grant funding enabled projects with broad objectives that would have struggled to get off the ground otherwise.

No views were expressed by national stakeholders.

Partnership working

93% of respondents considered partnership working to be more (57%) or much more (36%) effective than would have happened otherwise.

The majority of partnership chairs felt that funding for staff enabled people to work with and support other partners and challenged silo-thinking.

The majority of national stakeholders felt that the NIA initiative had led to broader and better coordinated partnerships than would otherwise have existed.

Other findings

Narrative comments added to the survey by respondents indicated an overall sense of achievement among partners.

88% of respondents considered NIAs to have contributed to Lawton’s vision, though a three year timescale was deemed too short to achieve large scale and lasting improvements.

A majority of respondents identified improvements in the development of a shared vision and sharing of information and resources.

A majority of respondents expressed that NIA status generated wider stakeholder engagement

The majority of partnership chairs felt that NIAs: provided a forum for bringing partners together around a common vision; and improved awareness of the landscape scale approach within partner organisations.

The majority of partnership chairs felt that the NIA government grant funding and NIA status acted as a catalyst for match funding and galvanising partners. Flexibility of use of funding was seen as critical.

Most partnership chairs felt that three years not long enough to make a real difference.

Some partnership chairs felt

Some national stakeholders felt that the NIA initiative served to accelerate and broaden the scope of activities that may have happened anyway.

The majority of national stakeholders felt that: the flexibility of funding enabled new types of partnerships; and that committed, enthusiastic partners made a relatively small amount of money go a long way.

Some national stakeholders also felt that the NIAs helped to bring statutory agencies together and improved communication between them.


Monitoring and Evaluation of NIAs: Collingwood Environmental Planning Final Report (2012-15) v




and had benefits in attracting match funding.

Additional workload and administrative burden were the main challenges expressed by the NIAs.

that the NIA government grant helped ‘plug a gap’ left by cuts to statutory agencies and local authorities who might otherwise have funded some of the types of activity completed by NIA partnerships.

Figure A shows that the number of non-ELS option applications across all NIAs was stable or declining from 2006 – 2009 before increasing every year up to 2014 where it decreased substantially. The linear trend line (trajectory) for the years pre-NIA government grant funding suggests that the number of non-ELS option applications during the grant funded NIA period exceeded what might have been expected to occur within the NIAs, but only for 2013.

Figure A: Individual NIAs and England Except the NIAs: Total Non-ELS Option Count 2006 – 2014

Note: BBC = Birmingham and Black Country. DP = Dark Peak. DVGH = Dearne Valley Green Heart. GTM = Greater Thames Marshes. HL = Humberhead Levels. M&M = Meres and Mosses of the Marshes. MB = Morecombe Bay. MD = Marlborough Downs. ND = North Devon. NV = Nene Valley. SD = South Downs Way Ahead. WP = Wild Purbeck.

While an initial assessment of the results from Approach 2 might suggest that the number of non-ELS option applications appears to have exceeded what might have been expected based on the historical trend and the national average, at least for 2013, more detailed analysis indicates this would be an invalid conclusion. Within individual NIAs there is no clear pattern in the pre-NIA or NIA periods and there are key changes in agri-environment policy and delivery over this time period which explain much of the observed trends; for instance the change from countryside stewardship schemes to environmental stewardship between 2006 and 2010 is responsible for much of the increases observed in Figure A and the reduction in available options in the final year of the funding round (2014) is likely to be the cause of the small number of applications for that year, and the reason for such a peak in the preceding year (2013). The aggregate trend also exhibits very high sensitivity to the trends of individual NIAs.

The matched comparison analysis for individual NIAs and their comparator areas (Approach 3) also resulted in no clear pattern, i.e. there appeared to be no statistical difference caused by the NIAs.

-

100,000

200,000

300,000

400,000

-

5,000

10,000

15,000

20,000

2006 2007 2008 2009 2010 2011 2012 2013 2014

BBC DP DVGHGTM HL M&MMB MD NDNV SD WPEngland Except NIAs Linear (Pre-NIA)

Engl

and

NIA

s


Monitoring and Evaluation of NIAs: Collingwood Environmental Planning Final Report (2012-15) vi

However, due to the nature of the non-ELS dataset it was not possible to provide a clear quantitative assessment of the difference made by the NIAs. While there appeared to be no significant difference caused by the NIAs, in either the trajectory analysis or the matched comparison analysis, it is not possible to conclude that there was no difference simply because there are so many confounding factors in play, including wider changes in agricultural policy over time and due to the wide variation among the NIAs themselves. Analysis of the online tool and the evidence from Approach 1 suggests that rather than increasing the total quantity of non-ELS agri-environment options the NIAs focused on improved coordination of options, spatially and in terms of the types of options.

What difference have the NIAs made?

Compared to what would have occurred anyway, the NIA initiative provided seed funding which brought partners together and allowed them to develop shared visions for the NIA areas. The evidence suggests that these, and other, NIA specific mechanisms, including the flexibility of the funding and the learning and knowledge exchange within the NIAs, increased the speed and scale of delivery of some activities and outcomes and potentially led to long-term benefits for communities and the environment.

Conclusion

The results from Approach 1 show the value of detailed qualitative research as it has provided insights into the counterfactual and the attribution routes whilst also contextualising the output data provided by the online tool.

The results from Approaches 2 and 3 suggest it is not possible to provide a quantitative assessment of the counterfactual for non-ELS agri-environment options at the scale of the NIAs initiative using the existing non-ELS dataset. This is due to the signal to noise challenge (change due to the NIAs is small relative to other factors), but also because the NIAs themselves are highly diverse and as such it is not appropriate to aggregate them to the initiative (programme) level. These lessons are consistent with the conclusions drawn in the Countryside Stewardship Facilitation Fund (CSFF) scoping study, undertaken as part of the wider NIA evaluation project2. The non-ELS dataset could be useful if any future related initiative was designed with specific objectives that utilised the relevant existing indicators.

A particularly important lesson relates to M&E and natural environmental policy/initiative design of this kind:

1. Flexibility at the individual project level implementation (as with the NIAs) constrains the extent to which aggregate evaluation at the programme level (initiative wide) is possible or meaningful; and

2. The alternative to flexibility is to creating rigid policy/initiative objectives that can be aggregated across the initiative through greater comparability. Ideally these would build on existing national datasets (and core indicators) to allow for efficient comparative analysis.

The following broad lessons for counterfactual evaluations more generally can be drawn:

Use logic model and attribution routes to identify mechanisms for delivery which are unique to your initiative. Use these to prioritise your evidence collection.

Work with stakeholders to refine and ground truth your logic model, attribution routes, hypothesis you wish to test and your results.

Engage with multiple stakeholder types using a range of research methods.

2 Bennett, T., Phillips, P., Sheate, W., Eales, R. and Baker, J. (2015), Countryside Stewardship Facilitation Fund Monitoring and Evaluation Scoping Study – Final Report by Collingwood Environmental Planning and GeoData for Defra (July, 2015).


Monitoring and Evaluation of NIAs: Collingwood Environmental Planning Final Report (2012-15) vii

Recognise the diversity of projects and account for this within your analysis - when is it suitable/not suitable to aggregate to the initiative (programme) level?

Look for opportunities for triangulation to enhance robustness, using multiple evidence sources considering the same questions or themes.


Monitoring and Evaluation of NIAs: Final Report (2012-15) 1 Collingwood Environmental Planning

1. Introduction to the Work

1.1 Context and report structure

One key challenge for the monitoring and evaluation (M&E) of the Nature Improvement Areas (NIAs) has been the assessment of the counterfactual, i.e. to provide an assessment of what would have occurred in the absence of the NIA initiative. This understanding is necessary to better understand the difference the NIA initiative has made.

The evaluation of the NIA initiative against the counterfactual is part of the existing M&E process including the collation of baseline data for M&E indicators within the online reporting tool and semi-structured interviews with NIA project officers in Year 2. It was agreed with the NIA M&E Steering Group that additional research was required to further understand the difference the NIA partnerships have made, in their areas and collectively, over and above what would have happened without their introduction3.

This report presents the results of that work using the following structure:

1. Introduction to the work – (this section) summarise the context to the work.

2. Methodological development and analysis – describes the methodologies, the processes followed and how the results were analysed.

3. Results – what difference have the NIAs made – sets out results of the research and assesses the difference the NIAs have made, compared to what would have happened anyway.

4. Methodological limitations and reflections – includes conclusions and describes the limitations to the methodologies, what worked well / less well, considers implications for future research and lessons for future evaluations.

5. Appendices – provides additional supporting detail.

1.2 Audience for this report

This report has been produced primarily for the Department for Environment, Food and Rural Affairs (Defra) and Natural England officials involved in the NIA initiative and related monitoring and evaluation. It is also assumed that the results will be of interest to stakeholders involved in the NIA initiative and to contractors and officials who are involved in the monitoring and evaluation of similar environmental initiatives.

1.3 Aim and objectives

The overall aim of assessing the counterfactual was:

To support and supplement the overall evaluation of the NIA initiative to be reported at the end of the three year funding period of the 12 initial NIA partnerships – by providing evidence related to the counterfactual.

The objectives were:

To provide an assessment of the difference the NIA initiative has made compared to what would have happened without the NIA initiative in place.

To implement an evaluation of the counterfactual focusing on selected outcomes and impacts of the NIA initiative.

3 This research was funded through an extension to the NIA M&E Phase 2 contract (Defra project WC1061)



To develop and adopt an approach that is practical, proportionate and cost effective within the budget available.

To innovate and test approaches for evaluating the counterfactual as part of natural environmental policy development and implementation.

1.4 Summary of the approaches

The three approaches were:

Approach 1: Qualitative ‘counterfactual scenario’ based on multiple social science research methods.

Approach 2: Trajectory analysis of environment stewardship data4.

Approach 3: Matched comparison of environment stewardship data in NIAs and non-NIAs and the rest of England.

Although the approaches are described separately the analysis presented in Section 4 is based on analysis which combined and triangulated (i.e. comparing the results from) results from all three approaches. The triangulation also included evidence collected under the existing M&E framework (notably the monitoring data recorded by NIA partnerships using the online tool), and work undertaken as an extension of the M&E Phase 2 contract in relation to the Scoping of the Evaluation for the Countryside Stewardship Facilitation Fund (CSFF)5.

1.4.1. Approach 1 – qualitative counterfactual scenario

The qualitative counterfactual scenario approach used social research methods to collate and triangulate responses from the NIA partnerships and national level stakeholders about what difference the NIA initiative has made, and what potentially would have happened if the NIA initiative did not existed.

The data collection involved a mixture of social research methods, including:

Semi-structured telephone interviews

Online survey

Analysis of NIA Funding Agreements.

These approaches have produced an evidence base with breadth and depth. The stakeholder groups engaged included national stakeholders, NIA partnership chairs and staff in NIA partner organisations, reflecting varied levels of engagement and interests to ensure that a range of opinions and views were considered. The online survey allowed the participation of multiple hard-to-reach respondents offering flexibility and anonymity for respondents. Interviews allowed the research team to explore perceptions in-depth and gain insight into responses, while the document review and analysis initially formed the basis of the survey and interview questions and following the research was used to help highlight and explain the observed results in the analysis.

1.4.2. Approaches 2 and 3 – environmental stewardship: trajectory and comparisons

Approaches 2 and 3 used environmental stewardship data to understand the uptake of agri-environment options within the NIA areas; comparing the trends for the pre-NIA and NIA periods (Approach 2) and NIAs with matched non-NIA areas (Approach 3)

4 Environmental stewardship refers to funding for farmers and land managers provided by the UK Government. More information is available at gov.uk https://www.gov.uk/guidance/environmental-stewardship 5 Bennett, T., Phillips, P., Sheate, W., Eales, R. and Baker, J. (2015), Countryside Stewardship Facilitation Fund Monitoring and Evaluation Scoping Study – Final Report by Collingwood Environmental Planning and GeoData for Defra (July, 2015).

https://www.gov.uk/guidance/environmental-stewardship



Agri-environment data was used because it was: recorded nationally; reported annually; and improving the uptake of agri-environment funding was referred to in the Funding Agreements of all NIAs.

For Approach 2 the hypothesis was that the trajectory of non-Entry Level Stewardship (ELS) agri-environment options would increase (in number and extent) during the NIA government grant funded period, compared to the preceding years. For Approach 3 it was assumed that uptake of agri-environment options would be higher in NIA areas compared to national trends and matched non-NIA areas.

Approaches 2 and 3 are recognised as innovative and this project has tested their utility for evaluating the counterfactual of natural environmental policy.

1.4.3. Why the three approaches were chosen

The NIA M&E Steering Group worked with the M&E team to develop a range of approaches that were suited to the available resources and the needs of the NIA M&E project. It was agreed at an early stage that three general approaches could be used: a qualitative ‘counterfactual scenario’; looking at the trajectory before the NIA and during the NIA; and comparing NIA landscapes to non-NIA landscapes.

It was then necessary to consider which stakeholders to engage with to develop the counterfactual scenario and which data to use.

After various iterations the two sets of interviews and the survey were considered the best way of engaging with a broad range of stakeholder types whilst also offering sufficient detail.

Regarding the data set for Approaches 2 and 3, it was felt necessary that both approaches were applied to the same data set (to allow for comparison) and that the chosen data set should be:

1. Recorded nationally.

2. Reported annually.

3. Relevant to all NIAs.

4. Available for at least the three year period prior to the establishment of the NIA partnerships.

5. Relevant to the objectives of the NIA initiative and all NIAs.

6. Provided at an appropriate spatial scale.

Environmental stewardship data was considered the only data set that met these requirements. The other candidate data set was Monitor of Engagement with the Natural Environment (MENE) but this did not meet criteria 2 and 6.

1.5 Overall logic model and its role in shaping research collection

A logic model is an approach frequently used in evaluations and the UK Government’s handbook for evaluation, the Magenta Book, suggests that logic models are used. A logic model demonstrates how an intervention is understood to contribute to possible or actual impacts. Within evaluation they are used to provide a framework to understand the intervention and therefore understand what information is needed to monitor and evaluate it.

Logic models are simple structures showing what is expected to go into a policy, what activities will occur during implementation and then an indication of the likely effects of these. The NIA M&E project has been based around a high-level logic model, but the counterfactual provided the opportunity to further elaborate the logic model to help understand whether and how the NIA initiative has made a difference.



1.5.1. Use of the logic model

The logic model used in the counterfactual work built on the high level logic model used in the NIA M&E project. It was developed through a series of iterations by the M&E Phase 2 project team and with the NIA M&E project Steering Group based on the evidence collected through Years 1 and 2 of M&E process. The final logic model is presented in Appendix 1. It should be noted that this is an initiative wide logic model and that individual NIA partnerships will have their own logic models reflecting their contexts and objectives.

The agreed logic model was used to refine elements of Approach 1. This was done through the identification of a series of ‘attribution routes’; these are the ways in which the NIA initiative is assumed to lead to outcomes and impacts. The attribution routes are included in Section 3 where an assessment is made as to whether the evidence validates the assumptions behind these attribution routes, or not (see Boxes Box 3 to Box 6).

Attribution routes are relevant to the counterfactual as they represent the main ways that the NIA initiative, as opposed to what would have happened anyway, is considered likely to result in outcomes and impacts. By identifying and then testing these attribution routes the M&E Phase 2 project team were better placed to say whether effects, for example from changes in the area of habitat management reported by NIA partnerships through the online tool, can be ascribed to the NIA initiative. The interviews within Approach 1 also allowed for the identification of others ways in which the NIAs have contributed to observed outcomes.

The logic model influenced the design of Approach 1 in the following ways:

Interview and survey questions were structured around the logic model

Interview and survey questions were reviewed to ensure that the identified ‘attribution routes’ and other key features of the logic model were considered within the questions.

Evidence relating to the attribution routes is presented, where appropriate, in Section 4.

1.6 How results were used within the NIA M&E Reporting

The results from the counterfactual are presented within the respective thematic section (i.e. biodiversity, ecosystem services etc.) within the Final Report. A synthesis of the results is presented in Part IV of the Final Report.



2. Methodological Development and Analysis

2.1 Approach 1 methodology

2.1.1. Approach 1 – survey with NIA partners

A quantitative online survey was undertaken with NIA partners (stakeholders representing organisations who have a formal or semi-formal relationship with an NIA partnership). The survey focused on identifying the impacts of the NIA initiative as experienced and perceived by each individual respondent and the NIA they represented/were most familiar with.

The questionnaire was short and focused. Questions were developed through multiple iterations within the project team and comments/edits by the NIA M&E Steering Group. Consideration was also given to existing best practice guidance for survey design6. Participants were assured that results would be reported anonymously, that their details would not be shared with anyone outside the research team and that all data would be deleted at the end of the project.

For the majority of questions respondents were presented with a five point Likert scale7 and were asked to assess how positive or negative the impact of the NIA partnership had been in their area, compared to what would have occurred without the establishment of the NIA initiative. Multiple-choice, drop-down lists and ranking questions were also used to engage participants in the questionnaire, while non-compulsory open-ended questions gave respondents the opportunity to elaborate on their responses and provided valuable insight on the quantitative results recorded. The survey questionnaire is included in Appendix 2 of this report.

The questionnaire was shared as an online survey using Survey Monkey, an online survey and questionnaire software. The survey was set up on the web and email invitations with a direct web link to the survey were sent to the 12 NIA project managers and M&E leads who were asked to forward it to all stakeholders they considered to be partners. The link was accompanied by an email to encourage participation emphasising the opportunity presented to partners and stakeholders to directly input into the M&E process and to share their experiences and insights. The project team recorded the number of partners and stakeholders that the survey was sent to so that a response rate could be calculated.

Over 260 individuals, including partner organisations and NIA partnership staff (project officers/managers, M&E leads etc.), were invited to participate via email. The survey remained open to participants for four weeks (January - February 2015). Within three weeks of the questionnaire being live the M&E team analysed those who had already responded and sent targeted reminder emails to relevant partners to increase the response rate. The questionnaire was designed to take a maximum of 20 minutes to complete with most respondents expected to complete it in less than 15 minutes.

A total of 122 responses were received, out of which 109 were used in the analysis8, corresponding to a 46% response rate. The spread across NIAs is shown in Figure 1.

6 MRS (2011) Guidelines for Questionnaire Design. 7 Likert scaling is a one-dimensional scaling method commonly used in social research where respondents are asked to rate a set of items or concepts based on their understanding/knowledge of the subject. In the NIA partner survey a five point rating scale was used, where partners were asked to assess the impact of the NIA initiative on the various elements evaluated or to indicate their agreement/disagreement with a set of statements. In the majority of the questions the scale rated from ‘Much improved’ to ‘Much worse’, with ‘Don’t know’ and ‘Not applicable’ choices added where deemed appropriate. The survey questionnaire is provided in Appendix 2 for a more detailed vie w. 8Responses that only answered the first question were excluded from the analysis.



Figure 1: Number of responses per NIA

Source: NIA Partner Survey 2015 Base: 109 respondents

Prior to the analysis, the data collected from the survey were cleaned9 and open-ended responses were reviewed and recoded as appropriate to be assigned to existing or new answer options/codes10. The analysis was done using Excel and covered:

Respondent characteristics – the NIA respondents represented/were most familiar with and the organisation and role of the respondents.

Descriptive statistics on the various impacts experienced - Improved/Worsened, More effective/Less effective, Agree/Disagree.

Ranking of changes and challenges experienced – weighted11 scores calculated for each of the choices ranked as top three.

Qualitative comments - providing insight to quantitative results.

Identified examples – including the examples provided by respondents.

Due to a relatively small sample size and varied number of partners across the NIAs, the number of respondents in the survey was not uniformly distributed across the 12 NIAs (see Figure 1Error! eference source not found.). The distribution of the sample and the characteristics of the data mean that these were non-parametric data12. Aggregating and reporting on results across non-parametric data poses a challenge, as aggregate results risk being unrepresentative of each individual group – in this case each NIA – within the data set.

Kruskal-Wallis non-parametric tests were used to understand whether there was significant variation

9 E.g. where some questions were not answered, those answers were removed so that the sample population for a specific question may be less than n=109. 10 Where ‘other’ was an option a wide variety of answers may have been provided. Some of those could be assigned to an existing code/category, while others, when mentioned more than once, were combined under a new code/category e.g. ‘Farmer/Landowner’ category added to the list of stakeholder categories. 11 In order to take account of the choices ranked as second and third, weights were assigned and the top three choices across responses were calculated on the basis of the weighted aggregate scoring each change or challenge gathered. The weights assigned to the number(n) of responses for each of the three ranks were: Ranked 1st: (n*3) Ranked 2nd: (n*2) Ranked 3rd: (n*1) For each of the change or challenge the sum of the above was calculated as (n*3)+ (n*2)+ (n*1) 12 I.e. the data were not distributed normally.

5

6

7

7

7

8

8

10

11

11

12

17

0 2 4 6 8 10 12 14 16 18

Greater Thames Marshes

Marlborough Downs

Morecambe Bay Limestone & Wetlands

Nene Valley

Northern Devon

Birmingham and Black Country

The Dark Peak

Humberhead Levels

Dearne Valley Green Heart

Wild Purbeck

Meres and Mosses of the Marches

South Downs



in responses, testing for the following explanatory variables:

NIA: effect of the specific NIA the respondent represents/is most familiar with.

NIA partnership size: effect of NIA partnership size, by number of partners (small/medium/large)13.

Type of respondent: Government Agency, Farmer, NGO etc.

The purpose of these tests was to determine whether and where it was appropriate to use aggregated descriptive statistics, e.g. percentage of all respondents. Non-parametric testing allowed the project team to report statistically robust results and avoid inappropriate aggregation and reporting. This exercise was designed only to identify statistically significant variation within the data.

Where there was no significant variation among respondents, aggregate results are reported. But where there was significant variation, the nature of the variation is identified (e.g. 10 NIAs reported improvement while two NIAs reported no improvement) and qualitative analysis (including open-ended comments made through the survey, interview results and document analysis) sought to explain it. Any significant variation was also indicated in figures with the use of an asterisk (*).

The statistical testing was undertaken for all questions (see Appendix 3), with the exception of the ranking questions, though only ‘interesting’14 variation is highlighted in the main body of this report.

2.1.2. Approach 1 - interviews with partnership chairs

Semi-structured telephone interviews were undertaken with each of the 12 NIA Partnership Chairs. The objective was to understand at the individual NIA partnership level how partners may, or may not, have worked together in the absence of the NIA initiative and the potential impact of this on related outcomes.

The 12 initial NIA Partnership Chairs were considered appropriate as their role meant they were able to offer insights into potential comparisons with landscape-scale delivery outside NIAs, to consider how the partnerships may have functioned without NIA government grant funding, and their role covered the breadth of NIA activities. They were considered well placed to reflect on any causal links between NIA funding, status and achievement of delivery.

An interview schedule was developed, and agreed with the Steering Group (see Appendix 4). The approach was based on using a relatively small number of questions based around the M&E themes with the interviewer exploring these topics during the interviews. The Partnership Chairs received the questions at least a week before the arranged interview. The questions were adapted, where necessary, to reflect individual NIA partnership characteristics. The interviews, lasting approximately 30 minutes each, were undertaken by phone and recorded (but not transcribed).

The interview discussions were written up in note form and the results analysed using the same thematic frame (based on the M&E evaluation framework) that was used in the analysis of the national level stakeholder interviews. The identity of each partnership chair was anonymised for the purpose of reporting.

2.1.3. Approach 1 - interviews with national level stakeholders

Semi-structured interviews were undertaken with national level stakeholders to gain insights and perspective from strategic level stakeholders into the counterfactual of the NIA initiative as a national policy intervention.

13 Each of the 12 NIA partnerships was categorised as small, medium or large based on the number of partners; Small (up to 10 partners), Medium (11-20 partners), Large (21+ partners) 14 We report only on ‘bold’ differences. Therefore if there is significant variation between the NIAs on different levels of improvement that is reported, where appropriate, as aggregated e.g. X% of respondents identified some level of improvement.



A list of potential stakeholders was initially provided by members of the Steering Group. The project team used these contacts to map out the appropriate interviewees to ensure coverage across a range of stakeholder organisations. Finally, seven semi-structured interviews were undertaken in January 2015 with participants from the Environment Agency, Forestry Commission, Wildlife Trusts, RSPB, the National Association for Areas of Outstanding Natural Beauty, the Game and Wildlife Conservation Trust and the National Farmers Union. The identity of national level stakeholders was anonymised for the purpose of reporting.

An interview schedule was developed, and agreed with the Steering Group (see Appendix 5). The general approach was to reflect on the three years of the government grant funded NIA initiative and what might have happened without it. The interviews provided insights from national level NIA stakeholder groups and enabled the project team to understand how the NIA partnerships have been working, any differences between NIA delivery and what is considered likely to have happened in the absence of the NIA initiative. It also explored aspects such as the value-added of NIA partnerships as a whole, the facilitation role NIA partnerships have had in realising wider benefits, the ‘brand’ value of being an NIA for partners involved and the benefits of operating at a landscape scale.

The interviews, lasting approximately 30 minutes each, were conducted by phone and recorded (but not transcribed). The interviews were written up in note form and the results analysed using a thematic frame based around the M&E evaluation framework (i.e. biodiversity, partnership working, ecosystem services etc.). The thematic frame is a social science technique which allows for the responses or parts of interviewee responses to be ‘coded’ (tagged) and then grouped with other related responses. This allows for a more structured, consistent and efficient analysis15. The populated thematic frame was used to organise and analyse all the data collected through the three different research elements described under Approach 1.

2.2 Approaches 2 and 3 - environmental stewardship data

Approaches 2 and 3 used the same dataset (Environmental Stewardship Data – England). The process by which these data were obtained, prepared and analysed is summarised here and described in more detail in Appendix 6.

In summary: it was necessary to ‘cookie cut’ the data so that the records relevant to NIA areas could be separated from the England wide data. The dataset was filtered to remove all entries of Entry Level Stewardship (ELS) only options (since we were interested in non-ELS agri-environment options) and arranged by the year that the option was entered into the system. These data sets were then analysed.

For Approach 2, using the cleaned and cut data a spreadsheet format (in Excel) was used to compare data within (individually) and across (collectively) NIA partnerships. This enabled the generation of graphs illustrating trends collectively and individually in the years preceding and during the NIA period.

This analysis was undertaken to generate graphs for:

The number of non-ELS options within an individual NIA area per year from FY 2005-06 to 2011 – 12 (pre-NIA) and from 2012-13 to 2014-15 (NIA).

The number of holdings with non-ELS options across the NIA areas from FY 2005-06 to 2011 – 12 (pre-NIA) and from 2012-13 to 2014-15 (NIA).

For approach 3 the graphical analysis was done in MiniTab as was a Mann-Whitney U test.

2.2.1. Justification for comparators

Approach 3 compared uptake between the NIAs and with two different landscape characterisation

15 Bryman, A. (2012) Social research methods (4th edition)



datasets. These were National Character Areas16 (NCAs) and Agricultural Landscape Types (ALTs).

These comparator areas were considered appropriate as the emphasis of the NIAs was on actions contributing to the whole landscape, rather than actions focused on the specific features. Therefore the comparison is best made between NIAs and equivalent landscape / agricultural areas.

The landscape character assessment approach, enshrined within the National Character Areas17 dataset, seeks to objectively identify landscape units based on the contributory landscape factors. The character areas reflect areas with similar combinations of geology, topography, drainage, vegetation, past land use and settlement patterns. Similarity across the continuous variables is based on principle component analysis and generates geographically-discrete areas even though areas across the country may have similar suites of variables.

The Agricultural Land Types approach (Swanwick et al., 200718) seeks to generate a generalised, higher-level, landscape-scale combination of these similarities and thereby represent broader landscape types based largely on the agricultural character. These are described as:

“distinct types of landscape that are relatively homogeneous in character. They are generic in that they may occur in different parts of the country, but wherever they occur they share broadly similar combinations of geology, topography, drainage patterns, vegetation and historical land use and settlement pattern”.

The justification for using these classifications is that they are used in targeting HLS. It was also used within an earlier review of the effects of environmental stewardship on landscape character and quality (Defra/NE 2013)19 although these have been aggregated further (to six classes) within other studies (Boatman, et. al., 201020). More detail on the statistical and analytical techniques used is presented in Appendix 6.

16

Natural England (2015) Natural Character Areas: http://publications.naturalengland.org.uk/category/587130 17 Swanwick, C. (2004) The assessment of countryside and landscape character in England: an overview, in Countryside Planning – New Approaches to Management and Conservation’ (eds K.Bishop and A. Phillips), 109-124. Earthscan. London. 18 Swanwick, C. Hanley, N and Termansen, M. (2007) Scoping study on agricultural landscape valuation. Final report to DEFRA. Department of Landscape, University of Sheffield. 19 (Defra / NE 2013) BD5303 Monitoring the Effects of Environmental Stewardship on Landscape Character and Quality Developing a method for reporting and monitoring the direct and cumulative impacts of Environmental Stewardship on the maintenance and enhancement of Landscape Character and Quality. LUC / Fabius Consulting. 20 Boatman, N, Willis, K., Garrod, G. and Powe, N. (2010) Estimating the wildlife and landscape benefits of environmental stewardship. Report to The food and environment research agency (FERA)

http://publications.naturalengland.org.uk/category/587130



3. Results - What Difference Have the NIAs Made?

3.1 Introduction

This section presents the analysis of the results for all three approaches with evidence from the main NIA evaluation materials brought in as appropriate. Results from each approach were viewed in isolation (the results from Approach 1 are summarised in Appendix 7) and in conjunction to allow similarities and differences in the results to surface and ultimately to provide a thematic assessment of what difference the NIAs have made (the counterfactual).

The results are presented by the four themes used in the main NIA evaluation with other findings presented separately. Approaches 2 and 3 are included within the ecosystem services theme as environmental stewardship relates to ‘sustainable agriculture’ which is reported under the ecosystem services themes following the NIA M&E framework.

The section begins with a summary assessment of the difference the NIAs have made.

Box 1: What difference have the NIAs made?

Compared to what would have occurred anyway, the NIA initiative provided seed funding which brought partners together and allowed them to develop shared visions for the NIA areas. The evidence suggests that these, and other, NIA specific mechanisms, including the flexibility of the funding and the learning and knowledge exchange within the NIAs, increased the speed and scale of delivery of some activities and outcomes and potentially led to long-term benefits for communities and the environment.

The rest of this section expands and evidences this statement.

3.2 Inputs and processes

Survey participants and interviewees overwhelmingly believed that partnership working within the NIAs had improved compared to what would have been possible without the NIA initiative.

In particular, respondents referred to increased engagement of non-environmental/traditional stakeholders, improved frequency, nature and quality of partnership working, greater learning through sharing and the adoption of a shared vision and agenda.

“Improved working relationships with NIA partners leading to constructive discussions on other issues where previously there had been little cooperation. Improved understanding of shared agendas…Landscape scale conservation focus has given us a wider perspective on activities of partners”. (NIA 12)

“We are now working with a broader range of partners than we would have done otherwise.” (NIA 3)

The partnership chairs further commented that the NIAs have successfully challenged silo-thinking and got people from different, often competing backgrounds to talk about common objectives. Flexibility on the part of statutory agencies was also considered valuable in allowing for plans to develop and change on the ground while their engagement in discussions on future steps and help in lifting barriers to delivery were also deemed vital.

National level stakeholders felt that the improved partnership working was reflected in the increased delivery of outcomes, with specific reference to non-biodiversity outcomes. This was partly attributed to the firm timescales and deliverables of the initiative that partners had to work towards, and partly to the role of Defra in providing forward planning and a clear signal for the need to work together. The NIA government grant funding was also mentioned as an enabling factor in bringing partners together at the outset.



Across the 12 NIAs, 93% of participants in the partner survey believed that partnership working was ‘more’ (57% of respondents) or ‘much more’ (36%) effective (Figure 2).

Box 2: Presentation of survey results

Where there was no statistically significant variation among respondents (as per the results of the Kruskal-Wallis non-parametric tests) aggregate results are reported. But where there was significant variation, the nature of the variation is identified (e.g. 10 NIAs reported improvement while 2 NIAs reported no improvement) and qualitative analysis (including open-ended comments made through the survey, interview results and document analysis) sought to explain it. Any significant variation was also indicated in figures with the use of an asterisk (*).

Figure 2: Improvements in partnership working from the establishment of the NIA initiative


Local support from Natural England was perceived to have improved, but not universally: while over 60% of participants in the survey experienced some improvement as a result of the NIA initiative there was significant variation in responses depending on the NIA. Ten of the NIAs saw improvements, while two NIAs noted no difference. There was also a correlation between the size of the NIA partnership and the perceived support; the smaller the NIA the more likely it was to report improvement in support, while the majority of ‘No different’ scorings came from the larger NIAs.

One in six partners, made reference to challenges in partnership working including inadequate support from partners (12%) and friction/disagreement with partners (7%) though one noted those were successfully resolved.

The evidence suggests that the attribution route described in Box 3 has been largely validated, although it is too early to judge how long, and in what form, partnerships will continue beyond the NIA initiative period.

36%

57%

5% 2%

1% Partnership working

Much more effective

More effective

No different

Less effective

Don't know



Box 3: The Value of Partnership Attribution Route

3.2.1. Development of a shared vision

The vast majority of partners (94%) in the survey (see

Figure 3) felt that the development of a shared vision was one of the biggest benefits of the NIA. It was also ranked as the third most important change experienced by participants to the survey.

Building partnerships and

facilitating partnership

working

Agreed funding agreements

Well supported and functioning

partnerships and Shared learning and

dissemination of expertise within the

NIAs

Effective, durable partnership

(beyond pilot period)

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Access to additional funding

Local support from Natural England*

Development of a shared vision

Coordination of activities

Learning through dissemination

Sharing of data and information

Facilitated the delivery of other plans

Added value*

Community and civil society involvement

Much improved Improved No different Worse Much worse Don't know Not applicable



Figure 3: Benefits from the establishment of the NIA initiative

Source: NIA Partner Survey 2015 Note: Asterisk (*) indicates there was significant variation in responses (refer to Appendix 3) Base: 109 respondents

Through interviews national level stakeholders and partnerships noted the importance of stakeholders sharing a common agenda, and felt that this has clear benefits for partnership working.

Across stakeholders, there was a view that partnership working in the NIAs has brought new partners together and facilitated the development of an understanding and appreciation among them that is likely to exist after the NIA initiative.

“Enhanced networks of contacts and development of a shared vision with appreciation of partners’ contributions to that.” (NIA 3)

“Those organisations which might superficially be seen as competitors have worked together to play to their strengths and move towards a collective, shared vision in a collaborative way.” (NIA 8)

The value of a shared vision and objectives was also emphasised in interviews with partnership chairs and national level stakeholders, who when considering the legacy of the NIAs, recognised the value of a network of partners with a joint vision that can effectively and efficiently work together in the future.

“You can’t overestimate the importance of having partners already there, already talking to each other with projects they can take of the shelf. This saves a lot of time and resource. Other initiatives provide these opportunities but the NIAs breadth of partners is a positive.” (NL 4)

“NIAs have been a mechanism for partners working together to create a joint vision and working together for a common goal.” (NL 7)

The evidence suggests that the attribution route described in Box 4 has been largely validated.

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%








Added value*





Box 4: Shared Vision Attribution Route

3.2.2. The importance of committed partners

Partnership chairs and national level stakeholders identified a difference in the NIAs in the skill, enthusiasm and commitment of the people involved. It was commonly felt that the energy and will partners invested in the implementation of the NIAs

“have made a small amount of money go an awfully long way.” (NL 4)

This was attributed to a good selection of project managers driving the delivery but also a feeling of pride by partners from participating in something innovative.

“The enthusiasm of those involved. Been around and seen a few initiatives, but the enthusiasm is marked. Heartening to see such enthusiastic responses.” (NL 2)

“…all partners have really pushed to make delivery successful. Not sure what has driven this, may be that everyone really believes in what they're doing, but result has been there hasn’t' been a need to chase people. Passion - in what is being done - has made the partnership really good.” (PC 9)

3.2.3. Learning and sharing of data and resources

A benefit of the development of a closer working relationship between partners was identified in the sharing of information, knowledge and resources. Numerous qualitative responses to the survey referred to the willingness of partners to share resources and support each other in the delivery of the NIA objectives.

“This [Partnership working] has involved sharing expertise and equipment, collaborating and cooperating on design and delivery of projects to achieve better outcomes, and working more closely together at a strategic level such as cooperating with writing joint funding bids etc…” (NIA 10)

The partnership chairs and national level stakeholders, also made references to a breakdown of existing barriers to information and data sharing that have unlocked knowledge valuable to landscape scale conservation.

“Data is / was [sic] often 'kept behind lock and key' in one organisation - but NIA status has freed this data to be used across organisations - greatly increased data sharing and joint working to use and get value from data. Everyone has been really good at freeing up information and data - as an input to landscape scale conservation this has been vital - to think big.” (PC 9)

“Previously people tended to be very stuck on boundaries, but now recognise that the natural environment doesn’t see boundaries.” (PC 6)

3.2.4. Dissemination of learning from the NIAs

National level stakeholders were keen to see that learning from the NIAs is shared. One expressed a concern about this wealth of information remaining unutilised:

Developing a shared vision

Assessing local opportunities for

restoring and connecting nature

Funding agreement identifying priority

actions

Delivery of those actions



“It’s not been clear what NIAs are piloting - never been written down what specific NIAs are piloting. There is a need to draw out lessons from them and what is different. Need to view them as pilots, rather than potentially successful projects delivering potentially good stuff but without any clarity about what it means for others.” (NL 6)

Best practice events were mentioned by national level stakeholders as a useful way of sharing information at a local level.

“These have been an opportunity for the NIAs to get together and hear the latest conservation thinking - on topics like habitat networks.” (NL 2)

Partnership chairs referred more generally to an open attitude towards the sharing of information that has characterised the NIAs from the outset, although national level stakeholders felt that there had been a missed opportunity and that non-NIAs had not been able to engage with or share experiences with the NIAs.

“Landscape scale initiatives outside of NIAs are not drawn in very well to the learning the NIAs are developing. It feels to some that the NIAs are an exclusive club.” (NL 5)

“NIAs meet up but these were not felt to be that accessible for other groups. This was felt to be a shame.” (NL 3)

Partners in the NIA survey identified learning through dissemination of knowledge as a key benefit of the NIA initiative (89% believed it was ‘Improved’ - 64% of respondents - or ‘Much Improved’ - 25%). A few participants suggested that similarly others could benefit from the learnings emerging from the NIAs. Capitalising on the experience of partners and practitioners could provide valuable advice on the benefits and pitfalls in funding, partnership working and delivery of similar initiatives in the future.

“This is a new way of working and culture change takes time in organisations; this is however being achieved. There needs to be a mechanism to capitalise on this into the future.” (NIA 3)

“Lots of good work and implementation of fresh thinking. [Would] Be great to see wider dissemination of what went right and what went wrong…” (NIA 10)

However, the latter respondent also commented that sharing of the challenges had not been particularly encouraged within the NIA initiative.

The evidence suggests that the attribution route described in Box 5 has been largely validated, but that some opportunities to maximise these benefits have been missed – notably by greater information exchange between the NIAs and other landscape scale conservation approaches.

Box 5: Innovation and information exchange

In-house and external skills and capacities

Sharing of experience and

expertise between and

within the NIAs

Knowledge exchange and Best practice

events

Shared learning and

dissemination of expertise within and beyond the

NIAs

A sense of innovation and

optimism



3.2.5. NIA grant funding

The NIA grant funding was widely perceived as a catalyst in partnership working, the mobilisation of

resources and the delivery of the NIAs’ plans and objectives. The flexibility of the NIA funding was

seen as its main enabling characteristic.

Reflecting on the counterfactual, a few partnership chairs expressed the opinion that the NIA

funding filled in a gap created by the recent staff and budget cuts, which left statutory agencies less

able to support conservation work on the ground. Had this not been the case, the NIA funding and

output could have been more impactful:

“Output could have been even more impressive.” (PC 5)

NIA funding as a catalyst

Both national level stakeholders and partnership chairs identified the NIA funding as the enabling factor for the establishment of the partnerships, attracting partners historically less engaged to landscape scale conservation. More importantly, as discussed under Partnership working, it has enabled effective planning and management on a landscape scale and coordination of partners. One of the national stakeholders referred to this as the “no surprises approach” (NL 2), characterised by the advance planning and sharing of plans across partners.

The funding gave the NIAs the ability to maintain or employ staff (e.g. Project Officers/managers) and supported the direct delivery of actions on-the-ground. It also helped ensure good governance and reporting for the duration of the NIA initiative. Partners in the NIA survey were very positive on the impact of the NIA initiative over and above what would have occurred without their establishment. That impact of the NIAs was identified in:

Work being undertaken that otherwise would not have been possible

“[We] have used the opportunity of the X NIA to provide on-farm advice in a coordinated fashion and have also coordinated educational days on-farm for members of the public. The NIA funding was the main reason this managed to actually happen, instead of just talking about it.” (NIA 8)

“We have been able to work with farmers and landowners on over 400 hectares of limestone grassland, developing management plans and securing HLS funding for the farmers. This would not have happened without the funding that came with the NIA.” (NIA 10)

Greater extent of work that otherwise would have been possible only at a small scale

“A much greater extent of habitat restored over a much wider area than would have been contemplated without the NIA.” (NIA 1)

“NIA funding has allowed X to undertake major landscape-scale moorland restoration works to complement work funded on surrounding areas by HLS.” (NIA 12)

“…As a result of the NIA we have worked on more sites because of the additional funding, and have advanced our thinking to deliver a sustainable strategy for long-term nature conservation.” (NIA 3)

“With the available HLS funding we could only have chipped away at the edges of the scrub overgrowth problem, but together with NIA funding and [partner] support we have been able to make a big difference which will last. This is very motivating.” (NIA 8)

Fast-tracked progress of work that otherwise would have taken longer

“Speeded up work on getting X into favourable condition…” (NIA 5)

“Broadly we've done 'more' and 'faster' than would have been likely otherwise.” (NIA 7)

National stakeholders and partnership chairs were more reserved in their statements, placing the impact of the NIA funding and initiative in the context of the extent of work delivered. Specifically,



national stakeholders believed the main contribution/impact of the NIAs had been in delivering things that had been already planned, more quickly and efficiently than would have occurred otherwise, thus accelerating plans that already existed.

The NIA grant also reportedly acted as a catalyst for mobilising resources and opening the way for match-funding opportunities, for example funding proof-of-concept projects (e.g. in biofuel production). Partnership chairs found the initial injection of funding necessary to tap into further resources. Specifically, grant funding was considered to free up time for partners to consider and complete other funding applications. Moreover, the range of new partners (e.g. universities, local businesses) is thought to have brought new funding opportunities through partnerships. As seen in

Figure 3, 84% of respondents in the partner survey also felt that access to additional funding ‘improved’ (55% of respondents) or ‘much improved’ (29%) as a result of the NIAs, while another 11 out of 12 NIAs thought the NIAs added value through mobilising non-monetary resources, such as equipment and volunteers. Qualitative comments also affirmed the importance of this effect:

“It has helped to lever additional funding from my own Local Authority” (NIA 5)

“With the availability of additional direct and match funding, we have been able to strengthen existing and emerging partnerships …, that may not otherwise have occurred, which bodes well for future relationships and projects.” (NIA 6)

The evidence suggests that the attribution route described in Box 6 has been largely validated, but it is recognised that some of the funding which has been attracted was displaced from other areas and would have been used anyway. There is however some evidence that the NIAs were more effective in their use of resources, due to mechanisms such as their shared vision and greater coordination with partners. Respondents noted that the long-term sustainability of funding for landscape scale innovations will depend on the priorities of other funding sources.

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%








Added value*





Box 6: Defra Seed Funding Attribution Route

Flexibility of NIA funding

One of the main differences of the NIA initiative, compared to other natural environment funding, was the flexibility of the NIA funding.

That flexibility, as discussed in interviews with national stakeholders and partnership chairs, was reflected in the ability of the NIA partnerships to allocate funding to people (e.g. project managers and facilitators) rather than specific projects. According to an interviewee:

“Flexibility on part of Defra / NE - not being too prescriptive has been really useful - has allowed for evolution and for things to change and develop on the ground.” (PC 9)

Having flexible funding was also thought to have allowed room for the NIAs to be more innovative. This was considered to be relatively unique as most available funding prescribes specific outcomes and even processes. Finally, national level stakeholders reflected that Defra being the funding source meant that the NIA initiative was not promoted by individual organisations and was broader in its remit and more likely to integrate public and private organisations and NGOs.

The timescale of delivery was only mentioned by two national level stakeholders, but they provide an interesting outlook on the difference in perspectives. For one interviewee the short timescale for delivery was considered to be one of the main benefits of the NIA funding. Having to spend the funds quickly was thought to create a financial incentive to get projects off the ground and maintain the partners’ focus. On the contrary, the same fact was described as a challenge by another participant to the survey:

“The lack of financial flexibility i.e. the desperate need to spend money within a financial year. [There] should be [an] opportunity to allow funding to be carried forward to following year.” (NIA 6) ……

3.2.6. Benefits of the NIA status/brand

The NIA status was believed to have had a galvanising effect on stakeholders, raising the profile of the conservation work and generating more interest and support. There was also a view that, in cases, it acted as leverage for partners in planning discussions and funding decisions.

The partnership chairs focused on the benefits the NIA status had in partnerships, evaluating the effect of the NIAs as regenerating:

“The NIA status provided a renewed focus and energised the existing partnership.” (PC 3)

Being part of a national initiative and having a government endorsed status as a NIA was also seen as an opportunity by local authorities to engage with local actors and build nature improvement works into local planning policies.

Further, the raised profile and designated NIA status had been used by some partners to leverage additional funding, which was a benefit also mentioned in the NIA partners’ survey.

Defra funding Funding attracted by the

NIAs from various sources and contributions in-kind

Sustainable financial base for partnership working and

continued landscape scale innovations



National level stakeholders noted that the process of receiving the NIA badge lacked clarity while it caused feelings of frustration among partners who have traditionally operated in a very similar way to the NIAs before those were created.

“There is a perception that instead of promoting landscape scale conservation generally - Defra have pushed NIAs as THE model.” (NL 6).

This reflected concerns from some national level stakeholders that there were tensions between NIAs and other landscape scale conservation approaches. Other national level respondents did not feel this was the case and, perhaps unsurprisingly, this issue was not noted in the partnership chair interviews.

3.2.7. Increased profile of landscape scale conservation

National level stakeholders had a view that the NIA initiative had raised the profile and awareness of the landscape scale approach and this had generated greater belief and commitment to landscape scale management.

An overall sense of achievement among partners is evident in the perceived improvements and benefits of the NIA initiative compared to what would have occurred without the initiative. A number of respondents noted a strong desire for the funding and NIA initiative to continue as three years were deemed by some too short a timescale to achieve large scale and long-lasting improvements. Nonetheless, 89% of respondents to the NIA partner survey believed the NIA initiative had contributed to Lawton’s vision of creating more, bigger and better places for wildlife, while it also instilled a sense of optimism about landscape conservation (see

Figure 4).

National level stakeholders, reflecting on the impact of the NIAs on landscape scale conservation (LSC), viewed it as positive, but not significant compared to what would have occurred without the NIAs. The general view was that improvements in landscape scale conservation were not necessarily an impact of the NIA initiative per se but rather part of a wider policy shift towards landscape scale conservation, of which NIAs were a part. Stakeholders were seen to have a better understanding of the concept and were actively looking for opportunities to integrate it. NIAs contributed to these changes and reinforced the landscape scale approach as a delivery model.

“The main benefit is that NIA provided an umbrella term for LSC activity (...) In that sense NIA as a term is useful when trying to embed the concept.” (NL 6)

36%

45%

26%

50%

44%

59%

12%

8%

11%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

The NIA has created a sense of innovation andoptimism about landscape scale conservation

The NIA has created more, bigger and better placesfor wildlife

The NIA has enhanced the benefits that natureprovides for people

Strongly agree Agree Neither agree nor disagree Disagree Don't know



Figure 4: Successes of the NIA initiative

Source: NIA Partner Survey 2015

Base: 101 respondents

Partners to the NIA survey also commented that the NIAs provided stakeholders with confidence in achieving landscape scale conservation by providing funding and generating support for potentially controversial landscape changes. This was directly linked to partnership working and coordination and sharing between stakeholders, which appeared to be a prerequisite in delivering landscape scale projects. NIAs were thought to have provided the

“Ability to apply aspects of a landscape approach to nature conservation in a conurbation”. (NIA 6)

“As an organisation we have focused more on a landscape scale approach over the whole area - joining up our sites with others.” (NIA 10)

3.2.8. Challenges of the NIA initiative

The most frequently mentioned challenges experienced as a result of the NIAs were additional workload mentioned by 48% of the respondents and administrative burden mentioned by 43% of the respondents (Figure 5).

Figure 5: Challenges experienced as a result of the NIA initiative


Some of the additional challenges described by respondents (‘Other’) included comments on the complexity and burden of the M&E process as well as difficulties in communication between partners and reconciliation of targets and objectives.

36%

45%

26%

50%

44%

59%

12%

8%

11%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

The NIA has created a sense of innovation andoptimism about landscape scale conservation

The NIA has created more, bigger and better placesfor wildlife

The NIA has enhanced the benefits that natureprovides for people

Strongly agree Agree Neither agree nor disagree Disagree Don't know

0% 10% 20% 30% 40% 50%

Other

Disagreement / Friction with partners

Interference with the delivery of other plans

Inadequate support from partners

Costs disproportionate to funding

Unforeseen problems in final delivery

High expectations from partners, civil society etc.

Administrative burden

Additional workload



“The complexity of the imposed monitoring initiative was a surprise.” (NIA 7)

“This is a new way of working and culture change takes time in organisations.” (NIA 3)

Participants were also asked to rank the top three challenges they experienced (Figure 6). Results followed the same pattern of the previous question with the top challenges identified as:

1. Additional workload

2. Administrative burden

3. High expectations

Figure 6: Ranking of top three challenges experienced as a result of the NIA initiative


The remaining challenges were not experienced to any significant degree, while qualitative comments clarify that ‘Unforeseen problems in delivery’, which was the fourth most experienced challenge, were mainly restricted to the beginning of the initiative with any problems resolved in due course.

3.3 Biodiversity

The majority of participants in the partner survey and partnership chair interviews felt that biodiversity benefits had been delivered over and above what would have happened anyway.

Responses to the NIA partner survey were especially positive with partners across the NIAs believing that there had been improvement in all elements of biodiversity ( Figure 7). Partners answered ‘Improved’ or ‘Much Improved’ in the following percentages:

88% believed there had been improvement in habitat quality

87% believed there had been improvement in habitat extent

86% believed there had been improvement in habitat connectivity

68% believed there had been improvement in species

0 5 10 15 20 25 30

High expectations from partners, civil society etc.


Additional workload

First Second Third



Figure 7: Effect of the establishment of the NIA initiative on elements of biodiversity


While many stakeholders discussed the successes of specific projects, fewer references were made to landscape-scale outcomes in the survey and interviews. Within the survey results the NIA partners were more confident expressing an opinion on habitat extent and quality; some uncertainty was expressed with regards to habitat connectivity (highest percentage of ‘Don’t know’). Some respondents felt that there is or should be a distinction between improved connectivity and improved knowledge of it, while others thought it might be too soon for results to be assessed.

“Whilst there has been some improvement it should be noted that over a three year initiative this is going to be difficult to evaluate. Significant changes for the long-term will only occur over a longer period.” (NIA 3)

Some reluctance was also expressed in the method of evaluating connectivity improvements, which is a limitation also identified in the literature and caveated in the NIA report (Biodiversity chapter – section 5).

“As an indicator this [Habitat Connectivity] has been trivial and varied between NIAs, the scientific literature is undecided on a definition and proxy to measure this. However I can say it has improved based on improving the quality of select habitats” (NIA 2)

In the survey improvements in species were felt to be less than for habitats. Respondents attributed this to the relatively short timescale of the NIA initiative and felt that some of the improvements expected may not have yet been recorded or indeed realised. Participants also mentioned a lack of benchmark species studies that could provide a baseline for comparison.

“It is too early to say whether habitat connectivity and species have improved or worsened, which only further surveys and assessments would determine.” (NIA 6)

Partnership chair interviewees broadly correspond with the results of the survey. For instance they felt that the NIA partnerships had played an ‘enabling role’, allowing projects which enhanced biodiversity to go ahead when otherwise they would have not. Others felt that NIA funding had enabled activities to happen much faster and on a larger scale than otherwise, even if they may have happened anyway.

The national level stakeholder interviewees had a different perspective. They felt that as, in their view, environmental stewardship funding was the primary mechanism through which the NIAs were improving biodiversity, most of the biodiversity benefits might have happened anyway. Some national stakeholders noted that biodiversity activities which were supported by NIA grant funding were largely additional and that the NIAs might improve the coordination of environmental

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Species

Habitat connectivity

Habitat extent

Habitat quality

Much improved Improved No different Worse Much worse Don't know



stewardship activities. Across the national stakeholders the sense was that the main added value of the NIAs was for non-biodiversity outcomes such as community engagement and ecosystem services projects.

3.4 Ecosystem services

There was general agreement across the interviews and survey that compared to what would have happened anyway improvements in ecosystem services had been perceived and that specific services were targeted in different NIAs. The most frequently noted improvement was recognition of the benefits of the ecosystem approach and the increase in awareness of ecosystem services from practitioners in local authorities, the local community and farmers.

“Putting more emphasis on ecosystem services in our approach and recognising the linkages between biodiversity and habitat creation/enhancement and ecosystem services.” (NIA 11)

“…Awareness of the 'invisible economy' from ecosystem awareness has been much improved” (NIA 8)

National level stakeholders and partnership chairs both mentioned the raised profile of the concept and the benefits of looking at works through the ecosystem services lens.

Specific flood management and water quality outcomes were mentioned in some NIAs and others reported on sustainable woodland products and carbon storage and sequestration. These differences were to some extent expected as improvements in some ecosystem services are linked to the priority habitats of each NIA and each NIA’s specific objectives21 as those were set out in their Funding Agreements.

The greatest perceived improvements were in support for pollinators (65% of partners perceived some level of improvement) and access and quality of green spaces (69% of partners perceived some level of improvement). The partners’ responses varied significantly for the other ecosystem services depending on which NIA they were involved with, for instance:

Publicly accessible rights of way: Seven NIAs perceived some level of improvement while five considered there was no difference. A review of NIA Funding Agreements reveals that among those that identified no improvement in public access, one indeed made no reference to it in their objectives while for another two it was not a main objective of their NIA. The other two NIAs whose partners perceived no difference compared to what would have occurred without the establishment of the NIAs had set out enhancement of public awareness and access as their core objectives.

Sustainable agriculture: 11 NIAs perceived some improvement. While one of the NIAs considered there was no difference, the document review confirmed that sustainable agriculture was in fact not an objective of the NIA.

Woodland products: Seven NIAs perceived improvement though five NIAs considered there was no difference. All five NIAs noting no difference had no references to woodland products in their objectives with a few of them also focusing on different priority habitats in their area. It is also worth noting that woodland products were also missing as an explicit objective from at least two of the NIAs who noted improvements. This could be explained by a number of projects in those NIAs relating to woodland management.

Water Quality: 10 of the NIAs perceived improvement while two NIAs considered there was no difference. One of the latter’s Business Plan made reference to the maintenance of what was considered a good quality of water. The other NIA though, included projects aiming to improve the quality of water through appropriate land management.

21

As laid out in each NIA’s Funding Agreements



Carbon storage and sequestration: Nine NIAs perceived similar levels of improvement, but three NIAs considered there was no difference. Out of the three NIAs noting no improvement, one made no reference to carbon storage while the other two had the objective to investigate the potential benefits of habitat improvements for carbon storage and develop an understanding of the impacts of land use on carbon sequestration.

Flood management: The majority of NIAs perceived improvement in flood management even though for at least three of them it was not framed as an explicit objective. One NIA made no reference to flood management and identified no difference.

3.4.1. Approach 2 – results of the trajectory analysis

Figure 8 shows that the number of non-ELS option applications across all NIAs decreased from 2006 – 2009 before increasing every year up to 2014 where it decreased substantially. The linear trend line (trajectory) for the years pre-NIA government grant funding suggests that the number of non-ELS option applications during the grant funded NIA period exceeded what might have been expected to occur within the NIAs. The national picture is more stable from 2006 to 2013, except for a spike in 2010 after which the number is consistently slightly higher; as with the NIAs there is a substantial decrease in 2014.

Figure 8: Individual NIAs and England Except the NIAs: Total Non-ELS Option Count 2006 – 2014


-

100,000

200,000

300,000

400,000

-

5,000

10,000

15,000

20,000

2006 2007 2008 2009 2010 2011 2012 2013 2014

BBC DP DVGHGTM HL M&MMB MD NDNV SD WPEngland Except NIAs Linear (Pre-NIA)



Figure 9: Individual NIAs: Total Number of Holdings with Non-ELS Option Applications 2006 – 2014


In Figure 9 2006 and 2008 are followed by drops in the total number of holdings. From 2010 there is an increase until, as with Figure 8, 2014 which shows a big drop in the number of holdings within non-ELS option applications. The trend in Figure 8 increases over the pre-NIA period (2006 – 2010) with the number of holdings in both 2012 and 2013 exceeding that predicted by the linear trend line. The national picture is broadly similar (Figure 8): decreasing from 2006 to 2009 before a substantial increase in 2010. 2011 and 2012 have the second and third highest number of holdings, 2013 is slightly lower and 2014 is much lower than all previous years.

Figure 8 and Figure 9 show that within any given year there is a great degree of heterogeneity between individual NIAs. The aggregate trend of a general increase over time, as described earlier, is not consistently repeated at the level of individual NIAs. Table 1 ranks the number of non-ELS option applications for each NIA from 2005 – 2014. This shows that there is no consistent pattern in the years with most options across the NIAs.

Table 1: Years Ranked for Individual NIA and Aggregate NIA Based on Number of non-ELS Option Applications

NIA 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014

BBC - - - 1 2 6 4 5 3 -

DP - - 1 6 4 3 7 5 8 2

DVGH - - 7 - 5 2 6 4 3 -

GTM - 1 9 2 7 4 6 3 8 5

HL 1 2 5 6 8 4 7 10 9 3

M&M 2 8 3 9 5 7 10 4 6 1

MB 1 5 2 4 7 8 9 3 10 6

MD - 4 3 6 - 8 5 1 7 2

ND 1 10 9 4 3 6 5 8 7 2

NV 1 5 8 3 7 10 6 9 4 2

-

500

1,000

1,500

2,000

2,500

3,000

3,500

4,000

0

50

100

150

200

250

300

2006 2007 2008 2009 2010 2011 2012 2013 2014

HL M&M MBMD ND NVSD WP GTMDVGH DP BBCPre-NIA England Except NIAs Linear (Pre-NIA)



NIA 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014

SD 1 3 4 6 9 8 7 10 5 2

WP 2 7 3 8 9 5 4 10 6 1

NIA Aggregate

1 5 3 4 6 7 8 9 10 2

NOTE: The year with most non-ELS option applications is ranked 10, and the least 1. The colours match the rank. BBC = Birmingham and Black Country. DP = Dark Peak. DVGH = Dearne Valley Green Heart. GTM = Greater Thames Marshes. HL = Humberhead Levels. M&M = Meres and Mosses of the Marshes. MB = Morecombe Bay. MD = Marlborough Downs. ND = North Devon. NV = Nene Valley. SD = South Downs Way Ahead. WP = Wild Purbeck.

Appendix 8 presents the graphs and table for individual NIAs for the total number of non-ELS options applications and a table of the number of related holdings within each NIA.

3.4.2. Approach 3 – results of comparative analysis

As the comparative analysis is based on individual NIAs it is not possible to provide a summary at the initiative level. Appendix 9 presents descriptive statistics (scattergrams and graphs) for the individual NIAs.

Across the individual NIAs there appears from the data analysis to be no consistent trend for the NCA and ALT comparisons. It is not possible, therefore, on the basis of the data analysis to provide an aggregated summary assessment of the difference the NIAs have made at the initiative level.

At the level of individual NIAs a Mann-Whitney U test was undertaken and with the exception of 2007 all p-values are greater than 0.05 thus the median density values are not statistically significant and the null hypothesis - that there is no difference between the median density values within NIAs compared to their ALT / NCA comparators - is not rejected (results presented in Appendix 9).



What difference have the NIAs made to non-ELS agri-environment option applications

An initial assessment of the results from Approach 2 suggests that over the government grant funded period across the 12 NIAs, the number of non-ELS option applications appears at first sight to have exceeded what might be expected based on the historical trend and the national average. However, more detailed analysis indicates this would be an invalid conclusion as within individual NIAs there is no clear pattern in the pre-NIA or NIA periods. The aggregate trend (as shown in Figure 8) also exhibits very high sensitivity to the trends of individual NIAs.

As there is no consistency in the observed trends across the NIAs individually it is considered to be unsound to infer a trend from them collectively. This means it is not possible to provide a quantified assessment of the difference that the NIA initiative has made to applications for non-ELS options on the basis of the current dataset.

Data provided by Natural England (Table 5.4 in the Year 3 report) show that the area of land under Environmental Stewardship across all NIAs increased by 10.8% over the period 2012 to 2015 compared to 7.2% across the whole of England22. The survey responses also indicate that the NIAs have made a positive difference with partners from 11 of the 12 NIAs feeling that the NIAs have resulted in improvements in sustainable agriculture (which includes environmental stewardship and management of woodlands).

In addition, national level stakeholders expressed that “much of the NIA initiative has been delivered through HLS” [NL 1]. Other stakeholders made similar comments indicating that in their view the NIAs have been working to improve uptake and use of environmental stewardship.

This suggests that across the NIAs there has been considerable activity related to agri-environment options and that the delivery of non-ELS agri-environment options has improved, but in a way that is not detectable through the methods deployed in Approaches 2 and 3.

It is likely that there are two inter-related reasons for this. The first relates to how the NIAs have been ‘improving’ the use of environmental stewardship funding and the second to the nature of the data in this policy area.

On the first point, respondents to the survey suggested that the NIAs have made a positive impact on the quality, rather than total number, of agri-environment options, for example:

“We have been able to work with farmers and landowners …developing management plans and securing HLS funding for the farmers. This would not have happened without the funding that came with the NIA.” (NIA 10)

“Through funding of a NIA land advisor we have delivered 13 HLS applications that have focused on restoring and linking priority habitats.” (NIA 9)

“We have been able to “provide more support to farmers in agri-environment schemes, improving the quality of delivery and therefore quality of the habitat they are managing.” (NIA 7)

“Co-ordination of agri-environment support with NIA objectives has led to development of more environmentally sustainable agriculture, although there is much room for improvement.” (NIA 2)

These responses, when combined with the other evidence reported in this section suggests that the initial hypothesis for Approaches 2 and 3, that the NIAs have increased the extent of land under environmental stewardship, is at best only partially valid. Instead, NIA activity has focused on improving the quality of environmental stewardship – improving targeting (spatially and in terms of types of options) and delivery.

22 Analysis based on data supplied by Natural England on Environmental Stewardship scheme coverage. Land area of England used for the calculation taken as 13,348,000ha (the total land area above MHW as used in the Lawton report).



Considering the nature of the data, the challenges of detecting changes in non-ELS agri-environment option applications that can be attributed to the NIAs are a result of the ‘signal to noise ratio’; meaning that there are multiple factors which are influencing the observed trends making it very difficult to detect the changes in the number of non-ELS agri-environment option applications. Examples of these confounding factors are presented in Section 4 under the limitations for Approaches 2 and 3. These factors include:

Diversity of the NIAs - The NIAs are a highly diverse group of partnerships whose ecology, priorities, staffing expertise, previous use of environmental stewardship and size vary substantially. Some aspects are controlled to a degree by Approach 3 that seeks to make comparison within similar landscape types, but even here landscape types are themselves aggregations of a variety of sub-landscapes so the comparison within and without NIAs will not necessarily be a close one.

The scale of the NIAs - Data provided through the online tool suggests that across the NIAs 12.4% of the priority habitats (not total land) across the NIAs are subject to management by the NIA partnerships. Therefore it is likely that within any one NIA most of the changes in environmental stewardship activity are not on land managed by the NIAs.

Previous activities within NIAs - 10 of the 12 NIAs were based on existing partnerships. These partnerships were actively using environmental stewardship to meet their pre-NIA objectives. Many of the NIA partnerships had past landscape scale actions (e.g. protected area strategies, HLS target areas) and overlapping (temporal and spatial) landscape initiatives.

Agri-environment policy – there were a number of policy changes over the timespan of the data. The most significant of these is likely to be the end of the 2007 – 2013 funding initiative resulting in few options being available in 2014 (and a push for final applications in the penultimate year of 2013, as seen). The pattern of renewal from 2005 to 2010 when options were migrating from the classic schemes (for which we have no data) to environmental stewardship was probably driving much of the increase over this time period.

Summary

The NIA initiative appears to be causing relatively marginal changes in the total number of non-ELS agri-environment option applications and related holdings. It is apparent that it is not possible to detect such marginal changes at the landscape scale. This is because of the signal to noise ratio, meaning that the data is responding to lots of confounding factors (noise), including changes to policy, and redundancies and time lags within the data. At the same time the NIAs are directly affecting only a small proportion of agri-environment schemes across all NIAs and are not prioritising increases in the number of options, rather they have sought to improve targeting and delivery (signal).

The qualitative data suggest that the NIAs have provided greater coordination of option types; improved spatial targeting; and combining NIA grants with agri-environment funds to improve quality of delivery.

3.5 Social and economic

Social and economic benefits were a challenging area for partners to assess. Nonetheless, compared to what would have occurred without the NIA initiative, partners in the survey believed there have been improvements in community relations (75% of partners noted improvement) and aesthetic and cultural quality (69% of partners noted improvement).

“Community involvement has been a key element of securing a future for the region. Without public knowledge and support of the special features of the area future work will always be limited, we have carried out extensive public dialogue and local working to understand the thoughts and feelings of the local population and in order that they understand what our aims and objectives are.” (NIA 3)



“Loads of 'community' work and better understanding of 'Lawton' agenda as a result.” (NIA 7)

Views on improvements in working with schools varied significantly depending on the NIAs. Whereas eight NIAs noted improvements, four noted No difference as their modal answer. This can largely be explained by the difference in priorities and objectives of the NIAs.

Three out of four NIAs that noted no improvements had not included any kind of activities to engage with schools in their Funding Agreements. The fourth NIA had a small budget dedicated to school visits which might explain the partner survey result.

Perceived improvements in health outcomes of communities also varied, but depending on the type of respondent. Community organisations, perhaps acting as facilitators enjoying a more direct involvement with the community, were more likely to note an improvement. Whether the NIAs had affected economic development had a mixed assessment, with 39% of partners perceiving ‘no difference’ (Figure 10).

Overall, the partnership chairs felt that the NIA initiative and related funding enabled people to work across traditional divides (referred to as silos) and provided seed funding for projects that would have been unlikely to be implemented otherwise.

Figure 10: Social and economic benefits from the establishment of the NIA initiative

Source: NIA Partner Survey 2015

Note: Asterisk (*) indicates there was significant variation in responses (refer to Appendix 3)

Base: 103 respondents

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Health outcomes of communities*

Working with schools*

Economic development

Aesthetic and cultural quality

Community relations / networks

Much improved Improved No different Don't know



4. Methodological Analysis and Implications

4.1 Methodological Limitations

4.1.1. Limitations of Approach 1

The survey had self-selection bias as partners chose whether or not to respond. The characteristics of respondents in the survey (Figure 11) reveal an uneven distribution of participants per stakeholder category, with NGOs, Government agencies (Natural England / Environment Agency / Forestry Commission / Defra) and Local government/Planning authorities forming the majority of participants, while private companies and farmers/landowners are less represented. This however, to a great extent, reflects the makeup of the NIA partnerships, according to baseline information23.

Figure 11: Characteristics of NIA partner survey respondents


The experiences of different partners reflects the diversity of the NIAs, but also the range of roles different partners had within their NIA; some were heavily involved, others less so. This meant it was likely that participants’ perceptions of the ‘impact’ of the NIAs depended on their involvement and the objectives of their NIA.

In recognition of this, and as described in the methodology section, the research team employed non-parametric statistical analysis to account for variability in responses, testing for three main variables.

Selection bias was inherent to national level interviews. 12 stakeholders were approached and seven agreed to talk. This element of Approach 1 was never intended to be ‘statistically representative’. Rather, the intention was to interview representatives of national organisations, agencies and NGOs with an interest in landscape scale conservation, to understand what the effects, if any, of the NIAs had been from a strategic perspective, and to be able to compare with views of those involved directly in specific NIAs (partnership chairs and partners). Interviews were held with representatives of environment NGOs, Government agencies and farming NGOs. It was felt that this reflected a sufficient range of opinion.

11 of the 12 partnership chairs were interviewed. It is possible that the interviewees presented a biased and positive image of the NIAs to increase the perceived benefits of their work, but in reality their responses appear balanced with positive and negative reflections being presented. The inclusion of national stakeholders provided an alternative, more neutral, viewpoint compared with the partnership chairs that allowed account to be taken of any such bias.

23 Source: analysis based on collated NIA information spreadsheet “NIA All Details FINAL July 2013” as provided by Natural England.

6%

18%

19% 35%

6%

3% 13%

Community and social enterpriseLocal government / Planning authoritiesGovernment agenciesNGOsPrivate companyFarmerOther / Not answered



A related limitation to Approach 1 was the reliance on individuals’ memories and their perceptions about the difference the NIA had made. This was mitigated to some extent by triangulation with different stakeholders and data from the online tool.

Due to the different roles of each stakeholder the questions and information received were not necessarily consistent across all research methods. Therefore there were cases where the triangulation of results for Approach 1 included only two of the research methods described in the methodology section as there were some themes that were not covered by both the interviews and the survey.

4.1.2. Limitations of Approaches 2 and 3

The nature of the data, the NIAs, and agri-environment policy means that there are some important limitations to the results of Approaches 2 and 3 and the ability to draw clear conclusions from them. Both of these approaches were intended to be experimental so that lessons could be learned for future application. Limitations include:

1. The dataset includes every environmental stewardship application (complete / incomplete, successful / unsuccessful) made between 2005 and 2014. This means that the data includes an unknowable number of failed applications and duplicates.

2. The data is a proxy. It is assumed that the number of applications for non-ELS agri-environment options reflects the number of applications for successful environment stewardship funding and related biodiversity improvements. There are some features of the data which reduce the value of applications as a proxy:

o The data shows the year that an application is made, this does not necessarily relate to when the environmental stewardship activity / on the ground works are undertaken. This reduces the value of the data as a proxy for activity.

o The duration of the option (i.e. the time from the application being made to work ending and the application becoming inactive) differs depending on its type, meaning that the trends in applications for options will differ depending on the types of options available within an area and the types of options applied for.

o It is not possible to differentiate between options within the NIA area that are the result of the NIA partnership and those that are not.

o The data processing has used agreement points rather than the area files for the agreements, but there is no specific relationship between the agreements holding area and the area actually affected by the options – especially where an individual holding may have areas both inside and outside the NIA.

3. There are a large number of confounding factors within agri-environment policy which are likely to be partly responsible for the trends observed in the data, examples of these include:24

o New funding and programming rounds running between 2007-13 and 2014-2020. The low number of applications in 2014 is likely due in part to the fact that it was the interim period between Rural Development Programme (RDP) programming rounds, so there were few options still open to which applications could be made.

o The closure of set-aside policy in 2008 reduced the total number of available options.

o In response to ‘Making Environmental Stewardship More Effective’ (MESME) and

24 The M&E project team discussed preliminary results with a agri-environment monitoring and evaluation expert from Natural England. This discussion informed this section.



the Enhanced Training and Information Initiative (ETIP) 25, certain previously non-ELS options were made available to ELS and taken out of higher level stewardship. This had the effect of reducing the total number of option types available from 2010.

o The expiry of options held under ‘classic scheme’, also referred to as Countryside Stewardship, from 2005 is responsible for some of the increases in non-ELS agri-environment options up to 2010 (which was the year at which all classic scheme options were up for renewal). For instance, the data show that in 2010 there was a big spike in applications for England except the NIAs.

o Options targeted under Environmentally Sensitive Areas (ESA) were also expiring over this time period and being replaced by environmental stewardship.

o The Campaign for the Farmed Environment (CFE)26 was active from 2009 and may have affected the distribution of options.

o The number of these confounding factors substantially reduces the efficacy of using a trajectory approach as the results for any given year are likely to be responding to factors that are outside the control of the NIAs.

4. Options that were inside an NIA area, but not within a holding polygon were excluded from the analysis. This was necessary to: allow for the analysis of holdings; provide greater attribution of options to the NIA; and, greatly simplified data analysis. It was calculated that for any given year this affected less than 1% of the records.

5. The analysis used point-based data rather than the extents of the agreements and did not disaggregate on the individual options (although within the context of a multi-objective NIA it may be justifiable to include all HLS (and equivalent) options).

6. The dataset does not control for other co-ordination activities operating over part of England through past and existing non-NIA partnership initiatives (e.g. Landscape Partnerships projects, RSPB Futurescapes, Butterfly Conservation Areas etc) where some of the objectives are the same as those reflected in the NIA initiative. Some of these other initiatives overlap the NIAs.

4.2 Reflections

This section reflects on Approaches 1 to 3, what worked well and less well and what learning there is for future evaluations of interventions in this area (nature conservation generally and partnership based landscape scale conservation specifically).

4.2.1. What worked well?

Approach 1

Approach 1 provided evidence that improved understanding of the NIAs, the difference they had made and mechanisms for improved delivery.

The logic model including the attribution routes supported the development of the questions used within Approach 1. The logic model was effective in identifying which questions should be prioritised. This allowed the survey and the interviews to be kept short.

Working with the NIA project officers to share the survey is felt to have increased the response rate as partners were more familiar and comfortable with the officers as they had worked together for some time.

25 Due to the migration to gov.uk no information detailing these policies could be found online. 26 https://www.gov.uk/government/statistics/campaign-for-the-farmed-environment

https://www.gov.uk/government/statistics/campaign-for-the-farmed-environment



The non-parametric statistical analysis undertaken on the survey meant that it was possible to reflect more robustly on the results across the NIA initiative whilst noting significant differences between NIAs and respondent (based on their sector and level of involvement).

The response rate to the survey (46%) was good; this suggests that working with project officers and targeted follow ups was effective.

Approach 2 and 3

Comparing the NIAs to the national trends provided useful context and allowed for a greater understanding of the effect of changes in national agri-environment policy.

The NCA and ALT comparators are considered to be a viable way of contextualising the trajectory results.

All approaches

The triangulation within and across the three approaches allowed for a greater understanding of the issues and where results from different NIAs and stakeholders groups agreed and disagreed.

The output data provided through the online tool, in combination with the evidence from Approach 1 meant that the results from Approaches 2 and 3 could be better understood.

It has been possible to provide a qualitative understanding of the counterfactual and to identify, test and evidence attribution routes.

4.2.2. What worked less well?

Approach 1

Although the response rate was good some stakeholder groups were less well represented than wished for.

Approaches 2 and 3

The first set of environmental stewardship data that was provided (in May 2015) was not fit for purpose as it was a snapshot of live applications in 2014. This was eventually resolved with a whole new dataset.

It would have been ideal to test the initial hypothesis - that NIAs were increasing the number of options and holdings with options - with stakeholders familiar with the NIAs and explore the results. This was not possible as the funded stage of the NIA initiative had finished and would have increased the scope of the experimental work significantly. The results suggest that the hypothesis for Approaches 2 and 3 was only partially valid, although this is a valuable result in itself.

It was assumed that it would be possible to provide an assessment of the difference the NIA initiative made to agri-environment options at the initiative level. This was not the case because of the highly diverse nature of the NIAs and because the action of NIAs was only one among many factors that could have influenced outcomes (signal to noise ratio).

All approaches

It has not been possible to provide a quantified assessment of the counterfactual for the NIA initiative. This is due the limits of the research methodologies, the nature of the data and in particular the lack of granularity / attribution of actions to NIAs and the NIAs’ diversity in context, approaches and outcomes.

4.3 Implications for future monitoring and evaluation

A number of implications for monitoring and evaluation of initiatives similar to the NIAs can be drawn from the results of this work.



4.3.1. The value of stakeholder views

The results and conclusions from Approach 1 illustrate the value of surveying opinions of partners and stakeholders engaged in such a policy intervention. The triangulation of the three sets of qualitative data provided for robust analysis of what proved to be very rich data sources.

The interviews and survey allowed for the logic model and attribution routes to be tested and ultimately validated.

4.3.2. The environmental stewardship data is potentially useful

The environmental stewardship data provided by Natural England is extensive and has potential value for use in evaluating counterfactuals, subject to a number of conditions/assumptions. It includes a number of features that could be used in future M&E projects, summarised in Table 2.

Table 2: Potentially relevant features from the environments stewardship data set

Temporal features Spatial Features Extent features Option Type

Features Administrative

features Including:

When work related to a specific option starts / ends

When an application was made

Including:

Which holding an option is in.

Coordinates of an option

Which National Character Area / region / county / town an option is in

Including:

The length / area / number of trees

Including:

What sort of option

Cost of the option

Including:

Whether an option is active / closed

How the application was made

Why it was closed (if closed)

Which of these features are considered relevant will depend on the sort of intervention that is being evaluated. For instance, the results from Approaches 2 and 3 (see Section 3) suggest that the main effect of the NIAs was on improving coordination of the types and spatial arrangement of options; any future work might wish to consider the features relevant to option type. This will not necessarily be straightforward. For example assessing changes in spatial features would require a very good understanding of conditions on the ground (relating to habitat quality, connectivity and other land uses) and is likely to be very resource intensive (requiring ecological or detailed modelling and/or surveying) and challenging.

Using the features relevant to option type it might be possible to assess changes in the types of options over time. It is suggested that any such assessment would have to be NIA (or other project) specific given the particular priorities and approaches of different NIAs. Considering the signal to noise challenge it would also be necessary to develop a detailed understanding of what the NIAs (or other projects) were seeking to do, i.e. which specific option types were they seeking to affect. Without this understanding or a substantial change in the amount of holdings being affected by the NIA it is likely that any changes would be undetected in the background noise.

In theory, this approach could have been deployed for the NIAs, but for it to be effective the following assumptions would need to have been in place:

1. NIAs had a suitably clear prioritisation of option types.

2. NIAs had implemented their prioritisation of option types

3. The change in option types was of a sufficiently large scale to be observable against changes in the NIA which were not the result of NIA activity.

4. There were no major changes in agri-environment policy including available options.

This research suggests that none (or few) of these assumptions were true over the grant-funded NIA initiative period and it is likely that a quantitative approach, using the available dataset, cannot provide a realistic or meaningful counterfactual at the initiative level.



Such an approach could be tested on other landscape scale initiatives in the future where non-ELS agri-environment options are used and two alternative hypotheses could be tested:

An increased uptake of specific types of options, e.g. landscape-scale options, is to be expected as a result of the initiative.

An increased option density per hectare is to be expected as a result of the initiative.

This could be possible but the same assumptions would need to hold true, and in practice it is likely that there are many more factors that cannot be controlled that would make it difficult to draw meaningful conclusions as to the counterfactual in such situations. The timescale over which the intervention is evaluated is also relevant – a longer time-scale (than three years) is likely to increase the signal to noise ratio and/or the degree of confidence in any trends that might be observed.

4.3.3. M&E must either reduce or account for flexibility

As intended by Defra and Natural England the NIAs were a highly context specific and variable initiative and, as shown in Section 3, flexibility was one of their strengths. This does present challenges for M&E27, to which there are two apparent responses:

1. Recognising that flexibility is inherent and disaggregating the analysis to the project level; or

2. Creating rigid objectives that can be aggregated across the initiative through greater comparability – ideally based on existing national datasets to allow for comparative analysis.

Disaggregating the results would allow for a more detailed consideration of the counterfactual at the project level. This could be supported initially by qualitative research, as under Approach 1, which would allow for the identification and testing of project-specific logic models that could then be tested using qualitative data where appropriate.

The alternative to focusing on the counterfactual for specific projects would be setting clear and rigid initiative objectives which are linked to specific existing datasets. For instance an initiative could specify that increasing the extent of area with non-ELS agri-environment options (or a specific option type or total cost of options) is a priority of the initiative and all related projects. It would then be possible to monitor changes before and after the initiative using existing monitoring initiatives and datasets as well as compared to other areas/nationally. It is recognised that setting fixed initiative level objectives reduces the scope to test a variety of approaches/interventions at the project level. It would however allow comparisons across projects within that initiative in a way that is not possible where each project is designed to be unique.

27 See also Bennett et al, 2015 (CSFF final report).



4.4 Conclusions

The results from Approach 1 show the value of detailed qualitative research as they have provided insights into the counterfactual and the attribution routes whilst also contextualising the output data provided by the online tool.

The results from Approaches 2 and 3 suggest it is not possible to provide an assessment of the counterfactual for non-ELS agri-environment options at the scale of the NIAs initiative; this is due to the signal to noise challenge, but also because the NIAs themselves are highly diverse and as such it is not appropriate to aggregate them to the initiative level. These lessons are consistent with the conclusions drawn in the CSFF scoping study, undertaken as part of the wider NIA evaluation project28.

The following broad lessons for counterfactual evaluations are based on these results:

Use logic model and attribution routes to identify mechanisms for delivery which are unique to your initiative. Use these to prioritise your evidence collection.

Work with stakeholders to refine and ground truth your logic model, attribution routes, hypothesis you wish to test and your results.

Engage with multiple stakeholder types using a range of research methods.

Recognise the diversity of projects and account for this within your analysis - when is it suitable/not suitable to aggregate to the initiative level?

Look for opportunities for triangulation.

28 Bennett, T., Phillips, P., Sheate, W., Eales, R. and Baker, J. (2015), Countryside Stewardship Facilitation Fund Monitoring and Evaluation Scoping Study – Final Report by Collingwood Environmental Planning and GeoData for Defra (July, 2015).



Appendix 1: NIA logic Model and Attribution Routes



The following attribution routes, the ways in which the NIA initiative is understood to have caused impacts, were taken from the logic model:

Shared vision / funding agreement

Developing a shared vision > Assessing local opportunities for restoring and connecting nature > Funding agreement identifying priority actions > Delivery of those actions

What was the effect of developing and delivering the NIA funding agreement

Defra seed funding

Defra funding > Funding attracted by the NIAs from various sources and contributions in-kind > Sustainable financial base for partnership working and continued landscape scale innovations

What has been the effect of the Defra grant funding in terms of securing funding from other sources

Innovation and information exchange

In-house and external skills and capacities > Sharing of experience and expertise between and within the NIAs > Knowledge exchange and Best practice events > Shared learning and dissemination of expertise within and beyond the NIAs > A sense of innovation and optimism

What has been the effect of sharing experiences and information between NIAs

The value of partnerships

Building partnerships and facilitating partnership working > Agreed funding agreements > Well supported and functioning partnerships and Shared learning and dissemination of expertise within and beyond the NIAs > Effective, durable partnership (beyond pilot period)

Compared to what would have happened without the NIA initiative, what has been the effect if working with NIA partners



Appendix 2: NIA Partners' Survey Questionnaire

The following is as the survey was presented online.

Nature Improvement Areas Partners' survey

This survey is being conducted by Collingwood Environmental Planning (CEP) as part of the Monitoring and Evaluation of Nature Improvement Areas (phase 2) research project (WC1061) for the Department for Environment, Food and Rural Affairs (Defra) in collaboration with Natural England. All personal information provided in this survey will be treated as confidential and will not be passed on to any third parties. The reporting of the responses will be in an unnamed format and used only for the purposes of this research, while all responses will be safely deleted after completion of the project.

The objective of this survey is to gather a broad range of insights on the difference the Nature Improvement Area (NIA) partnerships / programme has made within the NIAs compared to what might have been expected to happen if they had not been established. NIA partners and other NIA stakeholders are being invited to participate in the survey. The results of the survey will supply evidence for the monitoring and evaluation of the outcomes of the NIA partnerships / programme.

Thank you for participating in our survey. Your feedback is important.

Please answer all questions based on your experience from the NIA you are most familiar with.

(*) Indicates the question requires an answer

* 1. Please select the NIA with which you are most familiar

Birmingham and the Black Country

Dark Peak

Dearne Valley


Humberhead Levels

Marlborough Downs

Meres and Mosses of the Marches

Morecambe Bay Limestones and Wetlands

Nene Valley

Northern Devon

South Downs Way Ahead

Wild Purbeck * 2. Compared to what would have occurred without the establishment of the NIA programme,

which of the following, if any, have you experienced and to what effect?

Much

improved Improved

No

different Worse

Much

worse

Don't

know

Not

applicable


http://randd.defra.gov.uk/Default.aspx?Menu=Menu&Module=More&Location=None&ProjectID=18555&FromSearch=Y&Publisher=1&SearchText=WC1061&SortString=ProjectCode&SortOrder=Asc&Paging=10



Much

improved Improved

No

different Worse

Much

worse

Don't

know

Not

applicable

Local support from Natural England

Development of a shared vision for the

area

Coordination of activities across the area

by different organisations

Learning through dissemination of

expertise / experience

Sharing of data and information among

NIA partners

Facilitated the delivery of other

management plans and objectives (e.g.

Water Framework Directive, Green

infrastructure, Shoreline management,

low-carbon economy etc.)

Added value through the use of non-

monetary resources (e.g. volunteers,

equipment etc.)


in the design and delivery of proposed

activity

Other (please specify)

3. Please consider your answers to the previous question (Q2) and rank the three most important

changes you have experienced, by selecting from the drop down lists.

1

2



3

Other (please specify - maximum 200 characters)

* 4. Compared to what would have occurred without the establishment of the NIA programme,

which of the following challenges, if any, have you experienced?


Additional workload

Disagreement / Friction with partners

High expectations from partners, civil society, community etc.

Problems / Interference with the delivery of other management plans and objectives (e.g. Water Framework Directive, Green infrastructure, Shoreline management, low-carbon economy etc.)

Costs disproportionate to funding

Inadequate support from partners

Unforeseen problems in final delivery

None of the aforementioned


5. Please consider your answers to the previous question (Q4) and rank the three key challenges

you have experienced, by selecting from the drop down lists.

1

2

3

Impacts of the NIA programme

The next questions explore the impacts of the NIA programme. Please base your answers on the NIA

you are most familiar with, and assess its impact compared to what would have happened if it had

not been established.



* 6. How would you characterise the impact of the NIA on partnership working?

Much more

effective More effective No different Less effective

Much less

effective Don't know

7. In support of your answer to Q6, please provide any examples or comments on what you

(and/or your organisation) have done differently on partnership working as a result of the NIA.

* 8. How would you characterise the status of the following elements of biodiversity, compared to

what it would have been if the NIA had not been established?

Much

improved Improved No different Worse Much worse Don't know

Species

Habitat

connectivity

Habitat extent

Habitat quality


(and/or your organisation) have done differently as a result of the NIA.

* 10. How would you characterise the impact of the NIA on local communities and social and

economic benefits, compared to what would have occurred if it had not been established?

Much




Much


Health outcomes

of communities

Economic

development

Working with

schools

Aesthetic and

cultural quality of

the NIA

Community

relations /

networks



* 12. How would you characterise the impact of the NIA on the following elements of ecosystem

services, compared to what would have occurred if it had not been established?

Much


Publicly accessible

rights of way

Water quality

Carbon storage

and sequestration

Access and quality

of green spaces



Much


Support to

pollinators

Woodland

products

Flood

management

Sustainable

agriculture



* 14. Compared with what is likely to have been delivered without the NIA, to what extent do you

agree or disagree with each of the following statements?

Strongly

agree Agree

Neither

agree nor

disagree

Disagree Strongly

disagree

Don't

know

There was improved

information and learning

exchange between partners

Collaborative working among

NIA partners improved

delivery

The NIA has created more,

bigger and better places for

wildlife

The NIA has created a sense

of innovation and optimism

about landscape scale

conservation



Strongly

agree Agree

Neither

agree nor

disagree

Disagree Strongly

disagree

Don't

know

The development of a shared

vision for the NIA led to

improved delivery

The NIA has enhanced the

benefits that nature provides

for people

Defra grant funding helped

the NIA partners access

additional funding

15. Do you have any other comment(s) about the NIA programme?

* 16. How would you describe your involvement in the NIA over the three year government grant

funding period?

Day to day involvement

Frequently involved (e.g. about once a week)

Occasional involvement (e.g. about once a month)

Involved rarely (e.g. a few times a year)


* 17. Which of the following best describes the organisation you represent?

Community and social enterprise

Local government / Planning authorities (including National Parks & Areas of Outstanding Natural Beauty)

Academia and research

Natural England / Environment Agency / Forestry Commission / Defra

Environmental Non Government Organisation (NGO)

Private company (e.g. Water company, Consultancy etc.)




18. What is your role in the organisation? (maximum 100 characters)



Appendix 3: Summary of Non-Parametric Statistical Results from the Survey

The following table presents a summary of all the test results. Where there is no significant variation we note ‘No effect’ of the explanatory variable on the responses. Where there is significant variation the nature of that is indicated. Please note that, though this table includes all variation as it emerges from the statistical testing, in the main report we only report on ‘interesting’ variation which is also highlighted red in this table. Overall all tests were performed at a 0.95 significance level indicating that there is, at least, a 95% confidence level in the results reported. For the variation highlighted as ‘interesting’, the detailed Kruskal-Wallis statistical testing results and significance levels have been included in footnotes.

Highest % responses Effect of NIA (12 individual NIAs)

Effect of NIA size (S / M / L)

Effect of respondent type (7 categories)

29

Q2

Funding Access ‘Improved’ (55%) and another 29% ‘Much Improved’

No effect No effect No effect

Natural England support

‘Improved’ (57%) and 10% ‘Much Improved’, but another 25% ‘No difference’

10 Improved 2 No Difference

30

Yes31

No effect


‘Improved’ (48%) and another 46% ‘Much Improved’




5 Much improved, 7 Improved (at least)

Yes No effect



2 Much Improved 10 Improved

Yes No effect

Sharing of data ‘Improved’ (58%) and another 26% ‘Much Improved’


Facilitated delivery

‘Improved’ (49%), 17% ‘Much Improved’ and 23% ‘No difference’


Added value ‘Improved’ (48%) and another 35% ‘Much Improved’

11 Improved 1 No difference

32

Yes No effect


‘Improved’ (44%), 21% ‘Much Improved’, 26% ‘No difference’


Q4


Experienced by 43% of respondents


Additional Experienced by 48% No effect No effect No effect

29 Includes a category for ‘Farmer’ and ‘Other’, added subsequently to the survey to accommodate the analysis 30 Kruskal-Wallis chi-squared = 25.3186, Degrees of freedom = 11, Probability value = 0.008186 31 Kruskal-Wallis chi-squared = 6.2554, Degrees of freedom = 2, Probability value = 0.04382 32 Kruskal-Wallis chi-squared = 20.3274, Degrees of freedom = 11, Probability value = 0.04105






29

workload of respondents

High expectations Experienced by 22% of respondents


Friction with partners


No effect No effect Yes33

Interference Experienced by 12% of respondents


Disproportionate costs


No effect Yes34

No effect

Inadequate support



Unforeseen problems



Q6 Partnership working

‘More effective’ (57%) and another 36% ‘Much more effective’


Q8

Habitat extent ‘Improved’ (66%) and another 21% ‘Much Improved’


Species ‘Improved’ (58%), 18% ‘Don’t know’, 11% ‘No difference’


Habitat connectivity



Habitat Quality ‘Improved’ (63%) and another 25% ‘Much Improved’


Q10

Health outcomes

‘Improved’ (35%), 31% ‘No difference’ and another 30% ‘Don’t know’

No effect No effect

Local Gov., private org. & NGO ‘No difference’, Community org. median improvement, Farmers & gov. agencies ‘Don’t know’

35

Working with schools

‘Improved’ (37%), ‘No difference’ (27%) and 24% ‘Don’t know’

1 Much Improved 7 Improved 4 No difference

36

Yes37

No effect

Economic development

‘No difference’ (39%), ‘Improved’ (33%), ‘Don’t Know’ (25%)


Aesthetic and cultural quality

‘Improved’ (55%), 14% ‘Much


33 Kruskal-Wallis chi-squared = 12.1409, Degrees of freedom = 6, Probability value = 0.0589 34 Kruskal-Wallis chi-squared = 11.7984, Degrees of freedom = 2, Probability value = 0.002742 35 Kruskal-Wallis chi-squared = 13.5857, Degrees of freedom = 6 Probability value = 0.03462 36 Kruskal-Wallis chi-squared = 27.1315, Degrees of freedom = 11, Probability value = 0.00439 37 Kruskal-Wallis chi-squared = 15.3307, Degrees of freedom = 2, Probability value = 0.0004688






29

Improved’, 17% ‘No difference’ and 14% ‘Don’t know’

Community relations/networks

‘Improved’ (56%), 19% ‘Much Improved, 15% ‘Don’t know’


Q12

Publicly accessible rights of way

‘Improved’ (40%) and another 8% ‘Much Improved’, ‘No difference’ (32%), 21% ‘Don’t know’

1 Much improved 5 Improved 1 Slightly improved 5 No difference

38

Yes No effect

Support to pollinators

‘Improved’ (54%) and another 11% ‘Much Improved’, 21% ‘No difference’, 14% ‘Don’t know’


Sustainable agriculture

‘Improved’ (52%) (and 7% ‘Much Improved’), ‘Don’t know’ (21%) and ‘No difference’ (20%)

9 Improved 2 Slightly improved 1 No difference

39

Yes No effect

Woodland products

‘Improved’ (40%) (and 7% ‘Much Improved’), 33% ‘No difference’ and 21% ‘Don’t know’


40

Yes No effect

Water Quality

‘Improved’ (60%) and 11% ‘Much Improved’, 18% ‘No difference’


41

Yes No effect

CSS

‘Improved’ (46%) (and 8% ‘Much Improved’), ‘Don’t know’ (25%), 22% ‘No difference’

8 Improved (at least) 1 Slightly improved 3 No difference

42

Yes No effect

Access and Quality of green spaces

‘Improved’ (63%) and 6% ‘Much Improved’), ‘No difference’ (15%) ‘Don’t know’ (16%)


Flood management

‘Improved’ (45%) and 12% ‘Much Improved’, ‘No difference’ (26%), 18% ‘Don’t know’

1 Much Improved 10 Improved 1 No difference

43

Yes No effect

Q14 Defra grant ‘Agree’ (42%), 39% No effect No effect No effect

38 Kruskal-Wallis chi-squared = 29.8603, Degrees of freedom = 11, Probability value = 0.001667 39 Kruskal-Wallis chi-squared = 21.7521, Degrees of freedom = 11, Probability value = 0.02637 40 Kruskal-Wallis chi-squared = 41.2245, Degrees of freedom = 11, Probability value = 2.205e-05 (e-05 means ‘times 10-5’, which is close to zero) 41 Kruskal-Wallis chi-squared = 27.5124, Degrees of freedom = 11, Probability value = 0.003843 42 Kruskal-Wallis chi-squared = 34.3371, Degrees of freedom = 11, Probability value = 0.0003185 43 Kruskal-Wallis chi-squared = 28.2404, Degrees of freedom = 11, Probability value = 0.002974






29

funding helped the NIA partners access additional funding

‘Strongly agree’, 4% ‘Disagree’

The development of a shared vision for the NIA led to improved delivery

‘Strongly agree’ (48%), ‘Agree’ (39%), 9% ‘Neither agree nor disagree’


Collaborative working among NIA partners improved delivery

‘Strongly agree’ (51%), ‘Agree’ (40%)


There was improved information and learning exchange between partners

‘Agree’ (50%), ‘Strongly agree’ (38%), 8% ‘Neither agree nor disagree’


The NIA has created a sense of innovation and optimism about landscape scale conservation

‘Agree’ (50%), ‘Strongly agree’(36%), 12% ‘Neither agree nor disagree’


The NIA has created more, bigger and better places for wildlife

‘Strongly agree’(45%), ‘Agree’ (44%), 8% ‘Neither agree nor disagree’

5 Strongly agreed 7 Agreed (at least)

44

Yes No effect

The NIA has enhanced the benefits that nature provides for people

‘Agree’ (59%), ‘Strongly agree’ (26%), 11% ‘Neither agree nor disagree’


Q16

How would you describe your involvement in the NIA

‘Frequent involvement’ (33%), ‘Occasional involvement’ (30%), ‘Day to day involvement’ (26%)


44 Kruskal-Wallis chi-squared = 21.9506, Degrees of freedom = 11, Probability value = 0.02476



Appendix 4: Partnership Chairs Interview Schedule

Interview questions

1. What is your current role and organisation?

2. What was your role before starting as Partnership Chair?

3. What have been your responsibilities as Partnership Chair?

4. IF THE PARTNERSHIP IS BASED ON AN EXISTING PARTNERSHIP – How did being awarded NIA status and funding change the partnership, if at all?

5. IF THE PARTNERSHIP IS NEW – How was this group of NIA partners brought together? And why?

6. IF THE PARTNERSHIP IS NEW – What changes, if any, have you noted in how stakeholders work together in the NIA, compared to before the NIA partnership?

7. What has been the effect of the Defra grant funding?

8. What changes have you noted in NIA partners’ mobilisation of resources compared to what would have happened without the NIA?

9. What was the impact of developing and agreeing a shared vision and NIA funding agreement objectives?

10. In your experience how has sharing information and learning about landscape scale initiatives (including with other NIAs) changed since becoming a NIA?

11. Within your NIA what do you think have been the most notable effects, either positive or negative of the NIA programme, if any?

12. Compared to what you think would have happened without the NIA programme, how has your NIA partnership affected Biodiversity?

o Please provide specific examples if possible

13. Compared to what you think would have happened without the NIA programme, how has your NIA partnership affected Social and economic aspects)?


14. Compared to what you think would have happened without the NIA programme, how has your NIA partnership affected Ecosystem services


15. What has worked well within your NIA?

16. What has worked less well / what would you do differently?

17. What has surprised you most about implementation of the NIA programme?

18. Is there anything else you wish to add?



Appendix 5: National Stakeholders Interview Schedule

Interview questions

1. In your view what has been the overall effect of the NIA programme on landscape-scale delivery

2. To what extent, if at all, has the NIA programme changed attitudes towards landscape-scale conservation?

3. To what extent, if at all, has the NIA programme changed the capacity of partners to deliver landscape scale conservation?

4. To what extent, if at all, has the NIA programme had any effect on the sharing of information and learning on landscape-scale conservation?

5. To what extent, if at all, has the NIA programme affected the frequency, nature and quality of partnership working on landscape-scale conservation?

6. To what extent, if at all, has the NIA programme affected the processes and priorities of your organisation?

7. In what ways has your organisation contributed to the NIA programme?

8. In your view how do NIA outcomes differ from previous approaches to landscape-scale conservation?

9. In your view what has worked well about the NIA programme?

10. In your view what has worked less well?

11. What has surprised you most about implementation of the NIA programme?

12. Is there anything else you wish to add?



Appendix 6: Approaches 2 and 3 - Detailed Methodology

Preparing the data

Understanding the spatial relationship of the data

A ‘holding’ has a single unique ID code (AGREF) and is composed of one or more polygons (referred to as multi-part). The holding can fall entirely within an NIA, entirely outside an NIA or intersect part of the NIA. A further aspect of the intersection is that the constituent polygons of a holding may fall entirely within, outside or intersect the NIA boundary.

The option points typically intersect holding polygons, but there are cases where this is not the case and this can happen inside or outside the NIA. Option points are stacked on top of each other. For example a holding may be composed of 20 polygons and have 100 options associated with it, those 100 options points may visually look like 3 points in 3 of the 20 polygons but will have the 100 options distributed across those 3 identical point locations. So it is important to understand that not all polygons of a holding have an option point.

This multi-part nature and stacked points introduces illogical situations. For example a holding composed of 3 polygons, two of which fall entirely outside the NIA may have all its associated option points in one of these outside polygons. Meaning a straight clip (a ‘cookie cut’) of the holdings within an NIA could leave you a holding polygon with apparently no options.

Data had been provided for years 2005 to 2014, thus each years’ worth of holding data had an associated years’ worth of option points.

Preparation of dataset

1. The latest version of the NIA boundaries was downloaded from the EA Geostore website (http://www.geostore.com/environment-agency/).

2. The original data supply by Natural England in August 201545 was a 3GB CSV which when imported into ArcMap created a table with 7,048,210 rows.

3. A new tabular dataset was created from this which excluded all rows where the field AGD_SCHEME = "Entry Level Stewardship", this created a table of 2,847,285 rows. The remaining option types were:

a. Entry Level plus Higher Level Stewardship

b. Higher Level Stewardship

c. Organic Entry Level Stewardship

d. Organic Entry Level plus Higher Level Stewardship

This was done on the basis that ELS is not targeted at biodiversity and was not relevant to this analysis.

4. At this stage the data included rows where there were no XY coordinates. These were records where there is no parcel reference, are either rotational, linear or agreement level options and as such do not have the parcel reference assigned to them within the administration system. A new table was created excluding these to create a final working version of 2,355,673 rows.

45 A dataset originally supplied in May 2015 proved to be unusable for this purpose because it was structured around live applications as of 2014, rather than the date of applications made. A revised data request was therefore made to Natural England.



5. A new field was added called AGD_AGREE_START_YEAR and populated with the year taken from AGD_AGREEMENT_START_DATE. AGD_AGREEMENT_START_DATE was considered the most appropriate temporal filter as this was the earliest possible date that the record would be recorded. The alternative was to use the date that the physical works were projected to start (the feature’s code is “WOP_WORK_START_DATE”). Using this filter would have complicated the data as in some instances work might start as long as five years after the agreement was in place. Using the agreements start date provided the simplest and clearest link between activity within the NIA areas and uptake of options.

6. The new field AGD_AGREE_START_YEAR was used to ‘explode’ the dataset into 10 new spatial point datasets one for each year (2005 - 2014).

Processing of Option data

The following processing steps were conducted in ArcMap 10.3 using model builder to automate the processing.

It was necessary to provide all option data inside and outside NIA boundaries. This was done without regard to the holding polygons and the data was processed (cookie cut) by:

1. Select all option points in the NIAs

2. Export and assign which NIA they fell within

3. Export attributes to dBase format so they could be imported into Excel

4. Invert selection in (1) so all selected points were outside the NIAs

5. Export attributes to dBase format so they could be imported into Excel

6. Repeat steps 1 to 5 for each years’ worth of data

7. The dBase files were manually imported into Excel ready for analysis.

The complex spatial relationship of holding polygons to option points may have introduced bias difficult to explain. Thus the spatial test of points inside the NIA did not attempt to filter points that were inside the NIA but outside a holding polygon.

Analysis: Approach 2 trajectory

Using the cleaned and cut data a spreadsheet format was used to compare data within (individually) and across (collectively) NIA partnerships. This enabled the generation of graphs illustrating trends collectively and individually in the years preceding and during the NIA period.

This analysis was undertaken to generate graphs for:

The number of non-ELS options within an individual NIA area per year from FY 2005-06 to 2011 – 12 (pre-NIA) and from 2012-13 to 2014-15 (NIA).

The number of holdings with non-ELS options across the NIA areas from FY 2005-06 to 2011 – 12 (pre-NIA) and from 2012-13 to 2014-15 (NIA).

Undertaking an analysis of both was considered necessary to understand whether the activity in the NIA areas was on increasing the number of options on holdings that were already using non-ELS agri-environment options or on bringing in holdings that were not historically using non-ELS agri-environment options.

The nature of the data is that each individual record (column) represents a single agri-environment option application. On this basis the number of non-ELS options was calculated using COUNTIF function to filter out the records for each NIA per year. For example:

[=COUNTIF(TITLE OF THE WORKBOOK$COLUMN$1:$ COLUMN$99999, "NIA NAME")]

The number of holdings with applications for non-ELS options can be understood by counting the



number of unique holdings references (feature code is “AGD_AGREEMENT_REFERENCE”). For each year this was calculated by using advanced filter on the AGD_AGREEMENT_REFERENCE column, copying only the unique records to an empty column and then using the “=ROWS(RANGE)” function to calculate the number of unique holdings references.

A trend line including a three year forecast was then applied to the results in the pre-NIA. This was done using Excel’s linear trend line. This produced a theoretical trajectory for total number of non-ELS agri-environment option applications and related holdings in the pre-NIA period against which the results of the NIA period could be compared.

Due to the very low levels or options in 2005 records for this year were excluded as it was disproportionately affecting the trend line for the pre-NIA period.

The trajectory analysis was supplemented by the data for England excluding the NIAs. This was done for the number of non-ELS option applications and the number of holdings with such applications. The results were added to the NIA graphs using a second axis.

Analysis: Approach 3 matched comparisons analysis

The sample approach to comparative assessment used the same input data as Approach 2.

The national data comparison may not necessarily relate closely to the agricultural / landscape setting within which the NIAs were selected and therefore not reflect agricultural options on similar landscape features. Approach 3 took this further by seeking to control for certain characteristics of the comparator areas. NIAs area selection may have been influenced by past experience of collaborative and co-operative actions to support landscape scale multi-objective actions, and often through the support to uptake of HLS options. The challenge was to undertake a review of the trends of uptake of relevant options across the range of potential AES options and across areas that are equivalent in landscape / land cover terms.

Data analysis

Two landscape level GIS datasets were acquired / generated for this analysis: National Character Area (NCA) and Agricultural Landscape Types (ALT) as a combination of NCA areas with equivalent agricultural type characteristics. These subdivide the country into 159 National Character Areas, and ALT dataset was generated, built from recoding NCA polygons into the 9 ALT classes.

A challenge with this approach is that the NIAs do not match boundaries of a single NCA or ALT; in many circumstances the NIAs overlap into multiple NCAs or ALTs. However, given the nature of the NIA selection process and targeting of actions based on the principles of landscape scale conservation there is often a close correspondence between NIA boundary and NCA/ALT boundaries with NIAs only intersecting a few NCAs / ALTs (Table 3). NCAs are smaller discrete classes than ALTs, given that the latter aggregate NCA areas (Figure 12).



Figure 12: National Character Areas (NE 201546

) and Agricultural Land Types (Swanwick et al., 2007)

Note: The ‘other’ class in the ALTs represents special areas largely outside the agricultural setting e.g. New Forest, Breckland, Forest of Dean and are outside the NIAs although they may have HLS.

The aim was to represent similar landscape classes to those within the NIA rather than those areas with which the NIA interacts marginally.

Some of these intersections are ‘sliver’ polygons created by the intersection process and are a result of data captured at different scales and accuracy. In order to filter out these smaller polygons, to leave only the dominant classes of NCA/ALT the percentage area of an NIA for all intersecting ALTs has been calculated and the lower quartile value identified as a basis for excluding overlapping areas.

In the case of ALTs where the areas of the NIA intersecting with the ALT were greater than 6.6% of the area of the NIA the ALT has been included; below that it was excluded. The same process was run on NCA data and the lower quartile value 0.315% identified, reflecting the smaller size of the NCA areas.

The data are presented as normalised by area to provide a density figure (agreements by year per km2). The analysis of the within NIA and outside NIA for each individual NIA excludes options that fall within other NIAs.

Table 3: NIA intersection with National Character Areas and Agricultural Land Types

NIA Area NCA overlaps ALT overlaps

Birmingham and the Black Country

62,470 Cannock Chase and Cank Wood Mid Severn Sandstone Plateau Arden

Western dairy and mixed agriculture

Dark Peak 28,540 Southern Pennines Dark Peak White Peak

Upland Agricultural Landscapes

Dearne Valley 16,514 Nottinghamshire, Derbyshire and Yorkshire Coalfield Southern Magnesian Limestone

Upland fringe dairy and stock rearing Agricultural Landscape Types Chalk and limestone mixed arable

46

http://data.gov.uk/dataset/national-character-areas



NIA Area NCA overlaps ALT overlaps

landscapes


54,337 Greater Thames Estuary North Kent Plain

South eastern wooded and mixed agricultural landscapes Coastal areas

Humberhead levels

49,869 Humberhead Levels Humber Estuary

Eastern arable agricultural landscapes

Marlborough Downs

10,398 Berkshire and Marlborough Downs Chalk and limestone mixed arable landscapes

Meres and Moses

40,153 Shropshire, Cheshire and Staffordshire Plain Cheshire Sandstone Ridge

Western dairy and mixed agriculture

Morecambe Bay

49,139 Lancashire and Amounderness Plain , Morecambe Coast and Lune Estuary, Morecambe Bay Limestones, South Cumbria Low Fells

Upland fringe dairy and stock rearing Coastal areas Western dairy and mixed agriculture

Nene Valley 41,479 Nene Valley Chalk and limestone mixed arable landscapes Western dairy and mixed agriculture

North Devon 72,560 The Culm Upland fringe dairy and stock rearing

South Downs 41,520 South Downs Hampshire Downs

Chalk and limestone mixed arable landscapes

Wild Purbeck 46,165 Dorset Heaths South Purbeck

South eastern wooded and mixed agricultural landscapes Coastal areas

As a comparison the density outside an NIA for the same ALT was computed. These were counts of options outside the individual NIA but inside the same ALT and excluding any option points that fell within other NIAs. In this case, if an NIA intersects the same ALT or NCA the graph will be the same.

For each year (2006-2014) of option points the density (km2) of points within the dominant NCAs was calculated inside and outside the NIA.

For each year (2006-2014) the option points density (number of option points per km2) within the dominant ALTs and NCA were plotted (see results section).

A Mann-Whitney U test was undertaken using MiniTab to compare the results of the NIA with their comparator NCA and ALT areas (see results section).



Appendix 7: Summary of Results from Approach 1




Biodiversity The majority of respondents considered that biodiversity benefits had been delivered over and above what would have happened anyway.

The majority of partnership chairs considered biodiversity benefits to have been delivered over and above what would have happened anyway.

Some national stakeholders felt that biodiversity activities funded through environmental stewardship grants might have happened anyway, but most national stakeholders felt that NIAs sped up delivery and improved coordination of these activities.

Ecosystem services

Significant variation in responses about the extent that the NIA initiative has led to additional ecosystem service outcomes across NIAs depending on objectives and nature of NIAs.

The majority of partnership chairs felt that there was a greater focus on ecosystem service outcomes from habitat management than would have happened otherwise.

Specific benefits noted included flood/water management, woodland products and carbon storage and sequestration.

The majority of national stakeholders felt that the NIAs raised the profile of ecosystem services and some felt that improved coordination between Water Framework Directive (WFD) and biodiversity activities was achieved.

Social and economic wellbeing

Respondents felt that community relations were most improved by the NIA partnerships among these areas of activity.

The majority of partnership chairs felt that the NIA government grant funding enabled projects with broad objectives that would have struggled to get off the ground otherwise.

No views were expressed by national stakeholders.

Partnership working

93% of respondents considered partnership working to be more (57%) or much more (36%) effective than would have happened otherwise.

The majority of partnership chairs felt that funding for staff enabled people to work with and support other partners and challenged silo-thinking.

The majority of national stakeholders felt that the NIA initiative had led to broader and better coordinated partnerships than would otherwise have existed.

Other findings

Narrative comments added to the survey by respondents indicated an overall sense of achievement among partners.

88% of respondents considered NIAs to have contributed to Lawton’s vision, though a three year timescale was deemed too short to achieve large scale and lasting improvements.

A majority of respondents identified improvements in the development of a shared vision and sharing of information and resources.

A majority of respondents expressed that NIA status

The majority of partnership chairs felt that NIAs: provided a forum for bringing partners together around a common vision; and improved awareness of the landscape scale approach within partner organisations.

The majority of partnership chairs felt that the NIA government grant funding and NIA status acted as a catalyst for match funding and galvanising partners. Flexibility of use of funding was seen as critical.

Most partnership chairs felt that three years not long enough to make a real

Some national stakeholders felt that the NIA initiative served to accelerate and broaden the scope of activities that may have happened anyway.

The majority of national stakeholders felt that: the flexibility of funding enabled new types of partnerships; and that committed, enthusiastic partners made a relatively small amount of money go a long way.

Some national stakeholders also felt that the NIAs helped to bring statutory agencies together and improved communication between them.






generated wider stakeholder engagement and had benefits in attracting match funding.

Additional workload and administrative burden were the main challenges expressed by the NIAs.

difference.

Some partnership chairs felt that the NIA government grant helped ‘plug a gap’ left by cuts to statutory agencies and local authorities who might otherwise have funded some of the types of activity completed by NIA partnerships.



Appendix 8: Graphs and Tables for Individual NIAs (Approach 2 trajectory analysis)

This appendix presents the combined result tables for the number of non-ELS agri-environment option applications and related total number of holdings and then the graphs for individual NIAs for total number of option applications only.

Note: BBC is Birmingham and Black Country. DP is Dark Peak. DVGH is Dearne Valley Green Heart. GTM is Greater Thames Marshes. HL is Humberhead Levels. M&M is Meres and Mosses of the Marshes. MB is Morecombe Bay. MD is Marlborough Downs. ND is North Devon. NV is Nene Valley. SD is South Downs Way Ahead. WP is Wild Purbeck.

Table A8.1: Individual NIAs: Total Non-ELS Option Count 2005 – 2014

NIA 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014

BBC - - - 33 35 329 239 292 145 -

DP - - 280 625 538 505 762 620 2,844 421

DVGH - - 247 - 180 50 188 167 87 -

GTM - 139 1,904 287 1,036 687 870 525 1,703 885

HL 2 37 258 402 676 149 452 860 749 96

M&M 215 1,198 235 2,003 481 1,003 2,514 405 1,009 125

MB 107 911 474 663 1,366 1,453 1,567 600 2,366 1,163

MD - 472 197 548 - 1,250 547 64 792 125

ND 409 3,899 2,990 1,214 832 1,717 1,653 2,456 1,915 449

NV 5 561 731 337 655 1,056 654 997 530 35

SD 35 1,144 1,302 1,340 2,454 2,142 1,489 2,542 1,312 501

WP 404 1,278 496 1,915 2,287 999 789 3,274 1,256 297

Table A8.2: Combined NIAs and England Except the NIAs: Total Number of Non-ELS Option Applications 2005 – 2014

Total NIA England Except NIAs

2005 1,177 49,086

2006 9,639 266,671

2007 9,114 238,975

2008 9,367 232,771

2009 10,540 228,976

2010 11,340 360,255

2011 11,724 276,549

2012 12,802 270,886

2013 14,708 237,614

2014 4,097 99,344



Table A8.3: Individual NIAs: Total Number of Holdings with Non-ELS Options 2005 – 2014

NIA 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014

BBC 0 0 0 2 2 5 6 6 4 0

DP 0 0 2 7 14 6 5 4 33 6

DVGH 0 0 1 0 7 2 2 3 4 0

GTM 0 4 9 4 7 10 9 12 24 11

HL 1 4 1 8 7 6 15 24 16 4

M&M 3 20 6 23 10 0 18 12 23 4

MB 4 16 6 13 18 22 18 13 36 19

MD 0 4 2 5 0 11 8 3 4 2

ND 6 40 27 18 8 20 30 24 27 9

NV 1 16 0 8 13 21 13 17 20 2

SD 2 17 16 22 29 33 30 48 31 8

WP 3 9 8 26 17 22 9 24 18 4

Table A8.4: Combined NIAs and England Except the NIAs: Total Number of Holdings with Non-ELS Options 2005 – 2014

Total NIA England Except NIAs

2005 20 566

2006 129 2,381

2007 96 1,900

2008 138 1,877

2009 131 2,197

2010 177 3,455

2011 168 2,820

2012 190 3,025

2013 240 2,720

2014 69 1,184



Figure A8.1: Birmingham and Black Country: Total Number of Non-ELS Option Applications 2005 – 2014

Figure A8.2: Dark Peak: Total Number of Non-ELS Option Applications 2005 – 2014

Figure A8.3: Dearne Valley Green Heart: Total Number of Non-ELS Option Applications 2005 – 2014

-

50

100

150

200

250

300

350

2005 2006 2007 2008 2009 2010 2011 2012 2013 2014

Nu

mb

er

of

Op

tio

ns

Year

Pre-NIA

NIA

-

500

1,000

1,500

2,000

2,500

3,000

2005 2006 2007 2008 2009 2010 2011 2012 2013 2014

Nu

mb

er

of

Op

tio

ns

Year

Pre-NIA

NIA

-

50

100

150

200

250

300

2005 2006 2007 2008 2009 2010 2011 2012 2013 2014

Nu

mb

er

of

Op

tio

ns

Year

Pre-NIA

NIA



Figure A8.4: Greater Thames Marshes: Total Number of Non-ELS Option Applications 2005 – 2014

Figure A8.5: Humberhead Levels: Total Number of Non-ELS Option Applications 2005 – 2014

Figure A8.6: Meres and Mosses: Total Number of Non-ELS Option Applications 2005 – 2014

-

200

400

600

800

1,000

1,200

1,400

1,600

1,800

2,000

2005 2006 2007 2008 2009 2010 2011 2012 2013 2014

Nu

mb

er

of

Op

tio

ns

Year

Pre-NIA

NIA

-

200

400

600

800

1,000

2005 2006 2007 2008 2009 2010 2011 2012 2013 2014

Nu

mb

er

of

Op

tio

ns

Year

Pre-NIA

NIA

-

500

1,000

1,500

2,000

2,500

3,000

2005 2006 2007 2008 2009 2010 2011 2012 2013 2014

Nu

mb

er

of

Op

tio

ns

Year

Pre-NIA

NIA



Figure A8.7: Morecombe Bay: Total Number of Non-ELS Option Applications 2005 – 2014

Figure A8.8: Marlborough Downs: Total Number of Non-ELS Option Applications 2005 – 2014

Figure A8.9: North Devon: Total Number of Non-ELS Option Applications 2005 – 2014

-

500

1,000

1,500

2,000

2,500

2005 2006 2007 2008 2009 2010 2011 2012 2013 2014

Nu

mb

er

of

Op

tio

ns

Year

Pre-NIA

NIA

-

200

400

600

800

1,000

1,200

1,400

2005 2006 2007 2008 2009 2010 2011 2012 2013 2014

Nu

mb

er

of

Op

tio

ns

Year

Pre-NIA

NIA

-

500

1,000

1,500

2,000

2,500

3,000

3,500

4,000

4,500

2005 2006 2007 2008 2009 2010 2011 2012 2013 2014

Nu

mb

er

of

Op

tio

ns

Year

Pre-NIA

NIA



Figure A8.10: Nene Valley: Total Number of Non-ELS Option Applications 2005 – 2014

Figure A8.11: South Downs: Total Number of Non-ELS Option Applications 2005 – 2014

Figure A8.12: Wild Purbeck: Total Number of Non-ELS Option Applications 2005 – 2014

-

200

400

600

800

1,000

1,200

2005 2006 2007 2008 2009 2010 2011 2012 2013 2014

Nu

mb

er

of

Op

tio

ns

Year

Pre-NIA

NIA

-

500

1,000

1,500

2,000

2,500

3,000

2005 2006 2007 2008 2009 2010 2011 2012 2013 2014

Nu

mb

er

of

Op

tio

ns

Year

Pre-NIA

NIA

-

500

1,000

1,500

2,000

2,500

3,000

3,500

2005 2006 2007 2008 2009 2010 2011 2012 2013 2014

Nu

mb

er

of

Op

tio

ns

Year

Pre-NIA

NIA



Appendix 9: Graphs and Tables for Individual NIAs (Approach 3 comparative Analysis)

This appendix includes the following figures and tables:

Comparison of option density within and outside NIAs based on Agricultural Land Type (ALT).

Statistical analysis of option data intersecting ALT’s.

Comparison of option density within and outside NIAs based on National Character Areas (NCA).

Statistical analysis of option data intersecting NCA’s.

Comparison of option density within and outside NIAs based on Agricultural Land Type (ALT).

The following graphs combine the inside and outside density values into a single graph for each NIA. Dashed lines are density values outside the NIA, whilst solid lines are density values inside the NIA. An NIA can intersect more than one ALT so a single NIA will report density values inside and outside itself per intersecting ALT.

Figure A9.1: Birmingham and Black Country comparison of option density within and outside NIAs based on Agricultural Land Type (ALT )

Inside NIA - Western dairy and mixed agriculture

Outside NIA -Western dairy and mixed agriculture



Figure A9.2: Dark Peak comparison of option density within and outside NIAs based on Agricultural Land Type (ALT )

Figure A9.3: Dearne Valley Green Heart comparison of option density within and outside NIAs based on Agricultural Land Type (ALT)

Inside NIA - Upland agricultural landscapes

Outside NIA - Upland agricultural landscapes

Inside NIA - Upland fringe dairy and stock rearing

Outside NIA - Upland fringe dairy and stock rearing

Outside NIA – Chalk and limestone mixed arable landscapes rearing Inside NIA - Chalk and limestone mixed arable landscapes



Figure A9.4: Greater Thames Marshes comparison of option density within and outside NIAs based on Agricultural Land Type (ALT)

Figure A9.5: Humberhead Levels comparison of option density within and outside NIAs based on Agricultural Land Type (ALT)

Outside NIA – Coastal areas

Inside NIA - Coastal areas

Outside NIA – South eastern wooded and mixed agricultural landscapes

Inside NIA - South eastern wooded and mixed agricultural landscapes

Outside NIA – Eastern arable agricultural landscapes

Inside NIA - Eastern arable agricultural landscapes



Figure A9.6: Marlborough Downs comparison of option density within and outside NIAs based on Agricultural Land Type (ALT)

Figure A9.7: Meres and Mosses comparison of option density within and outside NIAs based on Agricultural Land Type (ALT)

Outside NIA – Western dairy and mixed agriculture rearing Inside NIA - Western dairy and mixed agriculture






Figure A9.8: Morecambe Bay Limestone and Wetlands comparison of option density within and outside NIAs based on Agricultural Land Type (ALT)

Figure A9.9: Nene Valley comparison of option density within and outside NIAs based on Agricultural Land Type (ALT)



Inside NIA - Upland agricultural landscapes

Outside NIA - Upland agricultural landscapes



Outside NIA - Upland fringe dairy and stock rearing Inside NIA - Upland fringe dairy and stock rearing






Figure A9.10: Northern Devon comparison of option density within and outside NIAs based on Agricultural Land Type (ALT)

Figure A9.11: South Downs Way Ahead comparison of option density within and outside NIAs based on Agricultural Land Type (ALT)

Inside NIA - Upland fringe dairy and stock rearing

Outside NIA - Upland fringe dairy and stock rearing




Figure A9.12: Wild Purbeck comparison of option density within and outside NIAs based on Agricultural Land Type (ALT)

Statistical analysis of option data intersecting ALT’s

The frequency of density values inside and outside were found to have a non-normal distribution and a Log10 transformation did not improve this. To test if there was a significant difference in option density inside and outside of an NIA the non-parametric Mann-Whitney U test was conducted using Minitab 17. For each year the density inside and outside an NIA was tested, the null hypothesis being that there is no difference between the median density values.

Table A9.1: Results of Mann-Whitney U test comparing NIA areas with related ALT

Year n Median Inside Density

Median Outside Density

Median Difference

95% CI p-value

2006 20 1.925 2.405 -0.410 (-1.819,0.270) 0.2733

2007 20 0.800 1.720 -0.935 (-1.509,-0.080) 0.0468

2008 20 1.210 1.430 -0.480 (-1.040,0.400) 0.3648

2009 20 1.370 1.520 -0.040 (-0.891,1.041) 0.9031

2010 20 2.100 2.070 -0.255 (-1.341,0.380) 0.4903

2011 20 2.085 2.200 0.050 (-0.890,1.179) 0.8817

2012 20 1.535 2.170 -0.790 (-1.510,0.189) 0.1719

2013 20 2.375 1.500 0.380 (-0.530,1.640) 0.3369

2014 20 0.290 0.650 -0.310 (-0.500,0.290) 0.2235

With the exception of 2007 all p-values are greater than 0.05, thus the median density values are not statistically significant and the null hypothesis is not rejected.

Outside NIA – South eastern wooded and mixed agricultural landscapes

Inside NIA - South eastern wooded and mixed agricultural landscapes





Comparison of option density within and outside NIAs based on National Character Areas (NCA) For each year (2006-2014) of Option points the density (Km

2) of points within the dominant NCAs was

calculated inside and outside the NIA. An NIA can intersect more than one NCA. The following graphs combine the inside and outside density values into a single graph for each NIA. Dashed lines are density values outside the NIA, whilst solid lines are density values inside the NIA. An NIA can intersect more than one NCA so a single NIA will report density values inside and outside itself per intersecting NCA.

Figure A9.13: Birmingham and Black Country comparison of option density within and outside NIAs based on National Character Area (NCA)

Figure A9.14: Dark Peak comparison of option density within and outside NIAs based on National Character Area (NCA)

Outside NIA - Arden

Inside NIA - Arden

Outside NIA – Mid Severn Sandstone Plateau

Inside NIA – Mid Severn Sandstone Plateau

Outside NIA – Cannock Chase and Cank Wood

Inside NIA – Cannock Chase and Cank Wood

Outside NIA – White Peak

Inside NIA – White Peak

Outside NIA – Dark Peak

Inside NIA – Dark Peak

Outside NIA – Southern Pennines

Inside NIA – Southern Pennines



Figure A9.15: Dearne Valley Green Heart comparison of option density within and outside NIAs based on National Character Area (NCA)

Figure A9.16: Greater Thames Marshes comparison of option density within and outside NIAs based on National Character Area (NCA)

Outside NIA – Nottinghamshire, Derbyshire and Yorkshire coalfield

Inside NIA – Nottinghamshire, Derbyshire and Yorkshire coalfield

Outside NIA – Southern Magnesian Limestone

Inside NIA – Southern Magnesian Limestone

Outside NIA – Northern Thames Basin

Inside NIA – Northern Thames Basin Outside NIA – Greater Thames Estuary

Inside NIA – Greater Thames Estuary Outside NIA – North Kent Plains

Inside NIA – North Kent Plains



Figure A9.17: Humberhead Levels comparison of option density within and outside NIAs based on National Character Area (NCA)

Figure A9.18: Marlborough Downs comparison of option density within and outside NIAs based on National Character Area (NCA)

Outside NIA – Humber Estuary

Inside NIA – Humber Estuary

Outside NIA – Humberhead Levels

Inside NIA – Humberhead Levels

Outside NIA – Upper Thames Clay Vales

Inside NIA – Upper Thames Clay Vales

Outside NIA – Berkshire and Marlborough Downs

Inside NIA – Berkshire and Marlborough Downs



Figure A9.19: Meres and Mosses comparison of option density within and outside NIAs based on National Character Area (NCA)

Figure A9.20: Morecombe Bay comparison of option density within and outside NIAs based on National Character Area (NCA)

Outside NIA – Shropshire, Cheshire and Staffordshire Plain

Inside NIA – Shropshire, Cheshire and Staffordshire Plain

Outside NIA – Cheshire Sandstone Ridge

Inside NIA – Cheshire Sandstone Ridge

Outside NIA – Morcombe Bay Coast and Lune Estuary

Inside NIA – Morcombe Bay Coast and Lune Estuary

Outside NIA – Morcombe Bay Limestones

Inside NIA – Morcombe Bay Limestones

Outside NIA – South Cumbria Low Fells

Inside NIA – South Cumbria Low Fells

Outside NIA – Lancashire and Amounderness Plain

Inside NIA – Lancashire and Amounderness Plain



Figure A9.21: Nene Valley comparison of option density within and outside NIAs based on National Character Area (NCA)

Figure A9.22: North Devon comparison of option density within and outside NIAs based on National Character Area (NCA)

Outside NIA – Rockingham Forest

Inside NIA – Rockingham Forest Outside NIA – Bedfordshire and Cambridgeshire Claylands

Inside NIA – Bedfordshire and Cambridgeshire Claylands Outside NIA – Northamptonshire Uplands

Inside NIA – Northamptonshire Uplands

Outside NIA – Northamptonshire Vales

Inside NIA – Northamptonshire Vales Outside NIA – Yardley Whittlewood Ridge

Inside NIA – Yardley Whittlewood Ridge

Outside NIA – The Culm

Inside NIA – The Culm



Figure A9.23: South Down Way Ahead comparison of option density within and outside NIAs based on National Character Area (NCA)

Figure A9.24: Wild Purbeck comparison of option density within and outside NIAs based on National Character Area (NCA)

Outside NIA – Low Weald

Inside NIA – Low Weald

Outside NIA – Hampshire Downs Inside NIA – Hampshire Downs

Outside NIA – South Downs

Inside NIA – South Downs

Outside NIA – Wealden Greensand

Inside NIA – Wealden Greensand

Outside NIA – Weymouth Lowlands

Inside NIA - Weymouth Lowlands

Outside NIA – South Purbeck

Inside NIA - South Purbeck

Outside NIA – Dorset Heaths

Inside NIA – Dorset Heaths

Outside NIA – Dorset Downs and Cranborne Chase

Inside NIA - Dorset Downs and Cranborne Chase



Statistical analysis of option data intersecting NCAs

The frequency of density values inside and outside were found to have a non-normal distribution and a Log10 transformation did not improve this. To test if there was a significant difference in option density inside and outside of an NIA the non-parametric Mann-Whitney U test was conducted using Minitab 17. For each year the density inside and outside an NIA intersecting the NCAs was tested, the null hypothesis being that there is no difference between the median density values.

Table A9.2: Results of Mann-Whitney U test comparing NIA areas with related ALT

Year n Median Inside Density

Median Outside Density

Median Difference

95% CI p-value

2006 35 1.040 1.780 -0.400 (-0.970,0.320) 0.2174

2007 35 0.190 1.040 -0.480 (-0.960,-0.130) 0.0136

2008 35 0.600 1.120 -0.350 (-0.830,0.260) 0.1943

2009 35 1.150 0.850 -0.080 (-0.420,0.720) 0.6553

2010 35 1.169 1.520 -0.290 (-0.980,0.650) 0.5689

2011 35 1.340 1.460 -0.120 (-0.760,0.780) 0.7158

2012 35 1.110 1.440 -0.250 (-0.800,0.520) 0.4665

2013 35 1.990 1.310 0.580 (-0.280,1.549) 0.2219

2014 35 0.160 0.320 -0.080 (-0.250,0.080) 0.2109

With the exception of 2007 all p-values are greater than 0.05 thus the median density values are not statistically significant and the null hypothesis is not rejected.

Documents

NIA Evaluation Report - GOV.UKrandd.defra.gov.uk/Document.aspx?Document=13506... · NIA periods and there are key changes in agri-environment policy and delivery over this time period