Uddannelses- og Forskningsudvalget 2015-16
UFU Alm.del Bilag 130
Offentligt
1618648_0001.png
June 2015
The Place of Research-based Evidence in
Policymaking
www.technopolis-group.com
 UFU, Alm.del - 2015-16 - Bilag 130: Baggrundsrapporter fremsendt den 7/4-16 fra Danmarks Forsknings- og Innovationspolitiske Råd
1618648_0002.png
 UFU, Alm.del - 2015-16 - Bilag 130: Baggrundsrapporter fremsendt den 7/4-16 fra Danmarks Forsknings- og Innovationspolitiske Råd
The Place of Research-based Evidence in Policymaking
Final Report
technopolis
|group|,
June 2015
Erik Arnold
Peter Kolarz
Annemieke Pickles
Jari Romanainen
www.technopolis-group.com
 UFU, Alm.del - 2015-16 - Bilag 130: Baggrundsrapporter fremsendt den 7/4-16 fra Danmarks Forsknings- og Innovationspolitiske Råd UFU, Alm.del - 2015-16 - Bilag 130: Baggrundsrapporter fremsendt den 7/4-16 fra Danmarks Forsknings- og Innovationspolitiske Råd
1618648_0005.png
Summary
This report discusses the use of research-based evidence in policymaking. It is based
on interviews with six ministries/directorates general in the UK, Netherlands, Finland
and the European Commission and forms part of a larger project of the Danish
Council for Research and Innovation Policy (DFiR), which aims to study and improve
Danish practice.
Historically, the new public management movement has been an important driver for
using a mixture of research-based evidence and monitoring data in policymaking. A
more recent impetus has been the idea – heavily promoted by the Blair and Obama
governments but increasingly becoming orthodox – that policy should be based on
‘what works’, rather than on ideology. The financial crisis has increased the
importance of evidence by reducing government funding for both making and
implementing policy and making it even more important that scarce budget is used
effectively.
Policymakers have to manage a ‘dynamic inconsistency’ between the pace of
evidence generation and the needs of current and future policies. They need a mix of
rapidly available evidence to underpin short-term decision-making and programmes
of longer-term work that help them address likely future evidence needs. Foresight
and related techniques are becoming more attractive because it provides a way to
think about future evidence needs (as opposed to being a way to satisfy those
needs).
Civil servants rather than politicians generate most demand for evidence because
they do the detailed design of policy, manage and monitor its implementation.
Politicians sometimes prefer ‘evidence-informed’ to ‘evidence-based’ policy. While
civil servants generally believe that they can identify selective use of evidence by
lobbyists, they feel that on occasion their political masters are not above using this
tactic themselves. But the more robust the evidence is, the greater its chances of
forming a basis for policy.
A growing number of policy issues cross the boundaries between ministries and their
sector responsibilities. Cross-ministry cooperation in evidence collection is fairly easy
where few ministries are involved, but wider issues need new external structures.
Ministries use informal as well as formal ways to access evidence. There is a broad
trend from using captive evidence sources such as government laboratories towards
‘marketisation’ and the use of a growing number of other types of organisation
including universities, institutes and consultants. Over-use of a small number of
evidence sources risks ‘lock-in’ to their ideas while depending only on the market can
put the sustainability of evidence-providers in doubt, especially in small countries.
The Place of Research-based Evidence in Policymaking
i
 UFU, Alm.del - 2015-16 - Bilag 130: Baggrundsrapporter fremsendt den 7/4-16 fra Danmarks Forsknings- og Innovationspolitiske Råd
1618648_0006.png
Technical councils and standing committees tend not to be central to evidence
collection – rather they are more important for legitimation and quality control.
Surprisingly, high-level councils that potentially could provide policy coordination
across government as a whole appeared to have a limited effect on policymaking at
individual ministry level.
International sources of evidence tend to be used for benchmarking and background
understanding of the international context, rather than for generating specific policy-
focused evidence.
In general, ministries are moving towards a model in which they have high internal
capacity to acquire and generate evidence, in part by engaging with the wider
evidence community – especially universities – in an open way. Their capacity to
absorb and use evidence and to specify their evidence needs has been increasing This
increased absorptive capacity and the integration of policymakers into the wider
research community goes hand in hand with greater transparency – especially in
terms of more frequently publishing the evidence used by government.
For the most part, ministries are more interested in the quality of evidence than in
who produced it. Policymakers generally make their own judgements about the
quality of evidence available to them. In some cases they may use a committee for
legitimation. They increasingly want to experiment with new types of evidence
though so far this is at an early stage.
Only the European Commission consistently ties evidence collection to a formal
policy cycle. At the national level, ministries vary greatly in the extent to which they
use such a cycle. Guidelines for doing individual evidence-related activities are
available to most policymakers and appear most influential where they are used
system-wide rather than being specific to a ministry. But policymakers caution
against their heavy-handed application.
Culture makes a difference to how people behave and the transportability of
evidence practices. It seems to have particular importance in relation to trust. UK
evidence use is influenced by the presence of Chief Scientific Advisers and an
increasing focus in government on demonstrating the societal impacts of policy.
Dutch ministries have a uniform long-term and strategic approach to collecting
evidence for policy. Finland has radically centralised the collection and funding of at
least some of the evidence needed for policymaking. The European Commission
leans more heavily on formalised processes and the use of expert panels in evidence
collection than is the case at national level. Sector ministries are strongly influenced
by the national administrative tradition within which they operate. There are
nonetheless similarities that result from the characteristics of the sectors themselves.
The practices observed in this study suggest paying attention to the following aims in
order to make good use of research-based evidence in policymaking.
ii
The Place of Research-based Evidence in Policymaking
 UFU, Alm.del - 2015-16 - Bilag 130: Baggrundsrapporter fremsendt den 7/4-16 fra Danmarks Forsknings- og Innovationspolitiske Råd
1618648_0007.png
1. Be as evidence-based as possible. There is broad agreement that policy
should be made on the basis of research-based evidence, wherever possible.
2. Use foresight and other techniques for thinking about the future as ways to
anticipate coming policy needs – and therefore the kinds of evidence that
will be needed to support them.
3. Devise and invest in research strategies that generate evidence that will be
needed in the longer term as well as in the immediate future.
4. Ensure that ministries are staffed with a significant proportion of people who
can specify the need for research as well as to make use of external inputs in
order to generate evidence for policymaking
5. Have ‘evidence champions’ – they might look like Chief Scientific Advisors;
they might look like ‘departments for knowledge’ or ‘knowledge
coordinators – to promote and coordinate the generation and use of
evidence for policymaking
6. Create funded arrangements for generating and sharing evidence to address
cross-ministry problems
7. Maintain long-term links with organisations like universities that work at the
boundary between research and policy but do not let these become
monopolies – you also need impulses for change from a wider set of
institutions (including foreign ones) working in competition
8. Publish evidence so that policymaking is transparent and others can quality-
assure as well as re-use the evidence you employ
9. Use a ‘light touch’ policy cycle, which suggests good practice guidelines for
collecting and using evidence but which is rather more firm about the
requirement to evaluate interventions both
ex ante
and
ex post
10. Be prepared to experiment and learn about new intervention designs and
ways to develop evidence
The Place of Research-based Evidence in Policymaking
iii
 UFU, Alm.del - 2015-16 - Bilag 130: Baggrundsrapporter fremsendt den 7/4-16 fra Danmarks Forsknings- og Innovationspolitiske Råd
1618648_0008.png
Table of Contents
1. Introduction
1.1 Scope of the study
1.2 Method
1.3 Structure of this report
2. The use of research-based evidence in policymaking
2.1 Drivers and demand for evidence
2.2 Supply and suppliers of evidence
2.3 Absorptive capacity and the enlightenment model
2.4 Quality and credibility
2.5 Evidence and the policy cycle
3. National and sectoral specificities
3.1 Specific aspects of national behaviour
3.2 Sector-specific aspects of evidence use
4. How do our findings compare with existing knowledge?
5. Good practice in the use of research-based evidence for policymaking
Appendix A Data collection checklist
Appendix B Interviewee details
Appendix C Country report: UK
Appendix D Country Report: Netherlands
Appendix E Country Report: Finland
Appendix F Profile: European Commission
Appendix G Literature review
Appendix H Bibliography
1
1
2
3
5
5
13
15
16
17
19
19
21
23
26
28
33
36
58
74
92
105
120
iv
The Place of Research-based Evidence in Policymaking
 UFU, Alm.del - 2015-16 - Bilag 130: Baggrundsrapporter fremsendt den 7/4-16 fra Danmarks Forsknings- og Innovationspolitiske Råd
1618648_0009.png
List of Tables
Table 1: Countries and Ministry Sectors Studied .......................................................... 2
Table 2: Overview – UK: Department for Education ................................................... 41
Table 3: Overview – UK: Department of Environment, Food and Rural Affairs .......... 42
Table 4: Overview – UK: Foreign and Commonwealth Office ..................................... 43
Table 5: Overview – UK: Department of Health .......................................................... 44
Table 6: Overview – UK: Department for Business, Innovation and Skills .................. 45
Table 7: Overview – UK: Department for Transport ................................................... 46
Table 8: Overview – NL: The Ministry of Education, Culture and Science (OCW)....... 61
Table 9: Overview – NL: The Ministry of Infrastructure and Environment (I&M)....... 64
Table 10: Overview – NL: The Ministry of Health, Welfare and Sports (VWS)............ 66
Table 11: Overview – NL: Ministry of Economic Affairs (EZ) ....................................... 67
Table 12: Overview – NL: The Ministry of Foreign Affairs (BZ) ................................... 68
Table 13: Overview – FI: Ministry of Education and Culture....................................... 78
Table 14: Overview – FI: Ministry of the Environment ............................................... 79
Table 15: Overview – FI: Ministry for Foreign Affairs.................................................. 81
Table 16: Overview – FI: Ministry of Social Affairs and Health ................................... 82
Table 17: Overview – FI: Ministry of Employment and the Economy ......................... 84
Table 18: Overview – FI: Ministry of Transport and Communications........................ 86
Table 19: Overview – EC: DG Education and Culture .................................................. 96
Table 20: Overview – EC: DG Environment ................................................................. 97
Table 21: Overview – EC: Foreign Policy Instruments ................................................. 98
Table 22: Overview – EC: DG Food Safety and Health ................................................ 99
The Place of Research-based Evidence in Policymaking
v
 UFU, Alm.del - 2015-16 - Bilag 130: Baggrundsrapporter fremsendt den 7/4-16 fra Danmarks Forsknings- og Innovationspolitiske Råd
1618648_0010.png
Table 23: Overview – EC: DG Research and Innovation ............................................ 100
Table 24: Overview – EC: DG Mobility and Transport ............................................... 101
Table of Figures
Figure 1 Options for obtaining research-based evidence ............................................. 8
Figure 2 Weiss’ typology of evidence use................................................................. 107
Figure 3 Conceptual framework for the absorption of research knowledge by civil
servants ..................................................................................................................... 111
Figure 4 The implications of complex systems theory .............................................. 112
Figure 5 findings of systematic reviews..................................................................... 113
Figure 6 The Policy cycle ............................................................................................ 115
Figure 7 Components of policy process and different evidence issues .................... 116
vi
The Place of Research-based Evidence in Policymaking
 UFU, Alm.del - 2015-16 - Bilag 130: Baggrundsrapporter fremsendt den 7/4-16 fra Danmarks Forsknings- og Innovationspolitiske Råd
1618648_0011.png
1.
Introduction
The Danish Council for Research and Innovation Policy (DFiR) commissioned this
study as one component in a larger effort to review and improve the use of research-
based evidence in Danish policy formation. The purpose of the present volume is to
provide an international backdrop to the work in Denmark. It will allow DFiR to
understand practice in a range of countries, some of whose characteristics are similar
to those of Denmark but which also comprise policy contexts that are rather
different.
1.1
Scope of the study
The comparator countries are the UK, Netherlands and Finland. In addition, the study
addresses the European Commission (EC).
Based on the focus of DFiR’s wider project, this report looks at six sectors or policy
domains and hence the corresponding ministries, departments and Directorates
General.
•
•
•
•
•
•
Education
Environment
Foreign Affairs
Health
Research and Innovation
Transport
For simplicity we refer to this mix of ministries, departments of state and
directorates general as ‘ministries’, while recognising that these categories have
differing purposes. In this report, ‘research-based evidence’ means evidence either
published in the peer-reviewed scientific literature or commissioned for the purpose
of supporting policy. The latter is not usually published in peer-reviewed journals but
is increasingly available in the form of studies, typically published on the World Wide
Web.
Our report focuses on policymakers but also considers politicians. Since not all
languages distinguish between politics and policy, it is perhaps useful to clarify the
distinction here. Politics is the business of (hopefully!) elected politicians, who
normally belong to political parties and make commitments to the electorate that
they will follow certain courses of action, ie policies. Politicians usually have a rather
broad-brush approach to policies. Policymakers are typically civil servants, who take
the broad lines of policy from the political level and translate them into specific
The Place of Research-based Evidence in Policymaking
1
 UFU, Alm.del - 2015-16 - Bilag 130: Baggrundsrapporter fremsendt den 7/4-16 fra Danmarks Forsknings- og Innovationspolitiske Råd
1618648_0012.png
actions that can be implemented. Policymaking involves answering a lot of ‘how?’
questions; politics more often asks ‘what?’ Both levels need access to evidence.
1.2
Method
We began by reviewing available literature about the use of research-based evidence
in policymaking (see Appendix G). We used the results of the review together with a
number of questions posed by DFiR and discussed with us ahead of starting the study
in order to generate a checklist of issues to investigate (see Appendix A). Based on
DFiR’s suggestions, we then identified administrations corresponding to the policy
areas set out above (Table 1).
The main phase of the study consisted of interviews with key individuals in each
selected ministry, as well as additional desk research, studying relevant documents
(eg work plans, research strategies and guidance handbooks for evidence use). We
conducted 1-2 interviews per ministry. Interviews lasted between 45 and 90 minutes.
Interviewees participated on the basis that responses would be reported in an
aggregated manner, with no use of attributable quotes. Although we took care to
cover all questions on our checklist, we opted for a semi-structured approach to
allow interviewees to elaborate on issues that had not been anticipated through the
checklist and literature review. Around 30% of interviews were conducted via
telephone, the remainder face-to-face in the ministries themselves. The interview
data were coded so that we could compare answers to our checklist of questions
across ministries within the same country, as well as across comparable ministries
across countries.
We targeted individuals at the highest possible level responsible for strategy and/or
evidence in relation to policymaking. The full list of interviewees is in Appendix A.
Most of the people we approached were very keen to participate and themselves to
receive this report, although the level of enthusiasm was lower in the EC, where two
administrations declined to be interviewed (see Appendix F).
Table 1 Countries and Ministry Sectors Studied
UK (Pilot)
Netherlands
Ministry of Education,
Culture and Science
(OCW)
Ministry
of
Infrastructure
and
Environment (I&M)
Finland
EC
Education
Department
Education (DfE)
for
Ministry of Education
and Culture (OKM)
DG Education
Culture (EAC)
and
Environment
Department
Environment,
and Rural
(DEFRA)
of
Food
Affairs
Ministry
of
the
Environment (YM)
DG Environment (Env)
Foreign
Affairs
Foreign
and
Commonwealth Office
(FCO)
Ministry of Foreign
Affairs (BZ)
Ministry for Foreign
Affairs (UM)
Foreign
Policy
Instruments (FPI)
2
The Place of Research-based Evidence in Policymaking
 UFU, Alm.del - 2015-16 - Bilag 130: Baggrundsrapporter fremsendt den 7/4-16 fra Danmarks Forsknings- og Innovationspolitiske Råd
1618648_0013.png
Health
Department of Health
(DoH)
Ministry of Health,
Welfare and Sports
(VWS)
Ministry of Social
Affairs and Health
(STM)
Ministry
of
Employment and the
Economy (TEM)
Ministry of Transport
and Communications
(LVM)
DG Food Safety and
Health (Sante)
Research &
Innovation
Department
for
Business, Innovation
and Skills (BIS)
Ministry of Economic
Affairs (EZ)
DG Research and
Innovation (RTD)
Transport
Department
Transport (DfT)
for
Ministry
of
Infrastructure
and
Environment (I&M)
DG
Mobility
and
Transport (Move)
Most interviewees were able to answer most questions. Moreover, we have strong
indications of high reliability of the interview data
•
Where two interviews were conducted per ministry, most answers given were
consistent with each other; this was also the case where the two interviewees
held quite different positions within the ministry. We can thus be confident that
selection bias – a known risk when working with a small number of interviewees
– is kept to a minimum
Interviewees also demonstrated a high degree of openness, very frequently to
the point of self-criticism. We can therefore be relatively confident that self-
censorship did not play a major role in our data collection
•
Limitations of this study include that
•
It addresses a small number of countries, so it does not reflect the full
international range of practices and approaches to the generation and use of
research-based evidence in policymaking
It relies heavily on interviews and therefore on the views and perceptions of
those people with whom we spoke. The scope for triangulation between the
interviews and other sources of data was limited
In particular, our interviews focused on professional policymakers (civil servants
and researchers in positions of giving advice to ministries). We were not able to
collect much information about the political perspectives of the ministers and
commissioners in overall charge
•
•
1.3
Structure of this report
The report is in two parts. This main report is intended for policymakers and general
readers. We look first at general results that appear to apply across all or most of the
administrations considered. We then describe some results that are more specific,
first to the way things are done in particular countries and, second, to the individual
sectors considered. We briefly summarise what was already described in the
The Place of Research-based Evidence in Policymaking
3
 UFU, Alm.del - 2015-16 - Bilag 130: Baggrundsrapporter fremsendt den 7/4-16 fra Danmarks Forsknings- og Innovationspolitiske Råd
1618648_0014.png
literature about the use of research-based evidence and comment on this based on
the results of our study. We conclude with some suggestions about good practice,
though in reading these, the reader should be mindful of the importance of both
national and sector contexts in determining the best way to develop policy.
The second part of the report comprises appendices that describe our method,
country by country findings and the results of our literature review in more detail. It
is aimed at those conducting the Danish part of the DFiR study and at others wanting
to verify, reproduce or extend the research.
4
The Place of Research-based Evidence in Policymaking
 UFU, Alm.del - 2015-16 - Bilag 130: Baggrundsrapporter fremsendt den 7/4-16 fra Danmarks Forsknings- og Innovationspolitiske Råd
1618648_0015.png
2.
The use of research-based evidence in policymaking
In this section, we set out the major conclusions that apply across countries and
ministry sectors. We consider why policymakers (generally civil servants) want to use
research-based evidence in setting policy, especially in this period of austerity. We
look at the way policymakers see the role of politicians in relation to evidence use
and then at the use of evidence in addressing policies that cut across the interests of
individual ministries. Next we look at the way ministries use different suppliers of
evidence. We point out how important it is for ministries to have people with the
capacity to work with research-based evidence, how evidence is quality controlled
and the extent to which the use of evidence is connected to a formalised ‘policy
cycle’ that systematically maps a course from problem definition through designing
and implementing a policy or programme, evaluating it and learning from the
experience in the design of future policy.
2.1
Drivers and demand for evidence
2.1.1
Demand for evidence in a time of austerity
The new public management movement has historically been an important driver
for using a mixture of research-based evidence and monitoring data, in
policymaking
All the people we consulted said that research-based evidence is of fundamental
importance in policymaking. They tended to see opportunities for increasing and
improving the use of evidence beyond present-day levels. They would regard such a
development as intrinsically a good thing. However, they also acknowledged that
policy itself can never be
fully
evidence driven and understand that political,
electoral, financial and diplomatic pressures as well as a host of other factors
naturally inform decision making.
Most saw the New Public Management movement as one now relatively long-
standing driving force behind this interest in research-based evidence. To varying
degrees, all the ministries had an ‘evaluation culture’ (Dahler-Larsen 2012, 2013) that
has its origins in the New Public Management going back to the 1980s (Hood 1991).
While the New Public Management is clearly an important influence, its focus has
historically been the on daily operations of the state. Typically, it is implemented
through ‘performance contracts’, for example between a ministry and its agencies.
In the UK these are known as ‘service level agreements’, which is a useful label
reflecting the focus of the new public management on the state delivering services to
citizens at agreed levels of quality, reliability and cost. That is, it tends to focus on
the delivery of existing policies rather than the development of new ones.
The Place of Research-based Evidence in Policymaking
5
 UFU, Alm.del - 2015-16 - Bilag 130: Baggrundsrapporter fremsendt den 7/4-16 fra Danmarks Forsknings- og Innovationspolitiske Råd
1618648_0016.png
Lack of resources is frequently noted as an issue in ministries. But whilst the recent
funding cuts often referred to by interviewees derive from the financial crisis and
austerity programmes of recent years, it is worth noting that trends towards smaller
ministries and more accountability through evaluation have been in place since well
before the financial crisis.
A more recent driver has been the idea that policy should be based on ‘what
works’, rather than ideology
It was after the Blair (and subsequently the Obama) administrations started
emphasising the importance of ‘what works’ as opposed to ideology in designing
policy (see Appendix C) that the need for research-based evidence extended further
beyond the monitoring perspective relevant to managing service level agreements
and towards the more wide ranging types of evidence relevant to setting policy
directions.
Implicitly, this increased role of evidence in policy formulation challenges the
boundary between ‘politically inspired’ policies and those generated by more
technocratic means, potentially challenging the boundary between the
responsibilities of civil service policymakers and the politicians they serve.
The financial crisis has increased the importance of evidence
Recent economic pressures have further increased demands for greater efficiency.
Across the board, funding cuts are a two-fold driver for evidence use. First, decreased
levels of funding for ministries reduce the resources available to carry out research or
to procure, generate or analyse evidence. At the same time, funding cuts have also
led to demands for greater accountability in policymaking. In many cases, this
involves an especial emphasis on quantitative methods of estimating the economic
benefits of policy. New policies increasingly need to be justified, notably though ex
ante impact assessment, interim and ex post evaluation.
Critically, this has also led to the need for more sophisticated data, as well as
methods to measure, assess or forecast wider impacts, both of a direct economic and
a wider social and economic nature. This dual effect means that the demand for
evidence has increased, but the resources with which to generate and analyse it have
shrunk.
6
The Place of Research-based Evidence in Policymaking
 UFU, Alm.del - 2015-16 - Bilag 130: Baggrundsrapporter fremsendt den 7/4-16 fra Danmarks Forsknings- og Innovationspolitiske Råd
1618648_0017.png
Policymakers need a mix of rapidly available evidence to underpin short-term
decision-making and programmes of longer-term work that help them address
likely future evidence needs
The main high-level conclusion when looking across our sample of three countries
and the European Commission is that there is a fundamental tension that decisively
shapes the landscape of evidence-based policymaking. On one side, pressures on
funding have led to a need for greater accountability, efficiency, standardisation and
clear explanation, direct utility and targeted focus of all activities undertaken by
ministries, including collection of evidence. At the same time, the complexity of
ministries’ activities, the political challenges of the present day and the need better
to understand a diverse range of long-term impacts of policies, necessitates a broad
and integrated understanding by ministries. This in turn triggers demands for cross-
ministry coordination and dialogue as well as the need for evidence collection of
wider scope not directly targeted at measuring or justifying particular, immediate
policy needs. This type of work is variously termed ‘strategic’, ‘prospective’ or
‘foresight’ and is essential in providing the policymaking sphere with a broader view
and making it more responsive and prepared for emerging policy needs. However,
these endeavours do not readily fit into the ‘New Public Management’ paradigm of
direct efficiency, accountability and targeted, immediate focus.
Policymakers have to manage a ‘dynamic inconsistency’ between the pace of
evidence generation and the needs of current and future policies.
The dynamic inconsistency between the timescales relevant in politics and research
is well known. The political incentive system rests on politicians being (re-)elected
and therefore on doing things that generate approval in the comparatively short
term. Hence ministers are always in a hurry. Research often cannot produce results
within a parliamentary term and very rarely can research results be put into social
practice in such a short time. One of the preconditions for successful research policy
is therefore to generate political rewards in a relevant timescale. Changing the
nature of the political debate so that the act of supporting research is itself seen as a
sensible long-term investment in the interests of the nation often does this. Where
research policy is not politically contentious, this brings political credit to those who
promote it. High-level research and innovation councils often help play a bridging
role between political and research timescales, legitimising the idea that supporting
research is a good thing in and of itself (OECD, 2009).
The use of research-based evidence for policymaking runs into a similar dynamic
inconsistency. Many policy problems (or opportunities) have to be addressed
immediately; often there is little time to wait for new or missing evidence to be
generated. The policymakers we interviewed have collectively a repertoire of
The Place of Research-based Evidence in Policymaking
7
 UFU, Alm.del - 2015-16 - Bilag 130: Baggrundsrapporter fremsendt den 7/4-16 fra Danmarks Forsknings- og Innovationspolitiske Råd
1618648_0018.png
behaviours for obtaining evidence, depending upon its urgency and whether they can
act proactively to obtain it (Figure 1).
•
Funding basic
research
Long
term
•
Scoping
•
Research agenda
setting
•
Deeper, specific
future studies
•
Keeping abreast of
the research agenda
•
Interaction with the
research
community
•
Links to government
labs and other
boundary
organisations
•
Rapid reviews of
existing evidence
Short
term
•
Strategic studies,
impact
assessments,
evaluations etc
•
Using existing
scientific expertise
for advice
Proactive
Reactive
Figure 1 Options for obtaining research-based evidence
Where policy has to react to a new and urgent situation (such as an epidemic) there
may be little choice but reactively to use available scientific advice. One of the
strengths of the UK system is that its Chief Scientific Advisers are ready at any time to
connect ministry evidence needs with research through their personal and
professional networks. The key seems to be as efficiently reactive as possible,
whether by using ‘captive’ information sources such as government labs or outputs
from other kinds of boundary organisations (SE quadrant of Figure 1).
Where there is a little more time, typically in the design of a new or modified policy
intervention, it is also possible to do or commission studies to underpin policy
development. All the ministries we interviewed had budgets for this. Inherently, the
fairly short time available focuses the evidence collection towards the collection and
use of evidence based on existing knowledge or theory – and therefore studies –
than original research (‘discovery’). Here the policy maker needs either to get a
good overview of what is known by reviewing existing literature, surveying the extent
of the problem etc or to engage more closely with the scope, design or evaluation of
the policy intervention. Some of this work – especially evaluation - may be
mandatory, providing a strong basis for the ministry to ask for budget (SW quadrant).
8
The Place of Research-based Evidence in Policymaking
 UFU, Alm.del - 2015-16 - Bilag 130: Baggrundsrapporter fremsendt den 7/4-16 fra Danmarks Forsknings- og Innovationspolitiske Råd
1618648_0019.png
Filling evidence gaps, however, requires research and a longer timescale. In some
cases, the need will be for more fundamental work than can be conducted in the
short term. Policymakers can proactively commission needed research and studies,
including basic research, or buy work such as road mapping that helps identify and
set research agendas required to understand and satisfy future evidence needs (NW
quadrant).
Not everything can be anticipated, so the policymaker needs also to be able to
monitor and understand signals emerging from research that is initiated bottom up
or that has been conducted based on others’ needs (NE quadrant).
Foresight is becoming more attractive because it provides a way to think about
future evidence needs (as opposed to being a way to satisfy those needs)
Across many interviews, participants noted the growing need for foresight studies
that are broader than typical policy evaluations and have a strong forward-looking
element, which aim to help develop an understanding of future policy needs, or of
emerging fields of concern and/or opportunity.
•
Foresight provides a way to reflect on future policy needs. It is not a useful way
to
predict
the future but is helpful as a technique for thinking about future
possibilities in a structured way so that the ministry is prepared for alternative
policy futures
Once alternative policy futures are understood, the ministry can also understand
its likely evidence needs – and use this understanding to inform its strategy for
research and evidence collection
•
Whilst interviewees are enthusiastic about foresight and rate it as a growing and
important field, it does not readily fit into the paradigm of focused and targeted,
policy-specific culture of evidence, exemplified by the strong focus on policy
monitoring, ex ante assessment and evaluation. Foresight by definition is broad, with
open outcomes, and its ultimate value is contingent on whether the threats and
opportunities it highlights actually materialise. Moreover, the robustness that
foresight can have is inherently in question: even with significant expert input,
looking ahead, or extrapolating from past and present to the future entails clear
methodological dangers.
As such, where ministry’s research budgets are cut, wider strategic and foresight
projects may be vulnerable. But despite this it is evident that its potential longer-
term benefits have led to a resurgence of foresight. High quality foresight requires
some degree of formal commitment. Whilst foresight activities are generally not
formalised in as much detail as for instance evaluations, the trend is towards having a
general strategy for foresight, noting for instance some key areas where a ministry
The Place of Research-based Evidence in Policymaking
9
 UFU, Alm.del - 2015-16 - Bilag 130: Baggrundsrapporter fremsendt den 7/4-16 fra Danmarks Forsknings- og Innovationspolitiske Råd
1618648_0020.png
intends to conduct and commission work to generate a greater understanding, as
well as the need to allocate a suitable budget to such endeavours.
2.1.2
Evidence and the political sphere
Civil servants rather than politicians generate most demand for evidence
While there has been pressure for increased use of evidence from the political level,
responsibility for the sustained evidence production and the strategies needed to
generate it remains firmly with the civil service. This is perhaps not surprising, since
in the countries we looked at the civil service tends the machinery of government
and implements policy over the long term.
Politicians sometimes prefer ‘evidence-informed’ to ‘evidence-based’ policy
There is broad consensus amongst our interviewees that evidence is not the only
aspect influencing political decisions. Other relevant factors include financial
concerns, electoral concerns – both tactically in terms of ensuring re-election, but
also more broadly to respond to public opinion or concern – as well as wider
diplomatic concerns (ie considering potential responses to policy from abroad).
Among the civil servants we interviewed, too, there was little appetite for a
‘dictatorship of science’, where the task of policymakers and politicians would be
simply to do what the evidence said. As our literature review suggested, ‘evidence-
based policy’ is perhaps a misnomer, and ‘evidence-informed policy’ is a more
accurate descriptor of the fact that at the political level there is a need to trade off
the implications of the evidence against other considerations. Interviews in from
Finland suggest that some degree of dialogue between politicians and evidence
specialists is changing the pattern of evidence use and the sources from which
government is prepared to take evidence. In particular, it is increasingly interested in
evidence that comes from outside the government labs and the ‘usual suspects’
among Finnish-based consultants, so government has used a greater number of
independent committees to investigate policy questions in recent years.
While civil servants generally believe that they can identify selective use of
evidence by lobbyists, they felt that on occasion their political masters were not
above using this tactic themselves
Our interviewees feel that ministries’ increasing ability to identify and use policy-
relevant evidence meant that they were becoming harder for people presenting one-
sided evidence to fool. But they can also point to instances where politicians had
effectively decided upon certain policy measures without reference to evidence,
leaving policymakers with the task of compiling ‘policy-based evidence’ after the
10
The Place of Research-based Evidence in Policymaking
 UFU, Alm.del - 2015-16 - Bilag 130: Baggrundsrapporter fremsendt den 7/4-16 fra Danmarks Forsknings- og Innovationspolitiske Råd
1618648_0021.png
event. The political level is on some rare occasions also capable of entirely
disregarding evidence unfavourable to its preferred policy options.
But generally, the more robust the evidence the greater its chances of forming a
basis for policy
Whilst policymakers largely accept that politicians need to take into account
considerations other than the evidence base, a strong evidence base can
nevertheless help override such considerations better than a weak evidence base. In
order to ensure consistently strong evidence bases, a certain degree of autonomy of
the policy sphere is important: undue political influence over the evidence process
can place limits on the quality of evidence.
Interviewees’ reflections on examples of successful and unsuccessful uses of
evidence in policymaking confirm this view.
1
Examples of unsuccessful use of
evidence in policymaking include using poor evidence as a starting point as well as
disregard for strong evidence. Sometimes, the poor quality of available evidence was
a result of the need to act quickly based on limited information – at the time of the
interviews, the Charlie Hebdo shooting and the Ebola crisis in West Africa were
among the examples given. Successful examples of evidence-based policymaking
fairly consistently involved devoting due time and attention to producing an evidence
base, evidence that was judged to be of a particularly high standard and that was
available in accessible form. When evidence had these characteristics it was judged
to be sufficiently robust decisively to influence political decisions.
Broadly, it is clear that there is widespread understanding that politics cannot blindly
follow the recommendations of evidence, and also that real world events sometimes
trigger short-term evidence needs, to which the policy sphere must respond.
However, excessive political influence on evidence use in ministries risks
undermining the capacity to produce the best possible evidence base, which in turn
can undermine the capacity of evidence to influence political decisions.
1
As interviewees were given assurance of aggregated reporting, we cannot disclose individual examples given, as
these would inevitably connect directly back to the specific interviewees.
The Place of Research-based Evidence in Policymaking
11
 UFU, Alm.del - 2015-16 - Bilag 130: Baggrundsrapporter fremsendt den 7/4-16 fra Danmarks Forsknings- og Innovationspolitiske Råd
1618648_0022.png
2.1.3
Cross-ministry issues
A growing number of policy issues cross the boundaries between ministries and
their sector responsibilities
Division into sectoral ministries is an established organisational feature of
government yet a growing number of policy issues cross the boundaries between
them, notably in relation to the so-called grand challenges: ageing populations,
climate change, sustainable energy, security and so on. As with wider strategic and
foresight evidence, the growing importance of cross-ministry work does not fit easily
with the structure-driven approach of New Public Management, where clear
delineation of responsibility and separation of functions are key organising principles.
The nature of many policy issues forces compartmentalised government into deeper
integration, including and understanding of how policies in one sector might well
have impacts and trigger new policy needs in others. Boundaries between ministries
have been additionally blurred through growing interest in impact assessment.
Especially where wider impacts (rather than direct outcomes) of policy are to be
measured or assessed, they often cross over into the remit of other ministries, eg a
policy in transport might have impacts on the environment, health and jobs.
Cross-ministry cooperation in evidence collection is fairly easy where few ministries
are involved, but wider issues need new external structures
The cross-sectoral nature of many issues requiring policy attention has gone some
way to breaking down the divisions between ministries, with most interviewees
noting at least some level of cross-ministry activity, including notably the joint
collection of evidence. In small-scale collaborations, involving for instance two or
three ministries, ministries are generally able to commission, conduct and publish
studies together without any significant problems.
However, at a larger scale, where an area of investigation becomes relevant to a
large number of ministries, a coordinated approach becomes difficult and
mechanisms are required to ensure results can be better achieved beyond the
traditional ministerial structure, without competing ministerial interests or
paradigms endangering success. In the UK, the Cabinet Office has therefore started
several ‘What Works Centres’ as evidence gathering facilities with cross-ministerial
involvement, whilst the EC has a system of assembling cross-DG panels with
representatives of each DG involved in order to lead activities. Whilst our data do not
allow clear conclusions on what an ideal approach should look like, some degree of
identification of issues relevant to several ministries, and an available system to
ensure coordination of evidence collection above the ministry-level appears to be an
important emerging practice.
12
The Place of Research-based Evidence in Policymaking
 UFU, Alm.del - 2015-16 - Bilag 130: Baggrundsrapporter fremsendt den 7/4-16 fra Danmarks Forsknings- og Innovationspolitiske Råd
1618648_0023.png
2.2
Supply and suppliers of evidence
Ministries use informal as well as formal ways to access evidence
Reduced research budgets mean that ministries have to become more embedded in
networks with evidence producers, notably universities, in order to access longer-
term developments in research. The ability to do this depends upon civil servants
being well educated and to some degree research trained, so that their absorptive
capacity is high. A corollary is that they tend to interact with a limited number of
(chiefly national) sources of evidence and this involves a risk that they obtain only a
partial view of the evidence base.
In the absence of captive evidence infrastructures, we see a particularly profound
transition in the UK towards much smaller ministries that are trying to become more
embedded in the national and international research landscape, as an intelligent
customer, sharer and provider of evidence, both through formal channels (eg
comprehensive publication portals), as well as informally through closer ties between
analysts and policy makers inside the ministry and evidence providers outside of it. In
the presence of evidence-providing agencies and captive institutes, this internal
versus external divide, and consequent transition from large and closed to open and
embedded ministries is not as clearly evident in Finland or the Netherlands. But
despite these differences in overall context, the ways in which ministries aim to
respond to the dual pressure on evidence use are comparable, or at least offer useful
observations where approaches do diverge.
There is a broad trend from using captive evidence sources towards ‘marketisation’
The UK has had few government labs since the 1980s. At the other extreme, the EC
has access to the JRC for all purposes. In Finland and The Netherlands, some
ministries have labs or captive institutes while others do not. As a result, few if any
ministries have external evidence suppliers at the boundary between research and
evidence production that are fully captive. All make use of universities, consultancies
and where relevant research institutes as sources of evidence. In the UK, universities
are seen as much more legitimate and credible than other sources; elsewhere there
appears not to be a status hierarchy.
Broadly, our interviewees indicated that while all research and study budgets were
under some pressure, reductions were most severe in the area of long term,
proactive work – presumably because the effects of reduced funding (in terms of
ministries’ declining ability to identify and tackle longer term policy issues) were not
visible in the short term. The pressure appeared to be lower where there were
relevant government labs – presumably because their established positions in the
state budget makes their budgets defensible, while budgets for long term, external
research or studies have fewer defenders inside the budgeting process.
The Place of Research-based Evidence in Policymaking
13
 UFU, Alm.del - 2015-16 - Bilag 130: Baggrundsrapporter fremsendt den 7/4-16 fra Danmarks Forsknings- og Innovationspolitiske Råd
1618648_0024.png
A corollary of marketisation is that no-one takes responsibility for the health and
sustainability of the supply side. Norway is outside the scope of this study, but it is
noteworthy that Norwegian ministries are required to take account of the continued
availability of supply, given the small size of the Norwegian market.
However, the possible relationships with evidence suppliers have various
imperfections
Relations with ‘captive’ labs or institutes enjoy the benefits of long term planning but
risk ‘capture’ of the research agenda by the institute. There is a similar risk where
policymakers become embedded in academic networks, even if they then do not pay
for the evidence they obtain. Our interviewees did not discuss risks associated with
consultancies except in the case of Finland, where the small size of the domestic
market means that the same small number of good firms tend to win most of the
contracts, again leading to a risk that ideas are not refreshed. Framework contracts
are increasingly used to simplify and speed up procurement. Again, while our
interviewees did not discuss the disadvantages of such arrangements, it is worth
pointing out that they can involve the same problem of lack of renewal of ideas and
that they inherently involve a trade-off for the ministry: easier procurement against
access to a more limited set of suppliers. In the Finnish case, government has
deliberately sought ‘different’ sources of advice from a range of external committees
in recent years.
Technical councils and standing committees tend not to be central to evidence
collection – rather they are more important for legitimation
Standing scientific councils or committees of experts are used to a degree in the UK
and the EC, less so in the Netherlands and Finland. Their main uses are quality
control and legitimation, rather than the provision of evidence. The EC more often
uses
ad hoc
expert groups to provide evidence and advice.
Surprisingly, high-level councils that potentially could provide policy coordination
across government as a whole appeared to have little effect on policymaking at
individual ministry level
There is a second and higher level of policy council such as the Finnish Research and
Innovation Council or the Dutch WRR that answers to government at a high level –
usually the Prime Minister. The UK maintains the UK Council for Science and
Technology, which answers
ad hoc
questions from government about science and its
implications for policy. Curiously, these organisations’ impact was barely visible to
our interviewees.
14
The Place of Research-based Evidence in Policymaking
 UFU, Alm.del - 2015-16 - Bilag 130: Baggrundsrapporter fremsendt den 7/4-16 fra Danmarks Forsknings- og Innovationspolitiske Råd
1618648_0025.png
International sources of evidence tend to be used for benchmarking and
background understanding of the international context, rather than for generating
specific policy-focused evidence.
The sources of international evidence mentioned by interviewees are almost only
international organisations such as the OECD, European Commission, World Health
Organisation and so on. These tend to collate statistics and to examine practices
rather than to provide end-users with custom studies relating to specific policy
initiatives. Policymakers in one country tended not to use research institutes or
consultants from another. The EC is not an exception, in the sense that it regards the
EU as its ‘home’ country and buys little evidence from outside the Union.
2.3
Absorptive capacity and the enlightenment model
In general, ministries are moving towards an ‘enlightenment model’
2
, in which they
have high internal capacity to acquire and generate evidence, in part by engaging in
the wider evidence community in an open way
Ministries’ absorptive capacity has been increasing, partly in response to shrinking
ministry size and research budgets
Pressures to reduce staffing in ministries result in a need for remaining staff to be
more capable. In relation to evidence acquisition, further pressure on budgets mean
that policymakers have to do more with less money. An additional pressure in the EC
is the longer-term process of agencification, in which operative tasks are being
moved out to executive agencies leaving directorates general to be more exclusively
focused on policy. As a result, within ministries, the ‘distance’ between individuals
charged with analysis and evidence collection on one hand and formulation and
implementation of policy on the other has decreased in recent years.
These changes mean that ministries make efforts to ensure that staffs are generally
literate in relevant areas of science and evidence collection. This does not equate to
ministries staffed by scientists but instead to ensuring some degree of knowledge of
both the policymaking world and the research and science world. It affects
recruitment criteria to some extent – with increased willingness to take on non-
generalists and to value research capability. Some ministries ensure they have direct
access to scientific literature and evidence bases more generally, as well as tools to
conduct reviews of evidence internally and at short notice.
2
We discuss various models of the relationship between research-based evidence and policymaking in Chapter 4. In
the ‘enlightenment model’ policymakers are well versed in both research and the policy process, and are able to
make good use of existing knowledge as well as to commission new studies where they can see knowledge gaps
The Place of Research-based Evidence in Policymaking
15
 UFU, Alm.del - 2015-16 - Bilag 130: Baggrundsrapporter fremsendt den 7/4-16 fra Danmarks Forsknings- og Innovationspolitiske Råd
1618648_0026.png
In the UK, the long-standing model of the Government Chief Scientific Advisor has
been expanded to create additional Scientific Advisors for each ministry, who can
offer rapid advice and ‘know-who’ when required. We view this as a particular
extension to the absorptive capacity of the ministries.
Systems with strong government labs or close relations with institutes (Finland and
The Netherlands) appear to have a lower density of research-capable people in the
ministries than those who have to collect evidence primarily through external
organisations.
Increased absorptive capacity and integration of policymakers into the wider
research community goes hand in hand with greater transparency – especially in
terms of publishing the evidence used by government
Open publication of studies and evidence conducted or commissioned by ministries
is not only viewed as part of government’s accountability but also viewed as
necessary if the civil servants are to involved in the wider pattern of knowledge
exchange within the research community. Open and user-friendly access to data and
reports also reduces the risk of duplication of effort. I the context of a wider policy of
making government data available to citizens, the UK is creating a government-wide
Web portal for all relevant evidence while The Netherlands is in the process of
developing a protocol to specify what evidence should be put into the public domain.
2.4
Quality and credibility
For the most part, ministries are more interested in the quality of evidence than in
who produced it
The policymakers we consulted are confident in their own ability and that of their
colleagues to assess the quality of evidence, no matter what its origins. For their
own purposes, therefore, they were happy to use whatever evidence they could
acquire. In the UK however, there was a clear preference for using evidence from
universities in public discussion. The universities have higher status than others and
are believed to be more independent and objective so the UK ministries tend to
believe that academic evidence is inherently more persuasive than evidence from
other sources.
Policymakers generally make their own judgements about the quality of evidence
available to them. In some cases they may use a committee for legitimation
Quality control of evidence is rarely formalised and most often relies on the expertise
of people in the ministry. It is generally done
ad hoc,
if it is done separately from
examining the evidence itself at all. In such cases an official might ask a colleague or
an independent (often academic) peer to look at the evidence. Generally, the final
defence against poor quality is publication: evidence used must be able to withstand
public scrutiny. Only in a small number of cases would evidence be subject to formal
16
The Place of Research-based Evidence in Policymaking
 UFU, Alm.del - 2015-16 - Bilag 130: Baggrundsrapporter fremsendt den 7/4-16 fra Danmarks Forsknings- og Innovationspolitiske Råd
1618648_0027.png
scrutiny, primarily in health ministries where there are standing committees that
consider various scientific and policy areas or agencies like the UK’s National Institute
for Health and Care Excellence, which produces clinical guidelines for the healthcare
system.
Policymakers want to experiment with new types of evidence though so far this is
at an early stage
Several of the ministries we contacted identified a trend towards experimentation as
an aid to designing and evaluating policy instruments. There is a particularly strong
interest in randomised control trials. However, only the UK’s Department of
Education said it is actually conducting or sponsoring such trials. A number of
ministries referred to the desirability of setting up cross-government policy
laboratories and mentioned the Danish MindLab as a model.
The rise of ‘big data’ presents an opportunity to generate significantly improved
capacity for monitoring and analysis. This is also at an embryonic stage, with
policymakers beginning to explore the possibilities that this might entail. Most often,
the intention is to combine ministries’ existing programme and policy data with other
larger data sets, eg on the life course of programme participants (be they individuals,
families, groups or companies). The hope is that this can lead to more robust
evidence about policy needs and effectiveness while also being ‘minimally invasive’
in the sense of avoiding the need to contact people or companies. While there is
policy interest in big data there are also substantial issues regarding data protection
law and the amount of experience with using big data for policy or evaluation
remains limited.
2.5
Evidence and the policy cycle
Only the European Commission consistently ties evidence collection to a formal
policy cycle. At the national level, ministries vary greatly in the extent to which
they use such a cycle
Policymakers disagree about the usefulness of a formalised policymaking process, or
a policy cycle, as a framework for triggering the collection and use of different types
of evidence. The EC is the only case where consistent use is made of a policy cycle,
from road mapping and options assessment, to ex ante impact assessment, policy
monitoring, interim and ex post evaluation, with foresight additionally being
increasingly systematised, and expert panels used to support parts of these
processes.
Outside the EC, this type of formalisation was highly variable, with no clear patterns
between countries or ministries. Some ministries make frequent use of the policy
cycle and see it as a helpful reference point, whilst other show low awareness of it.
The most common argument against formalisation at this level is that policymaking is
more complex and less predictable than standardised models imply and that it is
The Place of Research-based Evidence in Policymaking
17
 UFU, Alm.del - 2015-16 - Bilag 130: Baggrundsrapporter fremsendt den 7/4-16 fra Danmarks Forsknings- og Innovationspolitiske Råd
1618648_0028.png
therefore more important to build capacity in ministries to be responsive to a wide
range of different evidence triggers and needs that do not necessarily follow a
formula.
Guidelines for individual evidence-related activities are available to most
policymakers and appear most influential where they are used system-wide rather
than being specific to a ministry. But policymakers caution against their heavy-
handed application
There are significant differences between and within countries in the extent to which
they formalise and codify guidelines for individual evidence-related processes. The
EC has overall the most stringent and comprehensive frameworks for commissioning,
conducting, analysing, overseeing and implementing evidence for policymaking, and
additionally a high level of consensus that these frameworks should be followed
most of the time. At several individual national ministries, there is a lower presence
of handbooks, guidelines and other codified procedures; these are not always viewed
as being especially helpful. System-wide guidelines such as those produce by the UK
Treasury and the EC tend to be used.
Codified procedures are generally considered to be either useful or essential
•
For policy and programme evaluations, they effectively augment the legitimacy-
granting function of evaluations themselves: to a large extent, evaluations are
designed to ensure accountability and to demonstrate that ministries’ actions are
justified and effective. Adding a set of rules to ensure these evaluation follow a
commonly agreed standard, and ensuring that standard is adhered to enhances
evaluations’ perceived function to do so
For procurement: the selection process for an external provider of a particular
research or evaluation project needs to be codified, in order to instil confidence
that the most capable providers are involved in evidence provision
•
There is in some cases a formal guideline for impact assessment. This is welcomed
where a standardised tool or process is in place that is felt to make the impact
assessment procedure easier. In other cases, the need consistently to demonstrate
and forecast impacts is viewed as a burden, at times to the point of discouraging
interaction with analysts or further pursuing ideas
Beyond these points, there is considerable variation in the use of formalisation. Our
interviewees tended to argue that there is a need for a balance to be struck between
formalisation where this is helpful and a light-touch approach where formalisation
would create unnecessary administrative steps.
18
The Place of Research-based Evidence in Policymaking
 UFU, Alm.del - 2015-16 - Bilag 130: Baggrundsrapporter fremsendt den 7/4-16 fra Danmarks Forsknings- og Innovationspolitiske Råd
1618648_0029.png
3.
National and sectoral specificities
This chapter describes some of the influences of culture on the different ways
evidence and policy are connected. We point out special and interesting features of
the countries considered. Next we identify the extent to which ministries in different
countries that work in the same sector also behave in the same way, with respect to
generating and using evidence.
3.1
Specific aspects of national behaviour
Culture makes a difference to how people behave and the transportability of
evidence practices. It seems to have particular importance in relation to trust
Each country studied has its own specific context, with structures, systems and
attitudes rooted in culture, history and country-specific developments of the policy
landscape. We therefore find many key differences to be attributable to known
divisions between national organisational cultures (see eg Hofstede 2001), with
interviewees in the UK often attributing significant importance to key individuals,
both in the form of the Chief Scientific Advisors, but also in terms of particular
individuals in particular ministries bringing about changes due to their personal
interest. Moreover, in the UK we find more openly confrontational relationships
between policy and politics, as well as between policy makers and scientists. In
Finland and the Netherlands we find less of these features, and instead still
comparatively significant captive labs and research institutes, with much reliance on
public agencies and higher levels of trust, emphasised by fewer concerns about
quality or politically driven bias of evidence. The EC, due to its position in relation to
member states, exhibits caution around the subsidiarity principle and the need to
demonstrate European added value for all activities, which decisively shapes its
endeavours. This is at least one factor that has led to an especially sophisticated
policy cycle and high levels of formalisation.
UK evidence use is influenced by the presence of Chief Scientific Advisers and an
increasing focus in government on demonstrating the societal impacts of policy
The UK has had a long-standing tradition of employing a Chief Scientific Adviser, with
direct access to the Prime Minister. Today this General Chief Scientific Adviser heads
the 80-strong Government Office for Science while each ministry has a specific
adviser (supported by a deputy, an official and a personal assistant). Each ministry
therefore has two channels for evidence: the scientific adviser who addresses short-
term needs and contributes to the overall strategy; and other analysts who – as in
continental ministries – commission and collect studies and research. Most CSAs are
professors, providing links into the relevant academic communities.
The Place of Research-based Evidence in Policymaking
19
 UFU, Alm.del - 2015-16 - Bilag 130: Baggrundsrapporter fremsendt den 7/4-16 fra Danmarks Forsknings- og Innovationspolitiske Råd
1618648_0030.png
The government’s so-called ‘impact agenda’ has significantly increased the research
community’s desire to work with policy-relevant questions. Research Councils that
provide external research grants to universities now require that proposals explain
the societal relevance and value of research and the national performance-based
research funding system that allocates institutional research funding to the
universities (‘the ‘Research Excellence Framework’ or REF) allocates funding partly on
the basis of universities’ claims about the societal impact of past research. The
academic community sees policy influence as a key demonstration of relevance and
is therefore well motivated to engage with ministries’ evidence needs, even when
they are not paid to do so.
Dutch ministries have a uniform long-term and strategic approach to collecting
evidence for policy
All Dutch ministries have a ‘department for knowledge’ (or ‘strategy’) within the
Office of the Secretary General as well as a ‘knowledge coordinator’ whose job it is to
link the central department to the evidence needs of the other parts of the ministry.
This means that in principle the ministry benefits from having strategic intelligence
distributed to the relevant parts of the ministry while also maintaining a central,
strategic view on research and evidence.
Each ministry is required to produce a research strategy. Most have long-standing
links to institutes or government labs, which themselves have medium-term research
strategies so there is strong potential for coordinating the system of principals and
agents. There is also a risk that this causes a degree of inbreeding and lock-in – but it
should be noted that the ministries obtain evidence from a wider range of sources
than just the labs and institutes.
Finland has radically centralised the collection and funding of at least some of the
evidence needed for policymaking
Finland has traditionally been respected for the ability of the Research and
Innovation Council to coordinate policy across government. The Council has recently
been strengthened by moving its administration to the Prime Minister’s Office. The
office also administers a new fund (TEA) which finances evidence for policy. It works
by prioritising topics suggested by the various ministries and then launching a
competitive call for research proposals.
A second Finnish innovation has been to cut some of the government labs’
institutional funding and to channel it through a new division of the Academy of
Finland (STN). Its role is to fund longer-term policy relevant research anticipating the
demand for research-based evidence and supporting longer-term policymaking. Both
TEA and STN are said to be responses to overall reductions in ministry research
budgets.
20
The Place of Research-based Evidence in Policymaking
 UFU, Alm.del - 2015-16 - Bilag 130: Baggrundsrapporter fremsendt den 7/4-16 fra Danmarks Forsknings- og Innovationspolitiske Råd
1618648_0031.png
The European Commission leans more heavily on formalised processes and the use
of expert panels in evidence collection than is the case at national level
Formal requirements for generating evidence to support the policy cycle have
increasingly been imposed at the EC since the administrative reforms of the late
1990s. Officials tend to describe these requirements as a response to constant
scrutiny of the Commission by the Member States. Different units within the EC often
handle different aspects of evidence collection, so foresight tends to be separate
from evaluation, which in turn is separate from strategy and the programme design.
It may be that a strong policy cycle has the additional benefit of counteracting this
fragmentation. The EC is also able to make use of expert panels to supervise studies
and to give policy advice to the Commission than is the case at the national level.
These mechanisms provide legitimation but may only be affordable at the EC’s
considerable scale.
3.2
Sector-specific aspects of evidence use
Sector ministries are strongly influenced by the national administrative tradition
within which they operate. There are nonetheless similarities that result from the
characteristics of the sectors themselves.
As a broad conclusion, our findings show that each type of ministry is influenced in its
concerns and activities around evidence use by the academic disciplines most closely
aligned to its remit. By this we mean that for instance, we find a higher presence of
natural scientists and concern about collecting natural scientific evidence in those
types of ministries whose activities relate to those disciplines, notably Environment,
and to a lesser extent Health and Transport. Findings on these sectoral differences
can most often be viewed in this context. Key observations in this regard are
•
Environment ministries have a stronger presence of natural scientists internally.
They tend to make more use of captive laboratories for monitoring of
environmental data. Given the resources necessary to monitor the natural
environment, they tend also to have comparatively high budgets for research
and evidence collection.
Foreign ministries tend to have smaller budgets. They also draw heavily on
country experts and personal connections, as well as think tanks and other
sources outside their own country. More generally, ‘evidence’ in foreign affairs
ministries more often approximates to ‘intelligence’.
Given the clear differences between foreign and domestic policy, foreign affairs
ministries present somewhat of a special case: time constraints in the context of
immediate policy decisions are less common here, with longer term goals and
eventual persuasion and achievement of multilateral agreements and policies the
focus – the meaning of ‘policymaking’ often differs for these ministries, making
•
•
The Place of Research-based Evidence in Policymaking
21
 UFU, Alm.del - 2015-16 - Bilag 130: Baggrundsrapporter fremsendt den 7/4-16 fra Danmarks Forsknings- og Innovationspolitiske Råd
1618648_0032.png
comparison especially problematic. Foreign affairs ministries also have a
relatively low level of formalised guidance for the collection and analysis of
evidence.
•
Innovation ministries, typically charged with economic growth and development,
tend to have a higher presence of economists and statisticians, and evidence
procured and used by these ministries is more focused on these areas. Calls for
greater understanding and use of Big Data most often came from these.
Health ministries tend to have high budgets for research and evidence, though
significant amounts of this are not directly for policy-purposes: these budgets
also finance a lot of R&D, clinical trials, etc. Sometimes developments in care and
treatment practices influence policy, so the lines between policy and non-policy
research are less clear than elsewhere. Systematic reviews – a technique
prominent in medical research – have a high prevalence in health ministries. But
given responsibilities for public health and health outcomes more generally, we
find a broad inter-disciplinary mix of expertise and techniques in this sector.
•
22
The Place of Research-based Evidence in Policymaking
 UFU, Alm.del - 2015-16 - Bilag 130: Baggrundsrapporter fremsendt den 7/4-16 fra Danmarks Forsknings- og Innovationspolitiske Råd
1618648_0033.png
4.
How do our findings compare with existing knowledge?
The use of research-based evidence in policy formulation has been much studied
over the years – mostly from the perspective of researchers wanting to understand
how to increase the policy impact of their work
3
. The focus of this study is the
opposite of that: it aims to understand how demand for evidence arises and is
satisfied, in relation to policymaking. This section summarises key discussions in the
literature on evidence-based policy and indicates how our study findings compare
with them.
The New Public Management movement has been an important force encouraging
greater use of research-based evidence in policymaking. The movement has been
influential in many countries but especially in the Anglo-Saxon ones: UK, USA,
Australia and New Zealand. Both the Blair and the Obama administrations have been
particularly influential in promoting the idea that policy should more strongly be
based on ‘what works’ rather than purely on ideology, and this has provided an
important impetus towards increased use of evidence during the past two decades.
A very clear effect of the New Public Management has been growing use of
evaluation in making and implementing policy and an increased emphasis on
integrating evaluation with a ‘policy cycle’, in which policymakers follow an orderly
process of researching societal problems, designing interventions to correct these
problems, monitoring progress and evaluating outcomes. A growing number of
administrations have also published handbooks or guidelines that specify how the
policy cycle works and in many cases also guidance about preferred types of
evaluation methods, generally emphasising economic or econometric ones.
This study suggests that, under the influence of austerity and declining research
budgets, interest in the policy cycle has diminished compared with the peak of
interest, which was about ten years ago. While a strong focus on evaluation remains,
use of evidence for other policy purposes is becoming more
ad hoc.
At the same
time, ministry staff is becoming more capable in relation to using research.
Weiss (1979) produced a typology of evidence use that remains influential in studies
to this day. It shares several features with accounts of the use of research results in
industrial innovation. Her types are
3
This section of the report summarises key issues we identified in the literature. See Appendix G for the full
literature review and Appendix G for the bibliography
The Place of Research-based Evidence in Policymaking
23
 UFU, Alm.del - 2015-16 - Bilag 130: Baggrundsrapporter fremsendt den 7/4-16 fra Danmarks Forsknings- og Innovationspolitiske Råd
1618648_0034.png
•
The
knowledge-driven model,
where research uncovers opportunities for better
policies and these are adopted by policymakers (similar to the ‘supply push’ or
‘linear’ model in innovation)
The
problem-solving model,
where societal problems trigger research about
ways to find solutions, and these results are then transferred from research to
practice (similar to the ‘demand pull’ model in innovation)
The
interactive model,
where research-based evidence is used if and when
societal needs and research-based policy opportunities coincide (similar to the
‘coupling’ or ‘chain link’ model in innovation)
The
tactical model,
where government and stakeholders make opportunistic use
of evidence that supports their pre-existing positions. This is sometimes jokingly
referred to as policy-based evidence
The
enlightenment model,
in which policymakers are well versed in both
research and the policy process, and are able to make good use of existing
knowledge as well as to commission new studies where they can see knowledge
gaps
The
intellectual enterprise of society,
where policymaking and evidence
production interact, often led by fads or fashions, and where social scientists are
able to exploit interaction with policy to focus on generating research funds
•
•
•
•
•
These models are probably all in use in various places but we saw no evidence that
any one of them is dominant. However, there is a trend towards the enlightenment
model with ministries and policymakers becoming more embedded within the wider
process of generating knowledge in society.
Policymakers are supported outside their ministries by a range of different ‘boundary
organisations’ that generate evidence at the boundary between the scientific and
policymaking systems. Traditionally, government laboratories (labs) have played this
role but the literature implies that more or less independent ‘think tanks’, research
institutes and university groups are increasingly important.
Our study suggests that there is a trade-off between using government labs and
other external sources. The labs can be rich sources of evidence but risk taking over
the research agenda, often focusing on longer-term knowledge needs and potentially
under-supplying short-term knowledge needed for more immediate policymaking.
Administrations that make little use of government labs need stronger internal
capabilities to generate and use knowledge in order to operate wise knowledge
acquisition strategies. But even those that use labs have to maintain enough internal
capability to make sure the people in the labs serve the overall policymaking interest
rather than their own, internal research interests.
Innovation researchers have long emphasised the need for firms to have ‘absorptive
capacity’ (the ability to identify, acquire and exploit relevant external knowledge) in
24
The Place of Research-based Evidence in Policymaking
 UFU, Alm.del - 2015-16 - Bilag 130: Baggrundsrapporter fremsendt den 7/4-16 fra Danmarks Forsknings- og Innovationspolitiske Råd
1618648_0035.png
order to innovate. Students of policymaking make a similar observation. In the face
of declining external research budgets and the need not to rely too much on
boundary organisations, ministries are driven to increase their absorptive capacity
and therefore to move towards the enlightenment model. Our study suggests that
most ministries see absorptive capacity as increasingly important and try to
strengthen it, often in the face of budget pressure for reduced staffing levels. As a
result, ministry staff tend to become fewer but better qualified.
The literature review also implies that there are limits to what can be known in a
strong scientific sense about evidence-based policymaking. Nobody has been able
empirically to tackle the most central and obvious question in the field: Does the use
of research-based evidence produce better policy? That it does so is an article of
faith, at best based on experience and anecdote. The question does not seem to be
amenable to the kind of statistical treatment such as the use of econometrics or
counterfactual analysis, which advocates of evidence-based policy tend to
recommend.
In recent years, researchers have started to consider evidence-based policymaking in
the context of complexity. Policymaking is done in complex social systems and one
of the defining characteristics of complex systems is that some of their properties are
‘emergent’: namely, that they cannot necessarily be predicted by considering the
inputs and processes in the system. Policymaking appears also to be heavily context-
dependent. It may be the case that the context, in which policymakers try to use
evidence can itself be an overwhelming determinant of success. Our interview
partners are nonetheless unanimous in regarding a preference for research-based
evidence over other inputs as a precondition for making good policy.
A recurring theme in the literature is ‘dynamic inconsistency’ between the needs and
expectations of the political and short-term policymaking systems on the one hand
and the long-term nature of research and knowledge generation on the other.
Politicians need to win the next election. Policymakers have sometimes to help them
to do this, often by making or proposing policies that can immediately be
implemented in the face of short-term needs. Designing a headline-grabbing new
policing policy or the national response to an epidemic has to be immediate and
therefore based on existing knowledge. There is no time to wait for new research to
be commissioned and performed. At the same time, the ministry has to secure the
knowledge base it will need in order to address foreseeable future challenges.
Evidence-based policymaking therefore depends upon the ability to juggle short- and
longer-term needs that may actually be incompatible. This study suggests that in the
face of austerity it is hard to fund the longer-term work (because reducing it does not
produce effects that are visible in the short term). Some ministries try to
compensate for this by doing more foresight or embedding themselves better in
knowledge-producing networks, but many suggest that cutting longer-term work
reduces their ability to handle new policy challenges and is therefore building up
trouble for the future.
The Place of Research-based Evidence in Policymaking
25
 UFU, Alm.del - 2015-16 - Bilag 130: Baggrundsrapporter fremsendt den 7/4-16 fra Danmarks Forsknings- og Innovationspolitiske Råd
1618648_0036.png
5.
Good practice in the use of research-based evidence for policymaking
In this final chapter, we make ten simple points of good practice for policymakers in
the use of research-based evidence for policymaking. The points are based on the
practices we have observed in the course of doing this study and on what we
understand as the general views of the policymakers interviewed. Together the
points show both the directions in which policymakers focus their attention and what
they consider are the most valuable practices in the use of research-based evidence
in their own organisation and across government.
1. Be as evidence-based as possible – but no more. There is broad agreement
that policy should be made on the basis of research-based evidence,
wherever possible. Sometimes the needed evidence is not all there.
Sometimes there are political or practical considerations that have to be
taken into account. Policy still has to be made if government is to achieve
anything but the more closely the policy is driven by the evidence, the better
policymakers feel its chances are of reaching its objectives.
2. Use foresight and other techniques for thinking about the future as ways to
anticipate coming policy needs – and therefore the kinds of evidence that
will be needed to support them.
3. Devise and invest in research strategies that generate evidence that will be
needed in the longer term as well as in the immediate future.
4. Ensure that ministries are staffed with a significant proportion of people who
can specify the need for research as well as to make use of external inputs in
order to generate evidence for policymaking
5. Have ‘evidence champions’ – they might look like Chief Scientific Advisors;
they might look like ‘departments for knowledge’ or ‘knowledge
coordinators – to promote and coordinate the generation and use of
evidence for policymaking
6. Create funded arrangements for generating and sharing evidence to address
cross-ministry problems
7. Maintain long-term links with organisations like universities that work at the
boundary between research and policy but do not let these become
monopolies – you also need impulses for change from a wider set of
institutions (including foreign ones) working in competition
8. Publish evidence so that policymaking is transparent and others can quality-
assure as well as re-use the evidence you employ
26
The Place of Research-based Evidence in Policymaking
 UFU, Alm.del - 2015-16 - Bilag 130: Baggrundsrapporter fremsendt den 7/4-16 fra Danmarks Forsknings- og Innovationspolitiske Råd
1618648_0037.png
9. Use a ‘light touch’ policy cycle, which suggests good practice guidelines for
collecting and using evidence but which is rather more firm about the
requirement to evaluate interventions both
ex ante
and
ex post
10. Be prepared to experiment and learn about new intervention designs and
ways to develop evidence
The Place of Research-based Evidence in Policymaking
27