Evidence-based narratives in European research programming

Abstract

The article introduces and exemplifies the approach of evidence-based narratives (EBN). The methodology is a product of co-design between policy-making and science, generating robust intelligence for evidence-based policy-making in the Directorate General for Research and Innovation of the European Commission (DG RTD) under the condition of high uncertainty and fragmented evidence. The EBN transdisciplinary approach tackles practical problems of future-oriented policy-making, in this case in the area of programming for research and innovation addressing the Grand Societal Challenge related to climate change and natural resources. Between 2013 and 2018, the EU-funded RECREATE project developed 20 EBNs in a co-development process between scientists and policy-makers. All EBNs are supported with evidence about the underlying innovation system applying the technological innovation systems (TIS) framework. Each TIS analysis features the innovation, its current state of market diffusion and a description of the innovation investment case. Indicators include potential future market sizes, effects on employment and environmental and social benefits. Based on the innovation and TIS function analyses, the EBNs offer policy recommendations. The article ends with a critical discussion of the EBN approach.

Introduction

Prediction is very difficult, especially about the future.¹

When dealing with future innovations, forward-looking research as well as policy-makers face a dilemma, for it is the nature of both, the future and innovations, to represent uncharted territory. The future is by definition unknown and, in most cases, there is probably a reciprocal relation between innovation and knowledge: The more we know about an innovation, the less it can be considered to be an innovation. At the same time, the digital post-truth era provides permanent stream of data and information about the present, the past and possible futures that overwhelms non-experts and experts alike.

The research of this article was originally organized as part of the ‘REsearch network for forward looking activities and assessment of research and innovation prospects in the fields of Climate, Resource Efficiency and raw mATErials’ (RECREATE) funded under the 7^th EU framework programme for research, technological development and demonstration (FP7). RECREATE was a coordination action with the mandate to collect and analyse information about medium- and long-term research and innovation trends and prospects. The aim of RECREATE was to support the European Commission to create a research agenda for the Horizon 2020 framework programme of the European Union with a focus on climate change, raw materials and resource efficiency. For this purpose, the RECREATE network undertook a number of activities including the development of a scoreboard and different scenarios.

From 2013 to 2018, the RECREATE researchers in collaboration with staff of the European Commission’s Directorate General for Research and Innovation (DG RTD) have co-designed an approach, which we are referring to as evidence-based narratives (EBN). The approach aims to reduce complexity for effective communication with decision-makers under the condition of uncertainty. As we have received encouraging feedback from practitioners—from public administrators as well as from corporate executives—we believe it to be worthwhile to present it to the readers of the European Journal for Future Studies.

The following sections present the theoretical background against which RECREATE has developed the EBNs. The method chapter will give detailed description of the co-design process. Finally, the article will conclude by outlining and discussing the elements, advantages and limitations of EBNs as a tool for science-policy communication.

Policy-making and uncertainty

In the continuum from certainty to uncertainty, quantitative assessment methods such as computer-based modelling seem to be appropriate in cases of certainty and calculable risks [1] (Table 1). However, most policy questions related to innovation and sustainability transitions are on the other side of the spectrum dealing with unstable, globally connected systems far from certainty. Therefore, any assessments about future innovations seem to require a generation of intelligence, which does not reproduce and rely on certainty, but which is fit to deal with uncertainty.

Table 1 Conditions for decision-making and information-base [2]

In the following sections, we will outline how science can make sense of future developments in an uncertain and complex world and how individuals, including policy-makers, deal with it. As there is a significant discrepancy between both approaches, there is a need for a science communication that can bridge the gap.

Quantitative and qualitative methods to approach the future

For venturing into the uncertain, scientists use quantitative and qualitative approaches. The most common quantitative approach is forecasting [2], which attempts to describe in some detail a hypothetical sequence of future events by looking at trends in past and present data. Any forecasting, scenario analysis will draw on certain assumptions and is based on plausible conclusions [3]. Backcasting, in contrast, starts with the (normative) definition of a desirable state in the future (e.g. the 1.5 °C goal of the Paris Agreement) and identifies possible pathways to this future [4, 5]. Both approaches are difficult and constrained when it comes to predicting future innovations: Forecasting relies on past and present data (time series, longitudinal data, etc.), which might be available for the societal challenge to be addressed but, in most cases, will not be available for future innovations. Backcasting can be an effective planning tool for innovation, but it is only as good as their assumptions on innovations, their constraints and their interdependence [3].

Quantitative techniques based on scenario calculations are usually complemented or embedded with qualitative argumentation ideally based on rules and standard approaches that improve transparency and replication [6]. An example of such a qualitative argumentation to assessing the impacts of policy decisions on the environment is the so-called evidential reasoning (ER) [7], which aims at identifying evidence and drawing logical conclusions as a key to problem solving, learning and critical thinking. In line with critical rationalism, ER defines positive evidence, if there is no indication for the opposite (falsification), and it is therefore considered to be true. Another prominent example is the so-called case-based reasoning [8], which—in plain words—solves new problems based on the solutions that solved similar problems in the past. Case-based reasoning is a prominent type of analogy solution creation and a fundamental component of the decision-making process of individuals as we will see in the next section of this article. In line with the classical model of argumentation, this kind of reasoning should be supported by practical arguments rather than absolute theories [9]. The claims of the argumentation should be based on evidence. The warrant, which logically connects the evidence to the claim, should be backed up and defended against potential rebuttal and eventually be further qualified [10].

Simple heuristics

Cognitive research suggests that individuals to deal with complexity by applying simple heuristic and rules of thumb [11, 12]. The human mind is generically adapted to uncertainty [1]: Under the condition of complexity and uncertainty, individual decision-making is not primarily guided by statistics, calculations and complex modelling, but by simple rules of thumb and ‘gut feeling’, which is the result of individual and biological conditioning [13]. For example, executives of large organisations (e.g. in the private sector) tend to rely on their gut feeling. This is in contrast with the general preference in policy-making for legitimizing collective decision-making with ‘objective’ data. Gigerenzer shows that in practice the discrimination of simple heuristics in collective decision-making can ultimately lead to second best solutions. Another, quite frequent phenomenon in public policy-making is the ex-post construction of evidence for legitimizing political decisions, which were originally based on intuition. Gigerenzer’s findings are of high practical value for evidence-based policy-making, because they explain how complexity in science and policy-making could be reduced in a way that is more appropriate to the human cognitive condition. The necessity to reduce complexity by introducing simple rules of thumb seems to be counter-intuitive, if not avoided. The more attractive intuitive assumption and prevailing tendency in evidence-based policy-making seems to be that complex problems need to be addressed with increasingly complex quantitative methods, models and tools often in combination with complex reasoning and theories.

The way how simple heuristics could find their way into science and policy-making could be by the construction of narratives.

Narratives

[N]arrative is the preferred heuristic employed by all for the purposes of making sense of the world […]. In plain language, people tell and remember stories ([14], p. 13).

Research on the use and relevance of narratives is based on the understanding of the ‘homo narrans’ [15, 16], as a ‘storytelling animal’ [17], who learns and understands in a narrative mode [18] by selecting and ordering facts to give them meaning. All human action depends on how this process unfolds in narrative patterns that create meaning and order ([19], p. 23).

The human need for narratives is well reflected in the scientific literature [20,21,22,23,24,25,26]. In addition to the discussion of philosophical, psychological and anthropological aspects of narratives, different authors have proposed the use of narrative for analytical purposes in a number of scientific disciplines [27,28,29,30,31,32,33].

In principle, the term narrative describes the way specific items are put into a coherent story. Coherence and order are achieved by developing a plot and providing a temporal context, usually along the basic structure of beginning, middle and end [34,35,36,37]. Narrative can thus be distinguished from other forms of written or spoken text. The specific features that make narrative a particularly relevant form in the context of the social sciences are (1) relationality of parts, (2) causal emplotment, (3) selective appropriation and (4) temporality, sequence and place [38]. Applied to the context of research programming and the task of dealing with evidence and heuristics in transdisciplinary settings, we follow the systematization originally developed by Somers and Gibson [38], spelling out these narrative features for developing evidence-based narratives:

• 1. A narrative contains various elements that are related to each other in such a way that they form a coherent whole.

• 2. These elements are connected by a causal pattern, a logical course of action. They are given meaning insofar as they contribute to the development of the storyline—not because they fit into a chronological order or belong to specific categories (as for instance in a list or a strictly scientific text).

• 3. It can be shown that this type of emplotment in a coherent narrative necessarily means that items of evidence have been selected from a (theoretically) infinite number of potentially relevant items. It is important to note that in this way, narratives function as a means of interpreting and structuring reality.

• 4. A narrative structures evidence in time and space resulting in a specific (not necessarily chronological) sequence.

Purpose of EBNs at the science-policy interface

Narrative and heuristics cannot replace complex modelling in analysing dynamics of predictable systems, but for supporting innovation and sustainability transitions they offer in most cases an adequate basis for policy-development, at least in the early stages of the policy cycle. Thus, narratives have the potential of a strategy tool as boundary object [39] at the science-policy interface improving processes of translation and knowledge integration. Based on the recognition of the strategic potential as boundary object, we have developed the evidence-based narrative in a co-design process with the European Commission; a process we will describe in detail in the following section. Before diving into its evolution, we summarize the operational objectives (OBJ-x) of generating an EBN:

• (OBJ-1) to collect evidence about innovations, preferably from stakeholders that were involved in the implementation of the innovation in question

• (OBJ-2) to get a preliminary assessment of environmental and socio-economic impacts of the innovation

• (OBJ-3) to provide a systemic view on the functioning of the innovation system

• (OBJ-4) to draw policy recommendations for research and innovation policy.

Co-design of the method

To support narratives with robust evidence seems to be quite straightforward but the approach has evolved over time. The EBN method was subject to a process of co-design at the interface of demand from policy-making (i.e. DG RTD of the European Commission) and supply of scientific evidence from research (i.e. RECREATE). At the beginning of the process in May 2014, the director of Directorate I ‘Climate Action and Resource Efficiency’ at DG RTD formulated three political priorities, which would become the thematic focus of the EBN development:

• 1. Realising a circular economy through a systemic approach to eco-innovation (SEI)

• 2. Making Europe a world leader in nature-based solutions (NBS), and

• 3. Creating a market for climate change services (CIS).

DG RTD asked specifically for narratives, underpinned by robust evidence, which would allow to put research and innovation funding in relation to the primary political objectives of the Juncker Commission, i.e. to promote employment and economic growth after the financial crisis.

After internal discussions and meetings of the RECRATE researchers, the term ‘evidence-based narratives’ (EBNs) had been proposed to DG RTD, expressing the intention to support DG RTD in the development of narratives, with a link to a scientific evidence-base. After that the consortium members started to collect a large number of potential case studies. The following selection process was based on the discussion and interaction between the Commission services and RECREATE consortium, which had resulted in stressing the objective that the EBNs should make the case for an investment related to markets of the future, cost-benefit-ratios, a potential for upscaling at European scale, assessment of market size, effects on employment, etc. In brief, the intention was to frame the narratives primarily economically, while the creation of positive impacts on the environment were presumed as they had evolved within the context of challenge 5. The underlying assumption was this language would resonate best with the political priorities of the College of Commissioners. Other social or ecological considerations related to the Societal Challenge 5 appeared to be less of a priority.

The framing of narratives on innovations and their related expectations has an important role in shaping scientific and technological change with significant normative implications [40]. In the process of co-design of the method with the European Commission, the RECREATE consortium avoided normative choices by leaving the overall political framing to the European Commission. The scientists accepted the political framing and expectations of DG RTD as given hypotheses about the potentials of the different innovations which would be tested in the envisaged collection and processing of evidence. Thus, the EBN approach was not meant to legitimize the European Commission’s claims and expectations on relevant innovations, but to see whether these expectations (e.g. related to the political priorities such as growth and employment) would be sufficiently supported by evidence or not.

Between July and September 2014, the RECREATE consortium organized the case collection in a sequence of interaction loops with scientists and industrial partners. In September, about 90 proposals had been collected: 18 on nature-based solutions, 26 on climate information services, 45 on systemic eco-innovation. The evidence-base including their economic and environment potential, availability of quantitative qualitative evidence, scalability, the link to research and innovation as well as their level of maturity had been very uneven.

Eventually, the researchers convened a special meeting in which they selected best cases based on their expert judgement. Fourteen cases (Table 2, phase 1) were chosen; amongst them, 7 related to systemic approaches to eco-innovation (SEI), 3 to nature-based solution (NBS) and 4 to climate information services (CIS). The 14 cases spanned very different kinds of innovations, scope and settings, e.g. from technological to non-technological and social innovations, and from commercial to non-for-profit product and service development at individual, sector or community levels. As a common framework, the RECREATE team agreed to apply the logical structure of the technology innovation systems framework (TIS) [41] for collecting the evidence to support the narratives. The TIS framework was a natural choice since it encapsulates many of the desired narrative characteristics suggested by Somers and Gibson [38] to provide a systemic view on the functioning of the innovation system (OBJ-3). Hence as a tool, it imposes a well-defined and delimiting structure, it also frames interdependencies and causality and thereby a temporal—and arguably a spatial—sequence of things. The researchers translated the TIS framework into a standard template guiding the subsequent research.

Table 2 20 different EBNs as output of RECREATE activities between 2014 and 2018

The technology innovation system (TIS) framework is based on the idea that the analysis of the targeted dynamic innovation diffusion should focus on systematically mapping the activities that usually take place in innovation systems and finally resulting in the innovation diffusion. In technological environments, this framework has the power to broaden the view of innovators by taking into account ‘softer’ aspects that are required for an innovation to succeed and to be implemented. The collections of activities that are supportive of implementation of innovations are considered to be functions of innovation systems. An innovation system analysis is based on seven functions [41]:

• 1. ‘Entrepreneurial activities’ (EA) maps the level of concrete actions taken by new entrants or incumbent companies generating and taking advantage of new business opportunities. Possible indicators may comprise the number of new entrants, diversification activities of incumbent actors.

• 2. ‘Knowledge development’ (KDev) maps the system’s ability to learn, either by searching (research) or by doing (development). Possible indicators may comprise the number of R&D projects, patents or technology learning curves.

• 3. ‘Knowledge diffusion through networks’ (KDiff) maps the flow of information exchange within knowledge networks. Possible indicators may comprise number of workshops and conferences devoted to the specific innovation and other network activities.

• 4. ‘Guidance of the search’ (GoS) maps the selection from the results of the knowledge developing activities. Since financial resources are limited, strategic decisions by industry and government set foci guiding future investments and influencing the direction of change. Possible indicators may comprise targets set by industry or government and number of journal articles related to the specific innovation.

• 5. ‘Market formation’ (MF) maps the competition process with the embedded solution the innovation aims to replace or to change. Possible indicators may comprise the number of introduced niche markets, specific tax regimes and new environmental standards.

• 6. ‘Resource mobilization’ (RM) maps the financial and human capital resources that are needed for all the activities within the innovation system. A possible indicator may comprise funds made available for long-term R&D programs.

• 7. ‘Creation of legitimacy’ (CoL) maps the process of how the specific innovation becomes part of an incumbent regime or even overthrows it. This process is guided by advocacy coalitions, parties with vested interests in ‘creative destruction’ (Schumpeter). A possible indicator may comprise the rise and growth of interest groups and their lobby actions.

The same framework of system functions was used for all cases both technological and non-technological, though in some cases one or more of the functions would less relevant for the innovation considered (and vice versa). This was particularly true for the social (e.g. Table 2, no. 4) and non-technological innovations (e.g. Table 2, no. 8) that are related to larger innovation systems, which may or may not include technologies. In these cases, the individual innovation functions served more as thematic sources of inspiration for the author of the narratives than as comparable indicators, and in the end, this did not affect the quality of the individual narratives.

What is obvious from the descriptions of the seven TIS functions is that this innovation system analysis (ISA) framework seeks to develop indicators that enable a research to compare the functioning of various innovation systems, or to follow the historic development of an innovation system [41]. Furthermore, it was recognized that these system functions are interdependent and influencing each other. The development of one system function may lead to progress on another, thereby stimulating the whole innovation system. Thus, innovations often start with a limited number of functions that pull other system functions [41]. ISA can describe these ‘motors of change’ [41] and what is furthermore needed to bring the science and technology push motor to a market motor that is eventually required for full market uptake [42]. This extended assessment of the innovation system was not implemented. A deep dive into each and every case study would undoubtedly have led to more thorough information and understanding of what was needed to bring innovations further, but it would also have overstretched the available resources.

Co-development of results

After the goal of developing EBNs had resulted from a number of events, the understanding of an EBN changed throughout the development process shaped by the continuous exchange with the European Commission. In the following, the stages of the co-development process will be presented and how this influenced the EBN definition. The project team started with the following initial understanding of an EBN:

Evidence-based narratives are convincing stories underpinned by robust findings and data that serve the purpose of assessing potential benefits of investment in innovations that—once scaled up—offer favourable effects on the European socio-economic and environmental systems and can be used by policy makers to justify budget allocations as well as to inspire businesses to invest.

Phase 1: selection and development of first 14 EBNs

The first phase of the co-development process, which comprised the selection of cases as well as the development of a first batch of 14 EBNs, was concluded in January 2015. By then, the project consortium had received the comments of the DG RTD on those initial EBNs. From the comments, the project team could draw further insights on essential components of an EBN:

• First, EBNs should focus on emerging innovations that had not yet been discussed by the Commission services.

• Second, EBNs should be based on solid case studies including macro-data, trend analysis and prognoses on economic impacts in comparison to traditional products. These analyses should be based on desktop research as well as on interviews with key stakeholders.

• Third, EBNs should not be restricted to a description of the innovation but include a strong storyline referring to the role of stakeholders, to the decision-making process and financing structures, to barriers as well as success factors (e.g. OBJ-1).

The first set of potential cases that had been compiled was commented upon by the DG RTD, which indicated that it expected a focus on emerging innovations, which were not already known or financed by the Commission. It became clear that the cases themselves were appreciated but that a mere description of these cases and their environmental benefits was considered to be insufficient (e.g. OBJ-2). Ideally, the EBNs were expected to support the hypothesis of a ‘green growth’ in line with the prevailing political priorities of jobs and growth. DG RTD expected case studies and narratives solidly based on macro-data, with trends, analysis and projections that would justify the assessment of impacts.

Phase 2: review of five selected EBNs

The first review of the first 14 EBNs had been rather critical. The Commission commented that the narratives seemed to be simplistic, lacking a systemic approach and containing statements which were not sufficiently backed-up with evidence. The overall added value had been questioned. It had been recommended to broaden the evidence-base also by interviewing stakeholders (OBJ-1). Also, the expectation of policy recommendations was emphasised (OBJ-4).

During the second phase, the project team used those comments to adapt its operating definition of EBNs:

Evidence-based narratives are a tool to develop research and innovation policy recommendations underpinned by triangulated findings and data about selected innovations in the field of societal challenge 5.

Along the recommendation of DG RTD to concentrate on selected EBNs, the RECREATE team revised five EBNs of the first batch accordingly and asked the DG RTD to comment for a second time. The comments, made in March 2015, reflect that most of the initial points made by the commentators were successfully included in the revised versions. However, Commission responses still referred to a lack of systemic analysis and a detailed analysis of barriers and success factors.

Beginning of 2016 after another review, reactions were much more positive. The comments appreciated the density of evidence in terms of (quantitative data) as well as critical analysis. While the Commission stressed the importance of stakeholder opinions, the comments emphasised that in some cases micro level data might be helpful for policy-making. Eventually, the first batch of 14 EBNs was adopted as well as the 5 revised cases.

Phase 3: final round of EBN development

In the first half of 2016, a third and final ‘round’ of EBNs had been produced and reviewed. The general remarks and recommendations were much in line with earlier criticisms: Comments referred to the importance of novelty and originality of the innovation, the concreteness of information as well as to the critical reflection of the proposed innovations with regards to the existing policy framework. Interestingly, the commentators clearly stated that an otherwise strong EBN is allowed to question current policy preferences of the Commission.

This last batch of EBNs met simultaneously critical and positive responses of DG RTD staff, reflecting high and diverse expectations. On the one hand, the EBNs were expected to provide concrete information that can be directly operationalized. On the other hand, they were supposed to put the innovation into the context of macro- and micro-data as well as the complete policy framework. The comments reflect the principle challenge of reducing complexity outlined in the ‘Policy-making and uncertainty’ section while still meeting the operational objectives of an EBN, e.g. OBJ-1 to OBJ-4.

Results

All in all, the RECREATE consortium has produced 20 different EBNs. In order to ensure easy reading and comparability between the cases, a final EBN template has been developed and used for the discussion of each of the narratives. According to this blueprint, the structure of each of the cases comprises the following parts:

• 1. The first section ‘the narrative’ is a concise overview of the EBN and summarizes the most relevant findings. It assists the reader in capturing logic and content of the following parts. Together with (2), it aims to reflect the first operational objective (OBJ-1).

• 2. The section ‘understanding the innovation system’ establishes an understanding of the current state of the innovation. It explains the actual object of innovation and describes the actual market penetration.

• 3. Section ‘estimation of the investment case’ assesses effects that can be expected when the innovation is upscaled to the European level. Indicators considered in the assessment comprise future market sizes, jobs created and environmental and social benefits that could be expected. The section concludes with an outlook on possible investments needed in order to push diffusion. This section largely corresponds to (OBJ-2).

• 4. Section ‘innovation system functioning’ is based on an analysis of seven different functions of the respective innovation system (OBJ-3). In doing so, the transition management analysis tool of the technology innovation system (TIS) framework is used. The extent to which innovations meet the different functions is represented in a spider diagram.

• 5. If not discussed under TIS, the optional section ‘further evidence on the innovation system’ gives room for the presentation of additional evidence.

• 6. Based on the function analysis, the section ‘policy recommendations’ proposes possible actions, DG RTD could implement in order to push diffusion. This selection is understood as a first proposal for policy actions as impetus for further in-depth policy analyses (OBJ-4).

In the retrospective, it is possible to assign the EBN to three different phases in the EBN development that evolved during the years as unplanned result of the co-design and co-development with the European Commission (Table 2). Of the 20 EBNs, 12 are systemic innovations fostering a circular economy (SEI), 5 contribute to the creation and establishment of climate information services (CIS) and 3 establish nature-based solutions in cities and regions of Europe (NBS). Eight EBNs are about innovations that are based on the deployment of a certain technology (T), while the other 12 EBNs are related to social innovations with a focus on the organisational challenge of re-combining existing but loose and disconnected elements (NT). Two out of 20 EBNs are not more than concepts of innovations, while 18 EBNs were based at least on pilots or a first stage of market diffusion.

After 4 years of co-development with the DG RTD, the RECREATE consortium presented in total 20 EBNs. Based on the extensive testing, we propose the final definition:

Evidence-based narratives are a systematic approach applying the Technological Innovation System (TIS) Analysis of how to develop stories on emerging innovations each of which are following the same structure and proven heuristics. EBNs are underpinned by triangulated findings and data, and are as such able to serve as a communication tool for research and innovation policy makers assisting them in giving the best possible policy decisions in the context of uncertainty and limited resources.

The team agreed on a set of four practical guidelines for developing a high-quality EBN:

• 1. Select a clear and concise area that is micro and concrete enough to reach tangible results but not too small to prevent conclusions for up-scaling.

• 2. Provide evidence especially from direct interaction and interviews with key stakeholders, since the Commission often lacks bottom-up micro-evidence.

• 3. Ensure both economic and environmental evidence, increasing salience for inter-service consultations.

• 4. Be aware of not duplicating what is known and covered in other R&I programs or related research.

Example of an EBN

To exemplify, EBN no. 13 describes innovations within the field of urban climate information services using the case of Copenhagen as its background (Table 2). This EBN was one of the original 14 EBNs and prompted by the Commission’s adoption of a ‘climate service agenda’. Following a review of the first version by relevant stakeholders and experts from the Commission, a request was issued for further triangulation including additional evidence specifically with respect to the investment case and interactions with relevant stakeholders. Based thereof, an advanced EBN was developed, its results ultimately disseminated at several conferences and through a policy brief. The development of EBN no. 13 was parallel to the preparation of the ‘European research and innovation roadmap for climate services’, published by the Directorate-General of Research and Innovation in 2015, and hence the evidence was shared with the Commission’s Expert Group in this regard. This narrative focused on what is now part of several Missions in Horizon Europe: the use of climate information services to facilitate climate resilient urban planning—taking Copenhagen as the example and with a secondary focus on nature-based solutions and on how to enable robust cost-benefit analyses for optimizing environmental, social, cultural and economic benefits of planned investments. At its core, a software tool (technical innovation) was developed to facilitate the screening of flood risk by integrating different sources of information in a spatially explicit form, e.g. present and future climate data, urban topography, dynamical flood modelling and socio-economic factors. This innovation was instrumental in bringing together different administrative areas and competences in a different way (fostering additional social innovation), and in coordinating the necessary funds for implementing the overall Copenhagen Climate Adaptation, which has been hailed as one of the most ambitious in the world. The following summarizes the status of the seven innovation system functions formulated with the subjective score awarded by the EBN author in brackets:

Entrepreneurial activities (3): A plethora of research and development activities already exist academically and commercially, but large-scale demonstration is largely absent.

Knowledge development (2): Despite obvious needs, the current knowledge on climate change impacts—and how to adapt in an efficient and sustainable way—at the urban scale is lacking. Further climate and socio-economic research is needed, and there are critical challenges of data availability.

Knowledge diffusion (5): Dissemination through local, European and global networks is very efficient, the city of Copenhagen itself is a very active player and already collaborating with several cities in the world, including New York.

Guidance of the search (4): The issue of climate adaptation is high on the political agenda; there are many supporting and funded activities under various Horizon 2020 (ongoing and planned) and at the Member State level.

Market formation (2): Effective, standalone business models are weak or altogether missing, large dependence on public funding and policy interventions for adoption and scaling.

Resources mobilization (2): Public sources dominate (academic research), so far innovation and product development in this field is not very attractive for private investors but this picture might be changing.

Creation of legitimacy (1): Despite positive results from several cities, there is considerable concern amongst policymakers about the validity of climate projections and the value of analyses based thereof at the urban scale, including cost-benefit analysis. A formal mechanism for creation of legitimacy and quality assurance is (still) critically needed.

Some of the policy recommendations derived from the above were (1) to develop new and existing financing models; (2) to sustain long-term investments in research and development; (3) to set up city-scale demonstrators and generate real market cases, narratives and best practices; (4) to stimulate co-production and co-development with stakeholders and companies, effectively bridging the gap to academy; (5) and to define standards for, e.g. climate service provision.

Policy recommendations

In the following, we summarize some of the main findings from the 11 most advanced EBNs produced in phases 2 and 3 of the EBN development, including the abovementioned no. 13/Urban climate information services. Here, stakeholder consultations play a critical role both in terms of providing and confirming, the evidence collected. It should be further mentioned that in the case of seven of those 11 EBNs, the RECREATE consortium has issued a formal policy brief, presenting its central findings, including the associated policy recommendations.

We compiled the policy recommendations from the different EBNs and linked them to the relevant TIS functions based on recommendations found within the individual narratives and/or a post-assessment of the content carried out by the authors. The result is shown in Table 3.

Table 3 Number of policy recommendations given per TIS function across 11 comparable EBNs

All 86 policy recommendations mentioned in all 11 EBNs focus on the role of governments and/or the EU in supporting the innovation system and/or lift barriers related to one of the seven innovation functions. The overview (Table 3) indicates that the need for increased entrepreneurial activities (EA) is only suggested twice, which is perhaps not so surprising, since the concept of a narrative underpinned by evidence (data) implicitly suggests that entrepreneurial activities have already taken place in some form. Similarly, resources mobilization (RM) is linked with the highest number of policy recommendations (23; ‘Sustained long-term investments in both applied and basic research within a number of different areas’). RM is key for the development of any innovation system, in particular in their early stages [41]. Guidance of the search (GoS) (‘Robust regulatory frameworks to increase trust and formation of transparent markets’) and creation of legitimacy (CoL), which are linked with the second-highest number of policy recommendations (15) across different innovation systems, are crucial for innovation system development as well [41]. These systems are either helping to form favourable market conditions or creating widespread support for an innovation to take place (and consequently guaranteeing continuing resources mobilization). As such, the policy recommendations derived from the 11 EBNs seem in general to provide appropriate advice concerning the further development of the innovations.

The TIS function assessment is summarized in scores ranging from ‘strongly underdeveloped’ (0) to ‘strongly developed’ (5). All scores are compiled in a spider diagram. In the context of the present EBNs however, these scores were not allocated on the basis of a strict quantitative assessment of chosen indicators, but mere judgement by the authors on the basis of the evidence collected from literature and the interviews undertaken. As a result, it is in this case not immediately possible to compare scores across EBNs. That said, visual inspection (not shown) reveals that the four functions leading to the most policy recommendations across all the 11 EBNs are also the ones most often judged to be (very) weakly developed: GoS, MF, RM and CoL. In each instance, at least 5 (half or more) of the EBNs indicated that these functions were poorly developed.

Lastly, to assert whether information about the real state of an innovation system could be consistently extracted from an EBN, Fig. 1 compares three of the case studies, representing innovation in the expected early (no. 13/Urban climate information services), intermediate (no. 7/Selling solar services) and late (no. 16/ICT-based waste and resource management) stages of their development.

Fig. 1

Comparing three EBN in different stages of development

Disregarding a direct comparison of the scores, the early stage innovation is clearly characterized by high variation between the levels of development (from 1 to 5), followed by the intermediate level innovation, whereas the innovation system functions related to the late stage innovation with the sole exception of RM scores between 3 and 4 consistently. Further, in the case of the early stage innovation, the least developed functions seem to be close to market, which seems appropriate.

Discussion

The potential use of the EBNs lies in the hands of the European Commission. The responses to the various EBNs therefore constitutes direct intelligence regarding the expectations of the Commission and the intended purposes. Analysing their reactions and their comments therefore lead to recommendations for next generation of EBNs.

First reactions from DG RTD indicate that it is welcomed by policy-makers as an advancement in the science-policy dialogue. In September 2016, the approach had been introduced at the International Sustainability Transitions conference (IST) [43] to receive complementary comments from science for further improving the methodology. In parallel to the presentation at the IST, results of this research had also been introduced on September 7, 2016 in Bratislava at the conference Transition to the Green Economy (T2gE) of the Slovak Presidency of the Council of the European Union [44]. Furthermore, the RECREATE consortium received an invitation to prepare the EBN cases for a presentation at the ‘Global SCP Clearing House’ as part of the 10YFP programme of the United Nations, because EBNs were considered to be ‘useful’ and ‘welcome’.

Interestingly, much of the EBN work was aimed at using the System Innovation logic of assessing the 7 TIS-functions. This was chosen to come to a uniform set up of the work and the reports, but also to guide the authors towards well-founded policy recommendations. Remarkably, though, the Commission did not comment on the TIS approach nor on the usefulness and strengths of the policy recommendations. This might hint at the fact that the well-structured EBNs providing bottom-up evidence, already served the purpose of the Commission. However, even though their potential added value had been appreciated, it had been recognized that the exploitation of the potential would be restricted by limited attention capacities of the staff who is absorbed by the daily routine of administering projects and dossiers.

In late 2017, at the end of the EBN task, the authors developed an evaluation questionnaire that was sent out to the RECREATE members from science and industry in order to collect feedback on the usefulness and quality perception of EBNs as a tool for policy-making. All 18 respondents agreed that EBNs could be a helpful tool for EU research programming, e.g. by reducing complexity and structuring.

The RECREATE members from science as well as from industry appreciated the usefulness of EBNs in many respects. The joint evaluation emphasised that the EBN approach is not a methodology by itself, but an effective framework of using TIS analysis for deriving policy recommendations by contextualizing the analysis of seven TIS functions into a discussion of potential innovation impacts. There had been consensus in the RECREATE network that the policy recommendations could not have been developed without a guided process of co-development—and doing this in the form of an EBN exercise has strong advantages.

Conclusion

EBNs deal on the basis of relatively few data with future guidance in instable globally connected systems subject to a potentially infinite number of influencing factors. As they contain high degrees of uncertainty and assumptions, the construction of EBNs is based on an interdisciplinary theoretical approach: the RECREATE researchers followed conclusions of cognitive science for decision-making under the condition of uncertainty aiming at simple ‘rules of thumb’. Furthermore, the authors unfolded the potential of narratives for creating meaning and order by dealing with the complexity of the cases.

Independently and after the RECREATE project, DG RTD commissioned research on Radical Innovation Breakthroughs (RIB) for preparing Horizon Europe’s implementation [45]. In contrast to RECREATE’s approach of selecting relevant innovation in an iterative process with DG RTD, the RIB research team implemented an automated survey of recent scientific and technical literature, which had been filtered through panels of experts and combined with analysis of other foresight projects. Instead of analysing the functions of the TIS framework, the identified RIB had been screened for their potential impact on future global value creation, and assessed in terms of maturity, long-term diffusion potential and relative strength of the EU in research and innovation.

By making use of EBNs, the discussion of well-structured and easy-to-read micro and local innovation cases can be systematically embedded into the discussion of the orientation of an overlying political macro situation. This is helpful for arriving at ‘rules of thumb’ policy recommendations. The process of triangulation combines the discussion of TIS functions within the overall EBN framework with stakeholder interviews. This approach enables the development of general policy recommendations by basing it on actual micro and local evidence. This evidence-based framing creates an actual added value for policy making, especially in terms of awareness raising, knowledge sharing and building plausible chains of arguments. In this respect, EBNs have proven to be helpful in all contexts whenever quantitative assessments are not possible, as for example in the case of weak signal identification (no data and knowledge available) or complex model application in specific problem contexts (very particular data and knowledge available). Ultimately, basing policy and research on actual experiences on the ground is helpful in providing sound roadmaps and policies.

Throughout the evolution of EBNs, the relation to the European Commission’s standard impact assessment (IA) had been considered. At this point of EBN development, EBNs and IAs fulfil different roles in policy and innovation cycles. While IAs assess the impacts of certain policy options, EBNs assess policy pathways to innovations that may lead to certain impacts. If then, the purpose of EBNs is to support the development of sound research policies, the collection of EBNs should represent the programming area activities in an adequate way; therefore, the selection of potential innovations and their socio-economic framing (e.g. jobs and growth) becomes a critical issue.

EBNs belong to forward-looking activities at the interface of science and policy-making. They serve as heuristics that can provide orientation in a situation characterized by complexity and uncertainty: ‘Narratives reduce complexity, create a basis for current and future-oriented action plans, are a foundation for the co-operation between actors, and support reliability of expectations’ ([46], p. 84).

In contrast, quantitative storytelling (QST) [47, 48] aims at using quantitative evidence ‘via negativa’ ([49], p.781), by testing existing narratives and framing in policy-making. QST aims a falsification of assumptions on feasibility, viability and desirability of expected results of existing narratives and policy frames with the aim of identifying knowledge gaps. While QST aims at deconstructing existing narratives, the EBN framework is about the co-development and consolidation of new ones. With this constructive connotation, they are a strategy tool to support decision making within strategic management [50] and boundary object [39] meant to improve processes of translation and knowledge integration between scientists and policy-makers. Especially the link to research and innovation policy has a strong transformative connotation. The approach could be relevant not only for transformative and transition science but also for governmental and non-governmental decision-makers by providing a knowledge-based framework for potentially transformative policy-advice.

The proposed EBN method applies modern approaches of logical reasoning. Learning from examples and standardised input from various stakeholders can underpin policy advice in a suitable way and therefore may shape and support policy-making.

Science can provide methodologies, which allow to select and structure information. If this is being undertaken in a reliable and transparent way, it is beneficial for societal development and policy-making. The in-depth analysis of 20 case studies has shown how a systematic method can create a salient evidence base for (research) policy interventions. Our optimistic conclusion is that even under the condition of complexity, high uncertainty and fragmented evidence, i.e. quantitative and qualitative data from sources of distinct reliability, advice to decision making is possible, without compromising scientific standards. The EBN as an approach to evidence-based policy advice will need further development and testing, but the evidence presented in this paper seem to justify further exploration of our approach.