Skip to main content

But do they deliver? Participatory agenda setting on the test bed


In this contribution, we investigate how results produced in a large-scale participatory agenda setting process differ from results of expert-based foresight studies with a similar aim of informing EU research and innovation (R&I) policy. After providing a theoretical positioning and an overview of the EU-wide participatory agenda setting process CIMULACT—Citizen and Multi-Actor Consultation on Horizon 2020, we describe our developed analytical approach that includes five analytical steps and calculation of three metrics. By comparative reading, analysis and scoring of 16 expert-based foresight reports, we produced data for the metrics that allow for discerning between (a) how many of the analyzed foresight reports cover a respective topic from CIMULACT, (b) how well this basic coverage aligns qualitatively, and (c) comprehensive comparison of each CIMULACT topic with respect to all surveyed reports. To discern differences, we chose those results (research topics) from CIMULACT that were also sufficiently covered by expert-based reports. Our findings suggest that such citizen-based, multi-actor co-created policy advice qualitatively differs considerably from that elicited by expert-based reports, in terms of direction and focus of the proposed R&I agenda.


Images of the future shape the expectations of what tomorrow’s tomorrows will bring. They do so by influencing mind-sets, decisions, actions, and the allocation of funds [1, 2]. Progress in science and technology is a major driver for shaping the possibility space of futures, and simultaneously effecting the expectations of various actors. Those future-oriented actions and decisions impact the lives of many people, yet traditionally very few actor groups set the agenda, or have the opportunity to implement their expectations, for instance via research and innovation policy programming.

In order to harness expectations and provide policy advice for actively shaping futures, forward-looking activities (FLA) often elicit experts’ opinions. Additionally, stakeholder engagement has become a norm over the course of the past two decades [3]. This shift towards democratizing knowledge production has had a long development, described for instance as the participatory turn in science [4] or post-normal science [5]. Here, the co-production of socially robust knowledge by linking different types of knowledge is a main aim of transdisciplinary research (e.g., [6,7,8]. And also in futures studies, the shift towards participation which began in the 1970s [9, 10] gained momentum in the 1990s [11] and led to the development of a number of participatory frameworks for futures research [12,13,14].

Societies increasingly face complex, transnational, highly interrelated problems involving high degrees of uncertainty while being potentially open-ended issues—wicked problems in planning theory terminology [15, 16]—such as climate change, resource scarcity, or economic crises. Despite the age of the discourse and referenced problems, policy making is struggling to find new ways of addressing these complex challenges [17, 18].

However, transdisciplinary knowledge co-production is recognized as being effective for addressing societal challenges with regard to sustainability transitions [19, 20]. Public engagement literature states producing better decisions and raising legitimacy are among the most prominent reasons for commissioning and conducting participatory processes [21,22,23,24]. Additionally, on EU-level, public participation is pursued to strengthen EU-citizenship and build trust in EU institutions [25, 26], and can be traced back to projects like 2006’s “Meeting of the Minds,” a transnational citizens’ deliberation on Brain Sciences [27].

The European Commission reframed such wicked problems as societal challenges and built a large part of its framework program for funding research and innovation (Horizon 2020) on the expectation that science and technology will support solutions for those challenges [28]. Here, the governance concept of Responsible Research and Innovation (RRI) was implemented to strengthen this role in research and development actors with the guiding principles of inclusiveness, anticipation, reflexivity and responsiveness [29,30,31]. All principles embrace public engagement at their center—responsiveness, for example, aims at orienting research and innovation (R&I) processes towards societal needs and demands. However, Zwart et al. state that the newness of RRI mainly resides in an emphasis on socio-economic impacts, such as valorisation, employment, and competitiveness [32].

Participatory agenda setting for increasing responsiveness to societal needs and values

On the one hand, participatory methods for public engagement in research and innovation are abundant [33]. A recent analysis shows that with regard to research program development—agenda setting—the involvement of civil society organizations or other stakeholders is common practice, while citizens are only involved seldom [34]. On the other hand, expert- and stakeholder-centered forward-looking activities (FLA) have proven to be effective in supporting RI programming [35,36,37,38]. However, lay-people’s involvement in FLA with regard to advice-giving mechanisms for science and innovation programming is relatively new and rare [39,40,41,42].

Yet, program development offers a timely and unique space for the entry of public needs, values, and demands into the innovation process. Here, participatory foresight and co-creation can offer tools to support the shaping of responsive research and innovation agendas. Lay-people engagement by means of visioning evades some of the challenges attached to early engagement, such as opaqueness of the issue to be discussed. Visioning allows for moving away from immediate concerns or feasibility thinking and towards shared explorations of alternative, more sustainable futures, especially when applying longer time horizons [43, 44]. In this way, tacit knowledge, needs, and values can be collected and integrated with stakeholder and expert knowledge, and used in the co-creation of socially robust knowledge for responsible agenda setting. The next section will give an overview of such a process, applied during CIMULACT—Citizen and Multi-Actor Consultation on Horizon 2020. A more detailed positioning and theoretical background on participatory science, technology and innovation agenda setting, especially with regard to the project discussed in this contribution is described in [45].

Aim of this paper

Participatory agenda setting entails planning and conducting complex participatory processes, which include challenges in data analysis and knowledge management, in motivating and selecting participants or in ensuring the aimed for impact on political decision making. Several theoretical and practical challenges for public engagement in science and technology have been put forward in literature, for instance the lack of measurable increases in rationality considering the respective topic in deliberation, the failure to open up the debate, or the lack of impact on decision making [46,47,48,49,50,51,52,53,54].

Against this background, we compare results of a large-scale Europe-wide participatory process for research and innovation agenda setting (CIMULACT) with the findings of 16 expert-based foresight studies with a similar aim. Authors have had central roles of work package and task leaders in designing and implementing the discussed project. We are aware that this may hamper objective judgement of the process; however, we believe that this close proximity to the CIMULACT project, and our familiarity with both its results and the comparative method it employed, allows for in-depth extension of research work initiated therein. In doing so, we aim to shed light on the following research question: Considering the elaborate process, invested resources and intertwined challenges, is lay engagement in STI agenda setting worth the trouble? And, to be more specific, are results sufficiently distinct from expert-based studies to make the case for the continued or expanded use of citizen-based, large-scale participatory foresight processes?

CIMULACT—citizen and multi-actor consultation on Horizon 2020

CIMULACT was a coordination and support action (CSA) funded under Horizon 2020, H2020-ISSI-2014-2015: Call for integrating Society in Science and Innovation, Topic ISSI-2-2014: Citizens and multi-actor engagement for scenario building implemented between 2015 and 2018, and coordinated by the Danish Board of Technology. The project’s main aim was to engage citizens, stakeholders, and experts in co-creating research topics based on validated and shared visions, needs, and demands. Results of the project informed decision makers, while drafting the European research agenda Horizon 2020, national research agendas as well as the upcoming European framework program for research and innovation (FP9), therefore making these research and innovation more responsive to societal needs, one of RRI’s main objectives.

The project comprised an extensive participatory process consisting of several interconnected steps, applying several well-established participatory methods as well as developing new methods. The project’s overall methodology was based on previous large-scale experiences with participatory STI agenda setting, such as delivered by EU projects CIVISTI, or VOICES was an EU-wide consultation process, in scale comparable to the first phase of CIMULACT, engaging citizens in focus groups to produce advice for Horizon 2020 on the specific topic of urban waste as resource. CIVISTI is a long-term participatory foresight method developed and applied to find to identify new, emerging topics for Horizon 2020 [39, 40]. Within the method, citizens developed visions of a desirable future, while experts and stakeholders used these to derive recommendations for research and development policy. Finally, citizens prioritized these recommendations. The method was later adapted and applied to local and national scale in several case-studies [41].

The demand-oriented CIVISTI-logic, starting with collecting citizens’ visions to then derive advice for R&I policy, is still present in CIMULACT. Nevertheless, the methodology applied in CIMULACT is considerably different, for instance with regard to mixing actor groups (co-creation) or the final product—research topics instead of recommendations. It comprises six main steps: (1) Applying a standardized visioning method in thirty European countries, more than thousand citizens produced 179 visions of a desirable future. CIVISTI—Citizens Visions on Science, Technology and Innovation (2008–2011), (2) From these visions, experts extracted 26 underlying, crosscutting, implicitly, and explicitly mentioned societal needs. (3) Selected citizens from the visioning workshops, stakeholders and experts—overall more than a hundred participants—co-created 48 suggestions for research topics on the basis of the visions and societal needs in a facilitated workshop. VOICES—Views, Opinions & Ideas of Citizens in EU on Science (2013–2014), (4) In a second national face-to-face consultation phase, the consortium engaged multiple actor groups in all 30 countries to review and enrich the results and shape them towards becoming tangible research topics. Deliverable 3.2 Programmes and concepts for all citizen and multi-actor consultations, In parallel, an online consultation engaged more than 3400 participants in an argumentative Delphi to prioritize and enrich the drafted topics. (5) Integration of results of the online and face-to-face consultations to synthesize 48 one-paged summaries of the research topics with the sections challenge, scope, and expected impact. (6) With the aim of increasing applicability to EU research programming, a workshop was held to engage EC policy officers and experts, who merged highly similar topics, resulting in 46, and selected and refined 23 final topics, Deliverable 2.2 Final citizen’s based research topics,

The selection processes regarding citizen participants had two distinct methodological breaks within CIMULACT. To select citizen participants for the visioning workshops, broad principles were defined by the CIMULACT consortium to find common ground concerning the “ideal” participant pool. “In a highly deliberative collective visioning process, more than 1088 citizens in 30 European countries were engaged and developed 179 visions of desirable futures. Here, the method is not aiming at inviting a statistically representative sample of participants, but building a sample of maximum heterogeneity to include a high number of different perspectives. Thus, each national consultation invited around 36 participants, ensuring a high diversity within the following criteria: age, gender, education, occupation, and place of living. For example, six age groups were defined, ranging from 16 to 60 and older, to then have one member of each age group in all smaller working groups (table level). Four educational levels—pre-primary to higher education—and several occupation groups were defined to distribute participants at table level aiming at a high heterogeneity of backgrounds. The criterion of place of living ensured engaging city as well as country dwellers.” [45]. To accommodate alternative national cultures and institutional processes, each of the 30 national partners was allowed to implement their own selection method within a framework of standardized methods and approaches. In Germany and Austria, for example, participants were paid a small daily stipend; in order to incentivize participants who might not normally engage such an activity, in Denmark, paying such an incentive would often be considered an affront.

Citizens participating in follow-up workshops were often selected from these visioning participant pool based on availability and the participants level of engagement within prior workshops. It is also worth noting that examination of the national visions allowed the CIMULACT project to identify trends, issues, and prioritizations that were particular to specific nations, and clusters of nations whose research agendas might align with little regard for geographic location. However, all steps of the CIMULACT process subsequent to the visioning workshop took place on European level, or if conducted on a national level, revised and elaborated only parts of the intermediary which were than aggregated as a European result. Therefore, CIMULACT’s final results can hardly be investigated for national properties. Furthermore, CIMULACT results are products of group processes and therefore cannot be investigated regarding different gender or social lines of inquiry.

Method—comparing citizen- and expert-based topics for R&I agenda setting

Foresight studies employ a large diversity of methods, and neither reporting structure nor classification is standardized, which makes comparison challenging. While comparing national foresight reports, Gavigan and Scapolo [55] answered to these challenges with applying a national objectives analytical model, while Alsan and Atilla Oner [56] developed comparative matrices. Other authors have compared methodologies [57], or participants, organizers, and the difficulty to measure policy impacts [58]. While none of those comparative approaches showed the necessary specifications to be applied in comparing CIMULACT results to those of expert-based foresight studies, we applied some of their components. However, to provide a meaningful comparison, we developed a content-driven comparative analytical framework (Fig. 1), which includes five different analytical steps and calculation of three metrics.

Fig. 1
figure 1

Overview of workflow and method of comparison representing our method of comparison as well as the followed workflow. Five steps of analysis produce three different metrics. The analytical steps include preparation, comparative reading, qualitative assessment, aggregation, and review and discussion of differences, whereas metrics include degree of coverage, degree of alignment, and cross comparison. This represents our method of comparison as well as the followed workflow. Five steps of analysis produce three different metrics. The analytical steps include preparation, comparative reading, qualitative assessment, aggregation, and review and discussion of differences, whereas metrics include degree of coverage, degree of alignment and cross comparison


For selecting expert-based foresight reports for comparison, two primary selection criteria dominated the process, purpose, and scope of the respective report, with secondary criteria creating an additional filter. With regard to purpose, we selected studies that had a similar aim as CIMULACT: informing the development of research and innovation agendas and here mainly Horizon 2020, sourcing from databases, such as [59], The Publications Office of the European Union and the European Commission Joint Research Centre Publication Repository. In order to enable a proper comparison with regard to the to the highly integrated fields of research CIMULACT addressed, we selected foresight reports with most similar scope, i.e., broad ranging in terms of included research fields, geographical area, and socio-political analysis. We therefore excluded, for instance, foresight reports focusing on a specific sector or field of inquiry, or national in scope, even though they contained very detailed foresight knowledge. Categories were identified in other foresight mapping analysis [60], such as commissioning organizations, authoring organizations, publication date, time horizon, methodological approach, and the intended audience served as secondary selection criteria. The 16 reports finally selected can be found in Additional file 1: Table S1. As basis for comparison, we chose the 46 CIMULACT topics resulting from the last step of the participatory process, which include the final 23 topics, but also the larger set of topics that were not selected by EC officials during the project’s final conference. In order to prepare the CIMULACT topics for comparative reading, we identified and highlighted key words and critical concepts within the topics as comparative lenses. For more detailed information with regard to selection of reports and identification of key concepts, consult [61].

Comparative reading

Two independent readers used comparative reading templates, containing the highlighted text excerpts and thematic groupings prepared in the previous step, to match direct quotes from the 16 selected foresight report to the CIMUALCT’s corresponding critical concepts and keywords. Individual results were then aggregated into a single file for further processing and deriving synthesized findings.

Degree of coverage (DoC)

This score is derived from the aggregated comparative reading results as a basic unit of comparison, indicating whether an expert report did or did not mention an area of inquiry indicated within one of the CIMULACT research topics. Final DoC scores were calculated for each of the CIMULACT research topics by aggregating the results of the comparative readings across the 16 expert-based reports. For example, a DoC score of one (1/16) indicates that matching quotes from one expert-based foresight report were found for the respective CIMULACT topic.

Qualitative assessment

While the DoC score offers a good overview of how many reports cover the respective CIMULACT topic, it does not capture qualitative differences of how CIMULACT or the expert-based report cover the issue in question, i.e., direction of research, intention, or goals. To capture this, two independent assessors assigned scores measuring the degree of congruence or alignment between all direct quotes from the expert-based foresight reports and CIMULACT topics.

Degree of alignment (DoA)

Qualitative differences in alignment were assigned according to the following scoring scheme (0–3): zero alignment (0) no matching quotes found for CIMULACT topic; low (1) expert-based report discusses one or more aspects of the respective CIMULACT topic, but differs on goals, expected impact or perspective of analysis; medium (2) quotes generally match with CIMULACT topic, but miss key components; high (3) very high alignment between quotes and topic’s content.

Aggregation and review

Next, independent assessments were aggregated to calculate the alignment score for each topic. Average scores were calculated for cases where assessors’ scoring matched or differed by no more than one score. Cases of disagreement, with a difference in score by more than one, were revisited and discussed in depth with the aid of original texts until an agreement of no more than one score could be reached and a corresponding average score could be calculated.

Cross comparison (CrossComp)

By multiplying the first two metrics, DoC and DoA, this combinatory metric highlights the fall off points of similarity between expert- and citizen-based reports. This enables an efficient mode of quickly assessing each CIMULACT research topic’s comprehensive comparative score with regard to all of the surveyed reports. The maximum CrossComp score was calculated to be 16 (representing universal coverage and alignment), and final CrossComp percentages were calculated accordingly.

Discuss differences

Using CrossComp score, we sought to identify those research topics that were shared across citizen-based and expert-based projects, but whose comparative results shed light on the differentiating characteristics of citizen-based process results. Therefore, we have partitioned the CIMULACT research topics into six levels of relationality with regard to expert-based reports. Those CIMULACT research topics that have less than 15% CrossComp have been excluded from this analysis and are the topic of a forthcoming publication. The remaining five tiers of CIMULACT research topics show CrossComp scores and therefore comprehensive alignment of 15% < X < 20%, 20% < X < 25%, 25% < X < 35%, 35% < X < 45%, and X > 45% with the selected expert-based reports.

Results and discussion

In beginning to answer the question concerning the added benefit of large-scale participatory processes for R&I agenda setting, the comparative study results can be clustered in three primary groups of analysis. First and second of these groups are research topics that are unique to expert-based foresight studies and CIMULACT, respectively. These are topics that share little resonance across the two sources of foresight activities, and a detailed analysis of these results is the focus of a forthcoming publication. The third category of results then are those research topics that demonstrate a shared interest by both experts and citizens, and thus provide a fertile ground for determining the distinct benefits the participatory; citizen-based processes can provide policy crafting endeavors. Table 1 shows an overview of the finding regarding this category. Here, the question then becomes, not do they deliver, but what do they deliver that is sufficiently distinct to make the case for the continued or expanded use of citizen-based, large-scale participatory foresight processes? In coming to address this question, we sought to identify those research topics that were shared across citizen-based and expert-based projects, but whose comparative results shed light on the differentiating characteristics of citizen-based process results.

Table 1 Overview of results

General differences

Overall, the displayed data suggests a general misalignment between CIMULACT topics and the compared expert-based reports. Considering that all CIMULACT topics showed CrossComp scores of lower than 50% indicates severe differences in coverage and/or qualitative alignment, speaking to notable differences between citizen-based and expert-based results. Given that Qualitative Alignment scores were much lower than Degree of Coverage scores—we believe that these differences can be classified as “uniqueness” or “novelty” of citizen-sourced results.

Further supporting this argument, consider the following calculations. If a topic had only 50% DoC and 50% QA, its CrossComp score would be 25%. This means that if only half of the expert reports mentioned a research topic and discussed it in such a way that semi-aligned the expert and CIMULACT recommendations, a CrossComp score of 25% would be achieved. Though this is a seemingly low threshold, expert reporting achieved this for only 12 of the CIMULACT research topics. This means that nearly three-quarters of all CIMULACT research topics offered significantly differentiated content in a qualitatively distinct fashion.

Discussing exemplary topics as demonstrative of comprehensive alignment and differences

While each research topic could be the focus of a more in-depth analysis, we have selected one research topic from each of the five tiers to examine as demonstrative of the comprehensive alignment metric and a general understanding of the differences between citizen- and expert-based foresight towards policy craft (see Table 1, in bold). All selections are research topics from the final 23 topics chosen and elaborated in the final policy conference of CIMULACT (see step six, section “CIMULACT—Citizen and Multi-actor Consultation on Horizon 2020”), except the example for level five—data for all—as neither of the topics from this tier were included in the final 23 research topics.

Level 1: technology as a means of well-being

Given the technocratic emphasis of expert reports as outlined in our original research [61], the low overall alignment score for this topic was initially surprising, especially with so much emphasis placed on technological panacea for economic development, healthcare, energy, security, etc. It is hard to imagine how citizens could seemingly contribute something focused on technology writ large and yet so drastically out of line with expert foresight reports. Could this input be valid, and, if so, is it then a unique contribution?

In examining the citizen definition of well-being, however, hints towards an explanation for this seeming anomaly begin to emerge. In the CIMULACT research topic definition, we find terms such as: “we govern them [technologies],” “end-user centric,” “negative consequences [of technology],” “screen addiction,” “shifting relationships,” “thinner boundaries between virtual and real actions,” and “exploitation at the workplace.” Through this citizen-generated rubric for understanding “well-being” as a product of technological development, the qualitative differences between experts and citizens become clearer. While experts focus on the regional or global impacts of a technological field, citizens are concerned with the impacts of such technologies with respect to their personal agency, their relationships with devices and data, and their social contracts regarding work and community.

Level 2: good food research

The necessity of food to human (and other) life begins to explain why this research area was more widely shared between experts and citizens, but this same necessity also brings even improved CrossComp scores into question. Since the future prospects of food would be an essential component of any scenario, how can this CIMULACT research topic vary so widely from expert notions of future food research policy? Indeed, this question itself betrays part of its solution.

The explicitly detailed goals that citizens proposed for future food research demarcate CIMULACT research topics from expert foresight reports. While it must be acknowledged that over two-thirds of the expert report mentioned food as a research area, this only compounds the qualitative differences between these reports and citizen recommendations. According to our analysis, citizen recommendations were far more focused on the specific goals that research should be pointed towards—a combination of ends, as opposed to a singular field.

For example, the ambiguous term “biotechnology” was deployed widely across expert reports with respect to food research and was sometimes linked to more specific pursuits—genetic engineering, “bioeconomy,” or synthetic proteins. In many cases, research in these technological fields is linked to increasing food yield and land-use efficiency, often linked to changing global demographics. In a similarly ambiguous fashion, CIMULACT calls for research into “biotechnology,” but aims such research at “impact of new research driven paradigms on health, economy, environment, and sustainability in a comprehensive and systemic way.” While there is a clear shared interest in researching biotechnologies, there are differences concerning to what ends such research should be pursued, and what concerns shall drive that research.

CIMULACT further distinguishes its set of recommendations for Good Food Research by calling for the following: “Applied research can provide knowledge and information,” “dissemination of outcomes,” “respective regulations and policies,” “Implementation of educational programmes,” and “use new food in schools.” Again, the specificity of the modes of research to be conducted, the generation and dissemination of information, and the spaces in which research can interface with society remain unique to CIMULACT. These are more than just qualitative differences, but also reflect methodological approaches, participant selection, and multi-faceted awareness raising campaigns.

Level 3: evidence-based personalized healthcare

This topic area was mentioned in some fashion or another by 75% of the analyzed expert reports, clearly indicating that healthcare, and personalized healthcare specifically, is a shared priority area for future R&I policy. However, despite the broad expert coverage of elements mentioned in this CIMULACT research field, there remained overarching low alignment—again pointing to unique additions that wide-spread citizen-sourced foresight can provide.

In the instance of this CIMULACT research topic, content similarities clustered around the broad mention of “personalized health care monitors,” in particular “wearable” like smart watches and health trackers, and “personalized, holistic, data-based health services.” Both the means for increased personal data collection, and the ends to which such data could be deployed, were hot topics for future-oriented studies. However, these shared endorsements by both citizens and experts did not indicate that research should be conducted in the same way or be inclusive of the same participants.

Several key components of the citizen-based recommendations were consistently omitted from expert-based reports: bilateral technology skills training for both doctors and citizens, with social psychology for doctors, health and lifestyle for patients, and digital literacy skills for both. These were core demands within the citizen recommendations—technological and scientific development must go hand in hand with skills training for both sides of the health care equation. And yet rarely do we find emphasis on learning or co-learning of skills when these research fields are addressed by experts. It is clear that citizens see the benefit in these technologies, while simultaneously acknowledging the need for a shared knowledge and skills base to minimize the potential drawbacks such technologies can have (for instance, unsecured personal medical data).

Level 4: making dense and growing urban areas more sustainable and liveable

This topic was also covered by 75% of the analyzed expert studies and had medium level of qualitative alignment according to our comparison. With regard to its CrossComp score, it is among the top four CIMULACT research topics. Urbanization, “smart cities,” and urban design were among the terms frequently used by expert foresight work to address issues in line with this CIMULACT topic area. Many reports discussed the global trend of increasing urban populations, particularly with regard to the additional resources and infrastructure such migrations require. Some reports addressed conditions that can turn cities into hubs of innovation and creativity and ultimately drive economic growth. The difficulties of governing urban areas, particularly from a logistical standpoint, were often, were often addressed as sites for technological developments: big-data analytics, semi-autonomous digital agents (and the “internet of things”), system-wide real-time monitoring, and drone transportation, to name a few.

Many of these topics and issues are also recognized within the CIMULACT research topic, though qualitative differences emerge in the framing of research programs to address them. For instance, CIMULACT calls for “challenges of density, diversity, ecology, population development, and financial addressing [them] in combination, not on their own, using different forms of citizen consultation....” Through this prescription, CIMULACT explicitly deviates from expert reports in two important ways.

Firstly, it calls for research not within possible technological solutions, but comprehensive mapping and monitoring of the complex relationships that shape urban “systems” (logistics, transport, energy, community, culture, etc.). While expert-based reports acknowledged the inherent complexity of urban areas, few made direct calls for research into these systems, and fewer still inclusive of “social” or “soft” systems. Citizens explicitly called for research that reaches across layers of the urban “fabric,” and holistically looks at interwoven urban systems, to specifically discover what “sustainability” means for urban living and how it can be achieved. Second, it calls for research that deploys the tacit knowledge of the urban citizenry by calling for multiple mode of participatory engagement. This was a rarity in most expert reports and yet is the critical mode of engagement outlined in CIMULACT. While expert reports often refer to research being developed by public and/or private technological development labs as pointing to the way forward, CIMULACT advocates that co-generative process—between citizens, researchers, and policy makers—should be foundational to all urban research. Again, these differences are more than just subjective whimsy. Rather, they mark clear, unique desires of citizens both in what they propose as research goals, and the methodological approach they see as most beneficial.

Level 5: data for all: sharing the power of data

The growing role of “Data,” and the affordances that technological leaps in communication technology have lent to the process of data collection and analysis, have had a growing role in shaping the conversation around the relationship between data and society. This trend made the issue almost universally addressed by expert foresight reports and the CIMULACT project. Thirteen of the 16 reports in our comparison study made mention of “big data” or a related variation of the term.

This topic also had one of the strongest qualitative alignment scores between CIMULACT and expert reports, primarily due to the shared concern of data security and to a lesser degree data validation. Concerns with how personal data was collected, stored, and used were seen as core areas for future research towards ensuring that data could be put to use in a fashion that protected the citizens it was meant to serve. Additionally, both the citizens and experts shared the idea that research and development of modes of data analysis (advanced computing, algorithm design, etc.) were priorities for future R&I policy.

The unique citizen contribution to this discussion, and also the core reason for a CrossComp score below 50%, was the demand for “data of data, data access (open), and ethical data use...” These concepts were not widely shared with expert foresight reports, and serve to differentiate the citizen view of whom data should be used by, and the purpose big-data analysis should serve. While citizens called for increase data literacy skills, both in self-protection of privacy and in learning the skills needed to put data to use in their own communities, expert discussion of big data often assumed that both data and its analysis would fall under a proprietary framework of governance. Discussion of open access data was far from the norm in expert-sourced foresight, as was explicit research into securing such access and providing skills training to citizens.

A final deviation comes in the explicit call by CIMULACT for research into utilizing data and analysis as a mode of “participating in collective decisions.” Citizens it seemed were not just interested in accessing data, and learning how to conduct analysis, but they wanted this mode of input to foster agency in social and political processes. This research strand found little resonance within the expert studies, and yet serves to orient the rest of the CIMULACT research topic towards a broader, and perhaps more meaningful, goal with respect to the relationship between the citizenry and governance.


In this contribution, we provided an overview of method, results, and theoretical positioning of a large-scale participatory agenda setting process (CIMULACT). We then described our developed analytical approach, applied to compare these citizen-based results—research topics—to those of expert-based foresight studies with similar aim of informing EU R&I policy. For comparison, we chose those research topics from CIMULACT that were sufficiently covered by expert-based reports. In so doing, we provide findings to answer our research question: are CIMULACT results sufficiently differentiated from expert-based foresight to justify a continued or expanded use of citizen-based, large-scale participatory foresight processes for R&I agenda setting?

Firstly, it is important to acknowledge the limitations of this study and the methodology employed during our research. Comparative analysis of texts, particularly with respect to the hermeneutics involved within each text’s passages, is an inherently qualitative effort and demands a high level of human resources—time and cognitive processing necessary to read and deliberate productively. These expenditures placed restrictions on the number of expert reports we could include in our survey, and the overall number of textual analyses we could perform. Additionally, with respect to the mandatory qualitative approach, there can be no definitively correct assessment scores on which to answer our research question. We have done our best to create the comparative data and interpret its meaning towards a wider field, but inevitably differences of opinion will arise within the community (as they already have within out small research team). In addressing both the issue of resource restrictions and human inconsistency, we have already begun work on methods that can provide an additional baseline of analytical precision through natural language processing algorithms (research forthcoming). While we do not see this as a cure-all solution to the limits of this research, we do see a value in providing such methods to future researchers engaging in comparative foresight research.

Secondly, according to our findings, we assert that given a robust method, time, and sufficient resources, large-scale, inclusive processes for R&I policy-oriented foresight certainly provide unique results, in a form applicable to serve as input for agenda setting. As outlined above, from the thousands of pieces of citizen input, 46 distinct research topics were synthesized—each a narrative of multiple social needs coming into conjunction with opportunities for experimentation-driven innovation. Each research topic outlines both specific fields for R&I investment, while simultaneously outlining how those fields support one another in working towards more solutions for more complex social challenges. Clearly therefore, CIMULACT does deliver results in line with the project’s stated goal of informing EU R&I policy.

More importantly, however, when looking at research topics shared by both experts and citizens, our research concludes that citizen-based policy advice qualitatively differs from that elicited by expert-based reports, to a considerable degree, in terms of both direction and focus pursued. For instance, our research points to citizen-based topics’ focus on impacts of technological developments on personal well-being, relationships, work ethics or community life—all focal points that expert-based foresight studies seldom mention or emphasize. Additionally, while broad and inclusive in approach to certain challenges, citizen-based topics were often directed at very specific goals, often with distinct everyday life relevance, portraying clear societal needs.

Furthermore, if directions for research were commonly endorsed in the compared studies, citizen-based and expert-sourced advice to R&I policy differed with regard to how research should be conducted and whom should participate. Overall, CIMULACT topics frequently demanded multi-perspective and inter- and transdisciplinary approaches to stated challenges, while expert-based reports more often suggested traditional research formats in singular fields.

In light of these findings, we believe that a strong case can be made for the continued development and utilization of large-scale participatory methods deployed to inform future-oriented policy drafting processes. While acknowledging the increased costs that inclusive methods like those of CIMULACT are likely to incur, we believe that this comparative study stands as firm evidence of the methods’ value—supporting claims that citizens will create novel and pertinent results when given the opportunity.



Citizen and multi-actor consultation on horizon 2020


Citizens visions on science, technology and innovation


Cross comparison


Coordination and support action


Degree of alignment


Degree of coverage


European Commission


European Union


Forward-looking activities


Framework programme 9


Research and innovation


Responsible research and innovation


Science, technology and innovation


Views, Opinions & Ideas of Citizens in EU on Science


  1. Borup M et al (2006) The sociology of expectations in science and technology. Tech Anal Strat Manag 18(3–4):285–298

    Article  Google Scholar 

  2. van Lente H, Spitters C, Peine A (2013) Comparing technological hype cycles: towards a theory. Technol Forecast Soc Chang 80(8):1615–1628

    Article  Google Scholar 

  3. Nikolova B (2014) The rise and promise of participatory foresight. Eur J Futur Res 2:33.

  4. Jasanoff S (2003) Technologies of humility: citizen participation in governing science. Minerva 41:223–244

    Article  Google Scholar 

  5. Funtowicz SO, Ravetz JR (1993) The emergence of post-normal science. In: V.S. R (ed) Science, politics and morality. Theory and Decision Library. Springer, Dordrecht, pp 85–123

    Chapter  Google Scholar 

  6. Brandt P et al (2013) A review of transdisciplinary research in sustainability science. Ecol Econ 92:1–15

    Article  Google Scholar 

  7. Hirsch-Hadorn G, Pohl C (2007) Principles for designing transdisciplinary research. oekom verlag, München, p 120

    Google Scholar 

  8. Nowotny H, Scott P, Gibbons M (2001) Re-thinking science. Knowledge and the public in an age of uncertainty. Reino Unido Polity Press, Cambridge

    Google Scholar 

  9. Dator J (1972) Futures research: a second look, and yet a another look again. In: World Futures Research Conference. World Society for Futures Studies, Bucharest

    Google Scholar 

  10. Bezold C (1972) Anticipatory democracy. Vintage Books, New York

    Google Scholar 

  11. List D (2006) Action research cycles for multiple futures perspectives. Futures 38(6):673–684

    Article  Google Scholar 

  12. Inayatullah S (2000) Alternative futures: methodology, society, macrohistory and the long-term future. Tamkang University Press, Taipei

    Google Scholar 

  13. Ramos J (2002) Action research as foresight methodology. J Futur Stud 7(1):1–24

  14. Masini E (2006) Rethinking futures studies. Futures 38(10):1158–1168

    Article  Google Scholar 

  15. Hisschemöller M, Hoppe R (1995) Knowledge and Policy 8, vol 40, pp 40–60

    Google Scholar 

  16. Rittel HWJ, Webber MM (1973) Dilemmas in a general theory of planning. Policy Sci 4:155–169

    Article  Google Scholar 

  17. Head BW (2008) Wicked problems in public policy. Publ Policy 3(2):101–118

    Google Scholar 

  18. Boden M, Johnston R, Scapolo F (2012) The role of FTA in responding to grand challenges: a new approach for STI policy? Sci Public Policy 39(2):135–139

    Article  Google Scholar 

  19. Polk M (2015) Transdisciplinary co-production: designing and testing a transdisciplinary research framework for societal problem solving. Futures 65:110–122

    Article  Google Scholar 

  20. Wittmayer JM et al (2014) Making sense of sustainability transitions locally: how action research contributes to addressing societal challenges. Crit Policy Stud 8(4):465–485

    Article  Google Scholar 

  21. Rowe G, Frewer LJ (2005) A typology of public engagement mechanisms. Sci Technol Hum Values 30(2):251–290

    Article  Google Scholar 

  22. Abels G (2007) Citizen involvement in public policy-making: does it improve democratic legitimacy and accountability? The case of pTA. Interdiscip Inf Sci 13(1):103–116

    Google Scholar 

  23. Bobbio L (2010) Types of deliberation. J Public Delib 6(2).

  24. Fiorino DJ (1990) Citizen participation and environmental risk: a survey of institutional mechanisms. Sci Technol Hum Values 15(2):226–243

    Article  Google Scholar 

  25. Boussaguet L (2015) Participatory mechanisms as symbolic policy instruments? Compar Eur Polit 14(1):107–124

    Article  Google Scholar 

  26. Monaghan E (2012) Assessing participation and democracy in the EU: the case of the European citizens’ initiative. Pers Eur Politics Soc 13(3):285–298

    Article  Google Scholar 

  27. Goldschmidt R, Renn O (2006) Meeting of Minds – European Citizens’ Deliberation on Brain Sciences Final Report of the External Evaluation. University of Stuttgart, Stuttgarter Beiträge zur Risiko- und Nachhaltigkeitsforschung

    Google Scholar 

  28. EuropeanCommission. Societal Challenges. 2016; Available from:

  29. Owen R, Macnaghten P, Stilgoe J (2012) Responsible research and innovation: from science in society to science for society, with society. Sci Public Policy 39(6):751–760

    Article  Google Scholar 

  30. Stilgoe J, Owen R, Macnaghten P (2013) Developing a framework for responsible innovation. Res Policy 42(9):1568–1580

    Article  Google Scholar 

  31. Sutcliffe H (2011) In: E Commission (ed) A report on responsible research & innovation

    Google Scholar 

  32. Zwart H, Landeweerd L, Rooij Av (2014) Adapt or perish? Assessing the recent shift in the European research funding arena from ‘ELSA’ to ‘RRI’. Life Sci, Soc Policy 10:11.

  33. Engage2020 (2015) D3.2 public engagement methods and tools. In: Engage2020—tools and instruments for a better societal engagement in “Horizon 2020”

    Google Scholar 

  34. Kuhn R et al (2014) Deliverable 3.1 - report on current praxis of policies and activities supporting societal engagement in research and innovation. In: Engage2020—tools and instruments for a better societal engagement in “Horizon 2020”

    Google Scholar 

  35. Haegeman K, Weber KM, Könnölä T (2012) Preparing for grand challenges: the role of future-oriented technology analysis in anticipating and shaping structural and systemic changes. Tech Anal Strat Manag 24(8):729–734

    Article  Google Scholar 

  36. Haegeman K et al (2015) FTA supporting effective priority setting in multi-lateral research programme cooperation: the case of EU–Russia S&T cooperation. Technol Forecast Soc Chang 101:200–215

    Article  Google Scholar 

  37. Burgelman J-C, Chloupková J, Wobbe W (2014) Foresight in support of European research and innovation policies: the European Commission is preparing the funding of grand societal challenges. Eur J Futur Res 2:55.

  38. Könnöla T, Haegeman K (2012) Embedding foresight in transnational research programming. Sci Public Policy 39(2):191–207

    Article  Google Scholar 

  39. Gudowsky N et al (2012) Forward-looking activities: incorporating citizens’ visions: a critical analysis of the CIVISTI method. Poiesis Prax 9(1–2):101–123

    Article  Google Scholar 

  40. Jacobi A, Klüver L, Rask M (2010) Relevant research in a knowledge democracy: citizens’ participation in defining research agendas for Europe. In: Veld RJ (ed) Knowledge Democracy. Springer-Verlag, Berlin Heidelberg, pp 87–98

    Chapter  Google Scholar 

  41. Gudowsky N, Sotoudeh M (2017) Into blue skies—a transdisciplinary foresight and co-creation method for adding robustness to visioneering. NanoEthics 11(1):93–106

    Article  Google Scholar 

  42. Gudowsky N et al (2017) Transdisciplinary forward-looking agenda setting for age-friendly, human centered cities. Futures 90:16–30

    Article  Google Scholar 

  43. Levitas R (2007) Looking for the blue: the necessity of utopia. J Polit Ideol 12(3):289–306

    Article  Google Scholar 

  44. Eames M, Egmose J (2011) Community foresight for urban sustainability: insights from the Citizens Science for Sustainability (SuScit) project. Technol Forecast Soc Chang 78(5):769–784

    Article  Google Scholar 

  45. Gudowsky N, Peissl W (2016) Human centred science and technology—transdisciplinary foresight and co-creation as tools for active needs-based innovation governance. Eur J Futu Res 4:8.

  46. Irwin A, Jensen TE, Jones KE (2012) The good, the bad and the perfect: criticizing engagement practice. Soc Stud Sci 43(1):118–135

    Article  Google Scholar 

  47. Bora A, Hausendorf H (2006) Participatory science governance revisited: normative expectations versus empirical evidence. Sci Public Policy 33(7):478–488

    Article  Google Scholar 

  48. Bogner A (2011) The paradox of participation experiments. Sci Technol Hum Values 37(5):506–527

    Article  Google Scholar 

  49. Rask M (2013) The tragedy of citizen deliberation—two cases of participatory technology assessment. Tech Anal Strat Manag 25(1):39–55

    Article  Google Scholar 

  50. Porth E et al (2017) Deliverable D3.3 synthesis report specifying similarities and differences in stakeholder/actor perspectives. In: PROSO—promoting societal engagement under the terms of responsible research and innovation (RRI)

    Google Scholar 

  51. Wynne B (1991) Knowledge in context. Sci, Technol Hum Val 16(1):111–121

    Article  Google Scholar 

  52. Wynne, B., Public understanding of science, in Handbook of Science Technology Studies, S. Jasanoff, et al., Editors. 1995, Sage: London. p. 361–389

  53. Bucchi M, Neresini F (2007) Science and public participation. In: Hackett EJ et al (eds) Handbook of Science and Technology Studies. the MIT press, Cambridge, pp 448–472

    Google Scholar 

  54. Delgado A, Lein Kjølberg K, Wickson F (2010) Public engagement coming of age: from theory to practice in STS encounters with nanotechnology. Public Underst Sci 20(6):826–845

    Article  Google Scholar 

  55. Gavigan JP, Scapolo F (1999) A comparison of national foresight exercises. Foresight 1(6):495–517

    Article  Google Scholar 

  56. Alsan A, Atilla Oner M (2003) An integrated view of foresight: integrated foresight management model. Foresight 5(2):33–45

    Article  Google Scholar 

  57. Fernández Güell JM, López JG (2016) Cities futures. A critical assessment of how future studies are applied to cities. Foresight 18(5):454–468

    Article  Google Scholar 

  58. Calof J, Smith JE (2012) Foresight impacts from around the world: a special issue. Foresight 14(1):5–14

    Article  Google Scholar 

  59. ESPAS. Open Repository BAse on International strategic Studies (ORBIS)—European Strategy and Foreign Policy Anal System (ESPAS). 2016; Available from:

  60. Popper, R. Mapping foresight. Revealing how Europe and other world regions navigate into the future. 2009; Available from:

  61. Rosa AB, Gudowsky N, Warnke P (2018) Deliverable 5.2—report on comparison of research topics from CIMULACT with those from expert oriented foresight studies. In: CIMULACT—citizen and multi-actor consultation on horizon 2020

    Google Scholar 

Download references


The data was produced, and part of the analysis was conducted within the EU project CIMULACT, funded under the call H2020-ISSI-2014-2015: Call for integrating Society in Science and Innovation, Topic ISSI-2-2014: Citizens and multi-actor engagement for scenario building.

Availability of data and materials

Supporting data can be found in Additional file 1: Table S1 of the manuscript, as well as in: Rosa, A.B., N. Gudowsky, and P. Warnke, Deliverable 5.2—Report on comparison of research topics from CIMULACT with those from expert oriented foresight studies, in CIMULACT—Citizen and Multi-Actor Consultation on Horizon 2020. 2018 (in print).

Author information

Authors and Affiliations



All authors jointly shaped the presented research, and participated in data collection, analysis, and literature review. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Niklas Gudowsky.

Ethics declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional file

Additional file 1:

Foresight reports chosen for comparison cf. [61]. (DOCX 18 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rosa, A., Gudowsky, N. & Warnke, P. But do they deliver? Participatory agenda setting on the test bed. Eur J Futures Res 6, 14 (2018).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: