A new path in foresight validation? Discussing the socio-epistemic underpinning of foresight quality criteria

Abstract

At present, there are different attempts to establish quality criteria for foresight based on both, the epistemic and the procedural level and all these attempts contribute to a scientific discourse on foresight. But how can we evaluate whether these criteria are scientifically valid themselves and whether they correspond to a form of scientific criticism that is appropriate for foresight validation? This paper introduces exemplarily social epistemology as an underpinning theory for foresight validation by quality criteria. This is shown by the example of the foresight quality criteria developed by Kuusi, Cuhls and Steinmüller [12] in this Topical Collection, which are put in line with Helen Longino’s concept of transformative criticism. Longino has been chosen because her theory includes the consideration of contextual values in discussions of scientific objectivity and provides concrete criteria of transformative criticism. This paper is a major contribution to underpinning foresight by social epistemology and to validating quality criteria by a comprehensive form of scientific criticism, which takes the scientific interdisciplinary nature of foresight into account.

Introduction

It is often emphasized that foresight, and futures studies in general, are located between the arts and science (e.g., [1, 2]). This statement is justified by the methodological variety of foresight, the different fields and topics which can be applied as well as the interdisciplinary nature of foresight project teams and the involved stakeholders [3–5]. Hence, foresight processes are most balanced when different quantitative and qualitative methods are combined and the process includes heterogeneous teams and expert groups (see [6, 7]). At present, there are different attempts to establish quality criteria for foresight based on both, the epistemic and the procedural level [8–12] and all these attempts contribute to a scientific discourse on foresight. Kuusi, Cuhls and Steinmüller [12] for example elaborate quality criteria for the internal, procedural level but also on an external level questioning the quality of the outcomes concerning the customers’ interests. Concerning criteria that question the quality of foresight, it is important to note that these criteria ought to classify if foresight is pursued in a scientific way. This means that based on specific criteria foresight fulfills the requirements of a scientific method. Obviously, there is a need to set up criteria in order to distinguish the quality of foresight in terms of scientific procedures, but also the quality in terms of outcomes. But how can we evaluate whether these criteria are scientifically valid themselves and whether they correspond to a form of scientific criticism that is appropriate to foresight validation?

For this purpose, it may be expedient to apply modern philosophical theories emphasizing the social aspect of knowledge creation in foresight. For example, in recent years philosophers have often supported the position that science has to be pluralistic or social in order to meet an acceptable level of credibility [13–15]. Nowadays, this kind of philosophical research on the social dimension of knowledge is known as social epistemology [16]. It either examines social practices in terms of their impact on knowledge acquisition or on rationality [17]. According to Goldman, knowledge is defined as true belief derived by appropriate knowledge-producing methods which are either internal (including perception, memory, reasoning) or external (including testimony and discourses of others) [17]. In contrast to classical epistemology, social epistemology focuses on the latter point, that is, on “epistemic agents that influence the beliefs of each” [17]. In fact, in philosophy of science knowledge per se is discussed widely and controversially [18] – yet, the socio-epistemic approach is most appealing to contemporary science, as it sets up epistemic criteria that embrace the social and interdisciplinary character of knowledge creation in science. The American philosopher Helen Longino [13] for example has elaborated a concept of transformative criticism which takes values and external knowledge-producing methods and their interdisciplinary nature into account.

The interdisciplinary nature of producing knowledge in foresight can be seen as a typical example where external knowledge-producing methods have a major impact. In related work, interdisciplinary itself has often been discussed in the context of futures, research program evaluation and decision making processes [19–22]. Hence, the knowledge base of such activities, especially in the field of foresight, has never been evaluated by considering the socio-epistemic approach, which seems to be promising due to its emphasis on the social aspect of scientific knowledge.

The aim of this paper is to discuss whether certain foresight quality criteria fulfill the requirements of transformative criticism in order to reveal the potentials of social epistemology for foresight epistemology. As it introduces a first attempt on how to bridge foresight and epistemology by a socio-epistemic approach, it focuses only on one comparative example. Thus, it provides epistemic reflections and insights into a new possible concept for scientific criticism of foresight quality criteria.

The main hypothesis of this paper is that socio-epistemic underpinning may contribute to validate if foresight quality criteria are suitable as a tool for scientific criticism of foresight. For this purpose, it is crucial to define the aim of foresight and the way it is pursued (“What is scientific practice?”section). While “The complexity of epistemic considerations and aims of foresight” section summarizes the complexity of epistemic considerations and aims of foresight, “Scientific methodology with regard to Longino’s social epistemology” section concentrates on scientific methodology, mainly by referring to Longino’s concept of social epistemology. In the third section, the quality criteria for foresight as developed by Kuusi, Cuhls and Steinmüller [12] will be discussed in the light of the set of socio-epistemic criteria by using Longino’s theory. The results will be discussed in regard to their significance for possible theory building and validation in foresight in “Discussing relations: quality criteria, objectivity and relevance” section. Thereby, it is described how foresight can be understood as a socio-epistemic approach, when evaluated by appropriate quality criteria.

What is scientific practice?

Philosophy of science is full of investigations and discussions on scientific practice. In the last century, the positivists, including logical positivists and logical empirists, have been the most influential, followed by post-postivists or wholists and more recently by scientific realists, critical rationalists and pragmatists [13, 23, 24]. Eventually, as Chalmers claims “[…] there is no general account of science and scientific method to be had that applies to all sciences at all historical stages in their development” [23]. This conclusion is especially due to the fact that besides the different scientific methodologies to approach scientific inquiries, different aims are also pursued. For example, even though new experimentalists contributed a lot to experimental reasoning in the last century, the naturalist account, which is paramount to physical and biological sciences, cannot be transferred to other sciences, as experimental manipulation is irrelevant for disciplines like social or historical sciences.¹ In short, for epistemic considerations, scientific practice can be defined on two levels: firstly, regarding scientific methodology; and secondly, regarding scientific aims. Scientific methods can be referred to as positivist (also logical empiricist) approaches relying on empirical evidence or wholist approaches, which are rather theory-laden [13]. Concerning the aims of science, again two main conceptions can be distinguished: The one position sees the “construction of comprehensive accounts of the natural world” [13] as the main goal of scientific inquiry. Representatives of the other position claim that “the work of science is the discovery of the truth about the natural world” [13] – although being skeptical about the possibilities of reaching this aim. Hence, the former aims at knowledge extension while the latter aims at finding truth by scientific inquiry. Thus, the choice of methodology does not imply a certain aim. But theory and methodology choice differs even within scientific disciplines, as different paradigms may be pursued. This is especially due to the social impact on science, e.g., values and personal preferences [25, 26].

In her 1990 publication Science as Social Knowledge, the American philosopher Longino [13] analyzes different aspects of scientific reasoning in order to show the impact of social values on scientific research [13]. Her inquiry considers the main accounts of scientific knowledge, for the sake of brevity, reduced to positivist and wholist approaches. For describing scientific inquiry as social epistemology, she questions the shortcomings of the existing forms of scientific criticism by showing the importance of constitutive and contextual values in scientific inquiry as well as the inability of existing forms of criticism to take them completely into account. Therefore her theory can be seen as a socio-epistemic one [27]. A core issue of her inquiry is the constitution of scientific objectivity by considering these different forms of values. To handle this issue, she proposes different criteria for transformative criticism. As I will show in “Scientific methodology with regard to Longino’s social epistemology” and “A socio-epistemic discussion of foresight quality criteria” sections, these constitutive and contextual values as well as transformative criticism, may contribute to a better understanding of foresight quality criteria as a contribution to fostering a scientific debate on foresight theory.

The complexity of epistemic considerations and aims of foresight

In the field of foresight and futures studies in general, there are different approaches, which tangle epistemic considerations. For example, Mannermaa characterizes different foresight paradigms [28] and Aligica discusses different accounts of scientific criticism in the context of futures knowledge [29, 30] while Grunwald considers options for its argumentative validation [31, 32]. But mostly, foresight is described as a field which supposedly belongs to the social sciences [33, 34]. Fuller and Loogma [35] even discuss foresight as a social constructivist endeavor. Wendell Bell, who describes futures studies as a “[…] transdiciplinary action and social science” [1], provides a comprehensive link between critical rationalism and foresight epistemology. Bell’s definition of critical rationalism involves accounts of scientific realism as well as logical empiricism [1]. Further, he sums up the epistemologies, which underlie the different approaches used in foresight as follows: “Futurists focus on the transformation of hindsight into foresight. On the one hand, they speculate, think laterally, intuit, reason counterfactually as well as factually, cogitate linearly and dialectically, entertain outrageous–and even despised – notions, and creatively invent in order to unveil possible and probable futures. On the other hand, they specify past and present data using a multitude of standard and special methods, collecting, analyzing, and interpreting evidence in order to make posits about possible and probable futures and to construct surrogate knowledge as reliably and validly as they can.” [1] By this definition, which contains methodologies of various sciences, Bell tries to encompass a futures epistemology within the concept of critical realism.

Nevertheless, it should be noted that the impact of the social is also taken into consideration in epistemic discussions of foresight (see also [36]). Bell also emphasizes social biases that may threaten validity as a characteristic point when describing the features of scientific realism [1]. In the second volume of Foundations of Futures Studies he even claims that “[t]he critical realist theory of knowledge can incorporate the testing of value propositions just as it tests truth claims about the past and the present.” [37] As will be shown in the next chapters, Longino’s theory provides even better indications to support the claim that there is an epistemic base connecting objectivity, social aspects and values.

But a closer look at different descriptions of the aims of foresight reveals how important the issue of social aspects really is in foresight and especially in its validation. Following Slaughter, foresight can generally be defined as: “opening to the future with every means at our disposal, developing views of future options, and then choosing between them.” [38] Accordingly, Bell claims that “[t]he purposes of the futures studies are to discover or invent, examine or evaluate, and propose possible, probable and preferable futures.” [1] Besides these partly stretched definitions, in their recent paper Kuusi, Cuhls and Steinmüller also indicate different levels of purposes in futures research [12]. In general, they also correspond to the critical realist view of science. For example, in the previous version of that paper, they propose to consider future knowledge as well-justified:

“According to the conventional definition, knowledge about a topic is justified true belief concerning the topic. Because there is no way to directly ascertain the truth of an anticipation before its defined realization time, the knowledge concerning possible futures can be nothing else than well-justified or well-argued beliefs.” [11].

But there is not only a scientific aim inherent to futures studies. Kuusi, Cuhls and Steinmüller emphasize the following two challenges for the validation of futures research, which also underline the aim of foresight:

“Does the ‘whole picture’ meet scientific criteria? […]

Does the ‘whole picture’ serve their [the customers’ or the target group of users’] interests? Is it relevant for them?” [12]

These two points show the special characteristics of foresight: it is not only expected that foresight practices and outcomes correspond to scientific practices, but also that they fulfill certain aims given by a client or customer. In order to establishing foresight quality criteria encompassing (1) the aims of foresight, (2) the goals of foresight projects and (3) the variety of scientific backgrounds involved, there are two dimensions to be considered: On the one hand, the crucial scientific challenges for foresight are the scientific criteria and the target groups. On the other hand, Longino’s statement according to which scientific activities can be evaluated with respect to both goals, knowledge extension as well as finding truth [13]. Hence, it is crucial to clearly distinguish that the classical aims of science cannot be applied directly to foresight. This also implies that scientific criticism of foresight needs its own criteria and rules satisfying the aims and methods of foresight.

Scientific methodology with regard to Longino’s social epistemology

Regardless of the perception of the aims of science or the method used, a crucial characteristic of all forms of scientific inquiry is objectivity. The claim of objectivity responds to the idea that science should rely upon facts rather than wishes. Science can provide two different forms of objectivity. Firstly, “[…] objectivity is bound up with questions about the truth and referential character of scientific theories, that is, with issues of scientific realism” and secondly, “[…] objectivity has to do with the forms of inquiry.” [13] This means that while the first notion of objectivity refers to objectivity in terms of describing the natural world as it is, the second one relies on non-subjective criteria for developing, accepting, and rejecting the hypotheses and theories constituting a certain point of view [13]. By ascribing objectivity to a scientific method, two main points are intended: Firstly, by asserting the objectivity of data, relying upon them justifies theories and hypothesis, which are inducted. Secondly, while confirming or rejecting that a method is objective, we also evaluate if the methods are the proper means for an unbiased and unprejudiced assessment of hypotheses and theories [13].

Based on these observations, it is possible to define what a scientific method is and which scientific methods foresight consists of. A general definition supported by classical philosophical accounts is that “[…] method, the process by which knowledge is produced, is the application of rules to data.” [13] But how are these rules applied? To answer this question, a closer look at application, process, and rules is necessary.

The application of scientific methods can be summarized as “activities of scientific inquiry”, consisting of (1) producing theories, (2) producing concrete interactions with natural processes, or (3) producing models of it [13, 23].

By taking a closer look at this point, it becomes obvious that the way of conducting science cannot be abstracted from the people conducting it, meaning the process per se. As Longino points out, “[t]he integration and transformation of these activities into a coherent understanding of a given phenomenon are a matter of social negotiations.” [13] These definitions of application and process can also be applied to foresight. The “given phenomenon” Longino mentions may also be a future state or a future scenario that is developed. But foresight does not aim at constructing a “coherent understanding” about the future. Instead, Longino’s notion on “coherent understanding” can be applied to the discursive process of establishing quality criteria in foresight. The former quotation also emphasizes that the “nature of scientific inquiry” [13] is a social one characterized by the following three issues:

• Scientific disciplines are social enterprises, “the individual members of which are dependent on one another for the conditions (ideas, instruments, et cetera) under which they practice.”

• “initiation into scientific inquiry requires education”

• “[A]s the practitioners of the sciences all together constitute a network of communities embedded in a society, the sciences are also among a society’s activities and depend for their survival on that society’s valuing what they do.”

In this regard, the scientific community has to fulfill certain criteria, or rather, follow certain rules for producing objective scientific knowledge. This raises the question how objectivity is reached if science is social? According to Longino, “[a] method of inquiry is objective to the degree that it permits transformative criticism.” By maintaining a critical dialogue, objectivity grows by degree. Longino makes this point explicit by going beyond the classical forms of criticizing hypotheses in science, which are evidential and contextual criticism, and introducing transformative criticism as the outcome of intersubjective criticism. Defining evidential and contextual criticism first, and then showing how Longino induces transformative criticism can explain this best.

Evidential and conceptual criticism

Evidential criticism proceeds “on the basis of experimental and observational concerns” [13]. It questions “the accuracy, extent and conditions of performance of the experiments and observations serving as evidence, and questions their analysis and reporting.” [13] This form of criticism underlines why foresight is often claimed to be unscientific. The analysis and reporting can be applied to a foresight process only with restrictions as results produced in foresight, e.g., future scenarios, can hardly be repeated – and are not even intended to be.

Conceptual criticism proceeds on “the basis of theoretical and meta-theoretical concerns”. Here, three types of questioning can be distinguished [13]:

• Questioning the conceptual soundness of a hypothesis: Longino gives as an example Kant criticizing and questioning the Newtonian hypothesis of absolute time and space.

• Questioning the consistency of a hypothesis with accepted theory: such as e.g. traditionalists who rejected the heliocentric theory as being inconsistent with the existing Aristotelian concept of physics.

• Questioning the relevance of evidence presented in support of a hypothesis: Such a criticism is related to evidential criticism, yet, it questions not only data but also why certain assumptions were taken into consideration as giving evidence to a hypothesis. What matters is the relevance of certain evidence to a hypothesis. Here, criticism becomes intersubjective. Bell’s concept of critical rationalism, which is mentioned in 2.1, can be ascribed to this form of criticism.

As objectivity in the positivist understanding is limited to rejecting or accepting hypotheses based on observational and experimental data, it is only applicable to empirical scientific inquiry. This is conducted e.g., by “syntactically and deductively secured relation of hypotheses to a stable set of observational data” [13]. From a contextual perspective, the proof of evidential objectivity is insufficient as it does not control if and which background assumptions lead to strengthen certain hypotheses. The impact of this consideration on the objectivity of scientific methods is formulated by Longino as follows: “Because the relation between hypotheses and evidence is mediated by background assumptions that themselves may not be subject to empirical confirmation or disconfirmation, and that may be infused with metaphysical or normative considerations, it would be a mistake to identify the objectivity of scientific methods with their empirical features alone.” [13]

Longino extends the last point of conceptual criticism in order to describe that this last form of criticism is transformative. By doing so she aims at showing that this third type of conceptual criticism reveals the impact of background beliefs. The fact that there is such kind of criticism already reveals that background beliefs and assumptions do have an influence on the creation of knowledge and therefore objectivity is to be questioned. As Longino points out:

“Objectivity in the sense under discussion requires a way to block the influence of subjective preference at the level of background beliefs. While the possibility of criticism does not totally eliminate subjective preference either from an individual’s or from a community’s practice of science, it does provide a means for checking its influence in the formation of “scientific knowledge” [13].

It seems that as long as background beliefs concerning a hypothesis are discussed, rejected or altered in the context of criticism by a scientific community, it is possible to establish hypotheses as scientific knowledge free from preferences of any individual. Consequently, scientific knowledge can be defined as social knowledge. Nevertheless, it is important to keep in mind that values still enter scientific debates by individual’s values, but also by community values [13]. The next two sections will therefore describe (1) criteria of transformative criticism and (2) types of contextual values influencing and shaping knowledge.

Objectivity by criteria of transformative criticism

In order to describe how certain degrees of objectivity can still be reached, Longino has formulated “four criteria necessary for achieving the transformative dimension of critical discourse” [13, 14]. The transformative dimension of criticism is the nucleus of intersubjective criticism. The criteria are:

• 1. Recognized avenues for criticism: The most prominent form are peer review processes in scientific publishing. Also, journals and conferences are established avenues for presenting and criticizing scientific outcomes. These procedures help to shape and advance scientific knowledge. Longino claims that critical activities should receive nearly equal weight to “original research” in order to lead to valid and objective scientific knowledge. [13]

• 2. Shared Standards: If criticism ought to be relevant for a scientific discussion, it should also appeal to “public standards or criteria to which members of the scientific community are or feel themselves bound. These standards can include both substantive principles and epistemic as well as social values.” Of course, scientific communities differ. The standards can therefore consist of elements like “[…] empirical adequacy, truth, generation of specifiable interactions with the natural or experienced world, the expansion of existing knowledge frameworks, consistency with accepted theories in other domains, comprehensiveness, reliability as a guide to action, relevance to or satisfaction of particular social needs.” [13] Longino emphasizes that the weighting of the several standards varies not only from one scientific community to another, but also due to different social and historical contexts.

• 3. Community Response: “This criterion requires that the beliefs of the scientific community as a whole and over time change in response to the critical discussion taking place within it.” [13] The indicators for responsiveness are for example contents of textbooks, grants and awards. Critical discussions and responses can help to enhance understandings and assumptions that are guiding for the community.

• 4. Equality of intellectual authority: The last criterion refers to the possibility to “[…] disqualify a community in which a set of assumptions dominates by virtue of the political power of its adherents.”

Individuals take part in this scheme by participating in various critical discussions. All in all, these criteria for transformative criticism enable the evaluation of the objectivity of scientific inquiry itself, but also of scientific debates in society. This is achieved by the fact that at this point, background beliefs involved in science can be detected and discussed actively. Further, a contextual view of science goes beyond the limits of only applying evidential criticism, as it respects the diversity of scientific methods, and it reacts to the current practice of science but also to history. [13]

Types of contextual values influencing and shaping knowledge

Longino resumes that for the validation of sciences, both have to be considered:

• 1. “[…] the role of background assumptions in evidential reasoning”

• 2. “[…] the roles of (sometimes) conflicting goals of inquiry with respect to which hypotheses and theories are assessed.” [13]

The transformative criticism in a scientific community including the four criteria as described above helps to validate scientific inquiries and methods by involving both, the individual and social values that influence background assumptions and science. This is important to keep in mind as contextual values, interests and value-laden assumptions influence and shape not only scientific practice but also the results, even when constitutive rules of science are not violated [13]. In other words, it is possible that a certain scientific process is valid in terms of evidential criticism, but it may nevertheless violate scientific objectivity regarding contextual values. The latter can be validated by criteria of transformative criticism.

According to Longino, the practice of both pure and applied science can be affected and influenced by contextual values in different ways. Nevertheless, it is possible to produce good scientific knowledge when taking into account which of the contextual values might affect our research. Longino therefore sums up a list of five ways in which contextual values might affect our research from the outside [13]:

• 1. Practices. Contextual values can affect practices that bear on the integrity of science.

• 2. Questions. Contextual values can determine which questions are asked and which are ignored about a given phenomenon.

• 3. Data. Contextual values can affect the description of data, that is, value-laden terms may be employed in the description of experimental or observational data, and values may influence the selection of data or of kinds of phenomena to be investigated.

• 4. Specific assumptions. Contextual values can be expressed in or motivate the background assumptions facilitating inferences in specific areas of inquiry.

• 5. Global assumptions. Contextual values can be expressed in or motivate the acceptance of a global framework like assumptions that determine the character of research in an entire field.

These 5 ways help to recognize to which extent the research is affected and to give more emphasis to criticism as for example the intersubjective one. By doing so, it is possible to minimize the contextual values, which might lead to biased results and value-laden assumptions. Although discussing these points may lead to more clarification, e.g., in order to validate specific research programs, Longino acknowledges that even intersubjective criticism is only partially an effective barrier [13]. One remaining problem is for example that those value-laden assumptions shared by a whole scientific community may remain hidden. Obviously, when applying the different forms of criticism, it might appear that intersubjective criticism of processes and outcomes in the field of foresight may be most appropriate. Hence, quality criteria shall also encompass all the different forms of scientific criticism in order to reveal the quality of foresight procedures and outcomes.

A socio-epistemic discussion of foresight quality criteria

In terms of scientific inquiry and based on the aims of inquiry in foresight itself, foresight cannot be clearly assigned to any of the predominant scientific disciplines. However, the aim of foresight per se has an impact on certain groups or is even set by them, e.g., companies, societies, or nation states. Further, the applications and processes are usually used and developed by scholars of different disciplines. So in foresight the aim already reveals its social character, while in other scientific disciplines we cannot reveal that it is social until focusing on the application of rules.

Nevertheless, we are able to fragment foresight processes in order to evaluate their scientific quality by assessing if we can validate it. By doing so, we may examine if the foresight exercise fulfils criteria of scientific objectivity on a socio-epistemic level. Longino’s theory helps us because she does not formulate specific criteria of quality of science that have to be fulfilled by each discipline. Instead, she explains how pluralism in science is enabled and how a scientific community produces valid and objective science regarding specific criteria. This means, that each scientific community sets its own criteria due to the aims and application of methods. If we want to ensure valid scientific knowledge by foresight outcomes, we need to refer to standards that consist of elements that correspond to the scientific methods of foresight. Transformative criticism, therefore, may be the appropriate form for validating foresight outcomes, as it allows encompassing the special social character of aims and processes in foresight. In recent years, some concepts for quality criteria have been introduced to foresight by different actors of the foresight community, e.g., [8, 9, 39]. The criticism criteria for the objectivity of science given in the previous chapter indicate that there is a lot of discussion and confusion about the level of criticism not only in the established scientific communities but also in foresight. Some use evidential and some conceptual criticism, some even contain elements of transformative criticism, like e.g., Bell (see 2.1). Hence, still none of them has assigned their criteria to any type of scientific validation as proposed by the different accounts of scientific knowledge, although the social impact of futures research has often been discussed. In other words, the foresight community establishes quality criteria but these have not yet been discussed in a socio-epistemic framework, which differentiates the diverse forms of scientific criticism.

Introducing foresight quality criteria

In the following, the criteria of transformative criticism developed by Longino are applied to the foresight quality criteria for Futures Maps developed by Kuusi, Cuhls and Steinmüller [12] in order to investigate if the criteria resist scientific criticism on a socio-epistemic level.

The underlying understanding of a futures research practice in this context is that a Futures Map is produced. A Futures Map is defined as “the comprehensive description of the outcomes of a futures research process. It comprises all relevant pictures of the future identified during the process and all relations between these pictures and between them and the present state as well as assessments about time frames, desirability and possibility of these pictures.” [12]. The Futures Map is then evaluated by five criteria of internal validity and six criteria of external validity. For applying the quality criteria the authors refer to the elements of a strategy process in forward looking activities developed by EFFLA [39] as : I) Strategic Intelligence, II) Sense Making, III) Selecting Priorities and IV) Implementation. The five criteria for internal validity which are adopted from EFFLA [39] are, condensed to questions, as follows:

• a) What is the objective of the whole foresight activity? Are there hidden agendas?

• b) What type of activity has to be considered for what type of issues/time spans/ knowledge?

• c) What is the scope of foresight? What is the scope of relevant intelligence and sense making? Is there specific strategic intelligence or are there sense-making projects to be launched? How focused or wide should their scope be?

• d) What is an appropriate set of / combination of / methods to make use of specific actors’ strategic intelligence? And how can this be organized?

• e) What are the intended outcomes of the different stages in the process? In general, reports are written but often the activity as such is an outcome. How are the results presented? [12] As can be seen, main issues are objectives, scope and outcomes, but also the appropriateness of combining methods and tools for specific tasks and involving specific actors. These internal criteria for validation refer mainly to the scientific method applied in a foresight project. Thus, they question the rules applied in the process. The “six pragmatic criteria of external validity of the futures map” are the following:

  • Criterion 1 The number or the scope of possible futures that might be relevant from the point of view of the vision or acceptable futures.

  • Criterion 2 The most relevant or important possible futures are identified.

  • Criterion 3 All kinds of causally relevant facts are covered by the identified futures.

  • Criterion 4 Causally relevant facts are effectively interpreted with as few scenarios as possible.

  • Criterion 5 Many kinds of users of the Futures Map are able to understand and use it.

  • Criterion 6 Key customers of the Futures Map are able to understand and benefit from the Map [12].

All these points imply that a debate is needed to validate which is the more appropriate Futures Map for a certain foresight practice aim or interest of a target group. From an epistemic point of view, the debate can be seen as a social one when arguing with Longino’s concept. Criteria 1–6 can be confronted with Longino’s criteria of transformative criticism.

So in a first step I will embed the quality criteria by Kuusi, Cuhls and Steinmüller in the framework of scientific criticism by highlighting where contextual values play a role. In a second step I will show that in regard of transformative criticism, the quality criteria by Kuusi, Cuhls and Steinmüller represent a comprehensive map of criterion 2 of Longino’s transformative criticism, Shared Standards, but also respond to the other criteria of transformative criticism.

Which form of scientific criticism do the foresight quality criteria respond to?

Regarding the criteria suggested by Kuusi, Cuhls and Steinmüller, we can identify all three levels of scientific criticism as described by Longino. At a first glance, the criteria suggested for internal validity ought to concentrate on procedural aspects of a foresight activity. If these criteria aim at describing only rules for foresight as an acceptable scientific method, we would only have to deal with constitutive values in this case. Accordingly, one can say that these criteria belong to a validation by conceptual criticism. Scientific criticism on an evidential basis can be applied by questioning the appropriateness of applied methods and data only in b). Questions a) c) d) e) are partly situated between conceptual and intersubjective criticism. Criteria a) and d) for example focus on issues which involve the role of specific actors and therefore have to include the validation of contextual values. It can also be argued that the particular question in c) “What is the scope of relevant intelligence and sense-making?” requires a closer look at potential contextual values as it addresses relevancy. Hence, these cases have to be assigned to conceptual criticism. From this observation follows that from an epistemic point of view there is still confusion within the five criteria for internal validation, as it is not clearly limited to the process and applied methods per se.

For the sake of clarity in foresight validation the distinction between two forms of criteria makes sense: The internal criteria focus on processes and the external validity criteria test if results are supported by “weak signals, etc. and well-established theories” [12]. The criteria which Kuusi, Cuhls and Steinmüller suggest for external validity shall lead to “sound reasons to generalize – or to make abduction – from past and present facts to futures relevant conclusions.” [12] Their emphasis on the need for external validity reveals the intersubjective nature of knowledge creation in futures research and the strong relation of foresight to customers’ interests. It is possible to validate these criteria as a social scientific endeavor for two reasons: first, there is a strong focus on relevancy, which is a rather intersubjective point, and the criteria intend to contextualize outcomes with existing knowledge. Nevertheless, from a socio-epistemic point of view based on Longino, the artificial distinction into external and internal criteria appears to be redundant, as the six criteria of external validation already seem to cover all forms of criticism.

Kuusi, Cuhls and Steinmüller indicate that these six criteria can be summarized in pairs as “basic dimensions in validity evaluations” and that there are “[…] connections between the suggested criteria of external validity and the internal validity of futures research processes.” [12] As the first two criteria cover the phase of Strategic Intelligence, we can assume that we can apply criticism on the conceptual level within these two first criteria. In order to respond to criteria 1 and 2 methodological and procedural questions have to be answered, as well. For example, conceptual criticism in this case would correspond to the methodology choice for achieving the foresight aim. Further, if this phase covers Strategic Intelligence, one might also cover a) b) c) and partly d) from the internal criteria in the first of the external ones as they also focus on procedural aspects.

Still, an open point of discussion in this case is that criteria 1–4 emphasize relevance. Now, how is relevance understood in this case? More clarification is needed in the description of this criterion in order to evaluate if this point is possibly a case for validation by conceptual criticism. At least, criteria 1 and 2 refer to relevance in terms of methodological and procedural appropriateness and at the same time to their relevance for the customer. Due to the latter point, there are contextual values involved in these two criteria but also in the following two.

The second pair focuses on Sense Making in futures research activities, which interprets the relevance of the outcomes of Strategic Intelligence. They also define probable futures based on facts of the past and depending on their relevance to different scenarios [12]. Sense making and estimating probabilities for future events is mainly a qualitative process. Therefore, these steps are strongly influenced by contextual values. The appropriate form of criticism to validate outcomes on these levels is the third form of conceptual criticism, which includes intersubjective criticism. This is, e.g., the case when the appropriateness of a selected method is discussed.

The last pair, criteria 5 and 6, “[…] are related to all stages but they are especially important in the phases of Sense Making and Selecting Priorities” [12]. These criteria obviously refer to contextual values involved in the process and are not only to be taken into consideration of the third form of conceptual criticism, but also need to be considered in terms of transformative criticism. This pair of criteria indicates that the intersubjective and value-laden character of outcomes in foresight should actively be addressed in its validation. Hence, transformative criticism supports the point that applying these criteria can in fact be seen as an activity of scientific criticism of foresight outcomes.

Do the quality criteria contribute to transformative criticism?

As shown in the previous section, contextual values appear in each step in foresight processes, so that quality criteria should be able to encompass them in their criticism. The criteria of transformative criticism proposed by Longino may contribute to more objectivity and credibility for the futures research outcomes in the community.

For this case, the quality criteria by Kuusi, Cuhls and Steinmüller can be seen as a major contribution to establishing Shared Standards, the second criterion of transformative criticism suggested by Longino. For example, in their paper the authors contextualize their quality criteria with others, such as EFFLA [39] or Gerhold et al. [8]. Further, the criteria have been introduced to a peer review process in the form of a conference methodology session at the World Conference of Futures Research 2015. By this, a first step towards opening the discussion on Recognized Avenues for Criticism was taken, which is the first criterion for transformative criticism. A continuous discussion of the appropriateness of the different quality criteria is needed to establish a common understanding. However, the concurrently formulation of different quality criteria also indicates that the process of reaching Shared Standards in foresight quality criteria is at the very beginning. Finally, the quality criteria 5 and 6 already indicate that the authors see a need of validating foresight outcomes in regard of their comprehensibility and relevance for lay people as well as for experts. By these criteria the need for Community Response, the third criterion of transformative criticism, which would be applied to the quality criteria themselves, is also integrated in the quality criteria for validating Futures Maps. Hence, the quality criteria by Kuusi, Cuhls and Steinmüller do not only take into account the questioning of contextual values, but already involve some aspects of transformative criticism themselves.

Discussing relations: quality criteria, objectivity and relevance

After applying the quality criteria by Kuusi, Cuhls and Steinmüller to a socio-epistemic framework of scientific criticism, crucial points of discussion are the relations between quality criteria, scientific objectivity and relevance. Thus, the two main issues of foresight validation as formulated by Kuusi, Cuhls and Steinmüller (see 2.1) are refocused:

“Does the ‘whole picture’ meet scientific criteria? […]

Does the ‘whole picture’ serve their [the customers’ or the target group of users’] interests? Is it relevant for them?” [12]

The first question refers to objectivity, while the second one refers to relevance. If foresight is accepted as a socio-epistemic venture due to its aims and procedures, there is a base for investigating if different quality criteria also fulfill the requirements of a socio-epistemic validation. As indicated by matching the different forms of scientific criticism to the criteria, we can conclude that they cover all forms of criticism that is needed in order to establish objective processes and outcomes when creating Futures Maps. The previous chapter shows that the quality criteria suggested by Kuusi, Cuhls and Steinmüller correspond to social epistemology concerning all points, but also in regard of the distinction of internal and external validity. However, the distinction between internal and external validity suggested by Kuusi, Cuhls and Steinmüller needs more accuracy as it might lead to confusion in socio-epistemic discussions.

Seen epistemologically, the internal quality criteria ought to represent the level of validity that refers to constitutive values (referring to sound methodological procedures), while the external quality criteria amount to contextual values (referring to personal or group values). But as indicated earlier, these attributions are not clear enough as they are also used to distinguish the socio-epistemic view on knowledge creation from traditional epistemology (see reference to Goldman in the introduction). The epistemological question about internality and externality encompasses the notion that justifications either rely on facts to which the knowing one or individual has access to (or: knowledge evolves from individuals justifying their knowledge personally, also known as internalist position) or on reasons to which the knowing individual does not necessarily have to have access to (externalist position). Socio-epistemic accounts mostly rely upon the externalist position [17, 40]. In contrast, the distinction between internal and external validity in the quality criteria by Kuusi, Cuhls and Steinmüller do not directly refer to the future knowledge created, but to internal validity in forms of “well-organized process[es]” and “pragmatic and organizational approaches” and to external validity in terms of sound reasons and relevant conclusions. Hence, the criteria for external validity can be seen as related to the future knowledge produced by forward-looking activities. Therefore, when underpinning foresight by social epistemology, a clear understanding of internal and external is needed.

The investigation of the epistemic base of the foresight quality criteria by Longino’s socio-epistemic criteria allows the following answer: The proof of objectivity can validate the claim for scientific procedures. As shown, the six criteria of external validity cover the aspects of transformative criticism. They can therefore provide a scientific validation of Futures Maps or future looking activities in general. But the issue of relevance is rather answered indirectly. Relevance in futures research does not only refer to relevance within the scientific method – this can be validated alongside with objectivity. But relevance is also linked to the users’ interest, as the aim of a specific Futures Map also depends on the aim the customer or the user sets up. Thus, objectivity can serve as a necessary but not sufficient issue for asserting the relevance of special issues in the different steps of a foresight process or when validating Futures Maps. The relevance addressed in the external quality criteria 1–4 referring to procedural and methodological issues can partly be validated by conceptual criticism. But the issues of relevance depending on a) the adequacy of the procedures and b) the decisions in the phases of Sense Making and Setting Priorities for reaching the interests of the target group or customer remain biased: One can try to provide objectivity by transformative criticism and one can try to reveal the contextual values that may impede the results. Again, the decisions about relevance remain customer-oriented rather than epistemology-oriented and therefore lack a comprehensive epistemic validation.

Conclusion

One of the main challenges in foresight validation is the creation of quality criteria that meet the requirements of the different scientific disciplines involved in foresight processes. Hereby, different overlapping but also conflicting scientific aims and processes have to be taken into consideration. This means that even if there were standardized quality criteria for validation, the interdisciplinary character of foresight evokes a vicious circle. On the one hand, input from different disciplines, groups and stakeholders is needed for open out-of-the-box thinking for future projections but on the other it inevitably brings together different disciplines following different ontologisms and scientific aims. The challenge for creating quality criteria in foresight is to encompass all these different ontologisms, processes and contextual values but also to state explicitly the aim of foresight by an appropriate scientific criticism that is understood by all parties involved.

With the detailed depiction of scientific practice, the levels of criticism and the impact of values based on Longino’s theory I tried to show the range of pitfalls which have to be taken into consideration when creating quality criteria for foresight. Quality criteria do not only have to provide evidential criticism on the appropriateness of methods and data, but also intersubjective criticism reflecting values and relevance. By matching the quality criteria developed by Kuusi, Cuhls and Steinmüller with the different forms of criticism, I tried to find out if these criteria meet the requirements of scientific criticism. In the case of objectivity and to reduce conflicts of understanding regarding “internal” and “external” validity, the structure of the quality criteria may profit from more precision, i.e., by integrating the internal criteria into the first pair of external criteria. The external validity criteria have the potential to cover all forms of scientific criticism needed: the evidential, the conceptual and the transformative.

Further, I tried to show that transformative criticism is very useful for validating the future knowledge produced in forward looking activities as it encompasses all forms of scientific criticism and still leaves space for foresight as a strongly transformative and interdisciplinary field to establish scientific standards. The criteria of transformative criticism especially that of Shared Standards, also shows that the various activities in the futures community aiming at formulating quality criteria indicate more objectivity in the field. Hence, socio-epistemic criteria for scientific criticism may help to frame the epistemic foundation of foresight by respecting its particular practices and aims.

The case of quality criteria formulation helps the socio-epistemic justification and vice versa. Applying the socio-epistemic framework to the foresight quality criteria also showed that main issues, e.g., those of Community Response, are already considered in the quality criteria. Hence, the inquiry in this paper also supports the socio-epistemic position of science as social knowledge not only by the strong influence of contextual values in foresight, but also by the active debate on quality criteria as transformative criticism of the foresight community itself.

Another remaining question for further research will be to examine whether it is more appropriate to apply the socio-epistemic view to quality criteria or to specific futures methods themselves. Further work may also consist of applying transformative criticism to other sets of quality criteria and should also take into consideration other theories indicating socio-epistemic frameworks of science, e.g., Philipp Kitcher’s theory [15].