The futures Map and its quality criteria

The article discusses quality criteria for futures research in the frame of so-called ‘Futures Map’. This article is the edited and developed version of the article (Futura 1: 60-77, 2015). Based on the comments concerning (Futura 1: 60-77, 2015), we have developed the Futures Map frame and its validity criteria further. The new edition of the article is more focused on two basic questions. What is the relationship between the Futures Map approach and various scenario approaches? What are the common and different features of the quality criteria suggested in the Futures Map frame and other suggested quality criteria? The comparison is especially focused on the quality criteria defined by the German Netzwerk Zukunftsforschung. We suggest that it is reasonable to handle quality criteria on three basic levels: the philosophical basis; pragmatic general quality criteria; and special quality criteria suitable for specific contexts/uses. The prospects of futures research as a field of science depend first of all on their research methods. The article briefly discusses the choice of research methods as well as the evaluation of particular applications of methods in practical research projects.

This article continues the discussion opened by [31] published in spring 2015.^{Footnote 1} The article [31] was used as the anchor paper for discussions in six methodology sessions of the international futures research conference called “Futures Studies Tackling Wicked Problems” in Turku. As the earlier article, the new article describes the Futures Map frame and the quality criteria of the futures research defined in this frame. However, based on many changes, the new article differs much from [31]

According to [31], programs and projects that study the future from various methodological perspectives and focuses have multiplied. There is, however, no common understanding of these studies’ quality– not even of the criteria according to which the quality of foresight can be assessed or evaluated. The exception is the evaluation literature which mainly covers criteria of performance [1, 8, 16].

In this article, we will continue the discussion about the quality criteria of futures research opened by [31] The lack of a common understanding of the quality of futures research is connected to the lack of a coherent conceptual frame of the field that is vaguely defined by the concepts of ‘futures studies’, ‘foresight’ and ‘futures research’. A number of similar approaches under different names or with the same names can be found, e.g., scenario method or scenario technique (and sometimes even in short ‘scenarios’ where the term for the result is used for the method) for completely different procedures although the general idea behind them is the same. In this paper, besides the quality criteria discussion, we suggest a frame for the various concepts of futures research. The quality criteria we suggest are connected to what is called ‘Futures Map’ frame.

There are various reasons – some old and some recent – that motivate our effort:

• Practical work of futurists requires making a difference between high quality and low quality work, especially with the aim of improving the use of methods in futures research.

• Because creative imagination is an integral ingredient of futures studies, it is especially important to define what forms of creative thinking are acceptable in scientific futures research and in which phases and frames they are acceptable. For some years, training programs in futures research have started in several European universities. The unclear position of futures research in the field of sciences has seriously hindered its development towards an established discipline.

• Quality problems have also been an often discussed theme in the futurist network of the global Millennium Project. The quality requirement is a special challenge for the Global Futures Intelligence System GFIS (http://www.millennium-project.org/millennium/GFIS.html). The authors of this article belong to or are connected to the Millennium Nodes in Europe.

For the practice of futures research, the second challenge is especially difficult to manage. Using the illustrating metaphor, research without open minded creative imagination is like bones without flesh or pictures without colors. On the other hand, futures research without fact-based justifications is like flesh without bones or colors without a picture.

We will use the concepts ‘futures studies’, ‘foresight’ and ‘futures research’ in the following way: Futures studies refers to all kinds of approaches studying the future or futures. The concept foresight has a similar broad content though foresight stresses more the pragmatic side of futures studies and is defined as a systematic debate about different futures see e.g., [7, 9] or [33]. The concept ‘futurist’ refers to all ‘scientific’/ ‘research’ practitioners in the field of futures studies or foresight as long as their work is of a serious nature and not simply fantasy. Futures studies and Futures Research are mainly found in university environments. Foresight is found in applied sciences, research institutions, practical applications and meanwhile also consultancies.

The main focus of this article is the field of futures research. We reserve the concept futures research for those futures studies that are looking for pragmatically valid knowledge concerning possible futures. Validity means that this knowledge is based on facts, assumptions and reasons (including methodological approaches) that can be justified in discussions with other people, i.e., supported by well-founded argumentation.^{Footnote 2} It combines external validity (supported by facts and well-established theories) with internal validity (consistent reasoning, appropriate methodology).^{Footnote 3}

Plausibility” is the concept that many futures researchers have used instead of “validity” or “probability” [15, 23]. Why do we use “validity” and not the less demanding concept “plausibility”?

We realize the limitations of futures’ related knowledge very well. According to the conventional definition, knowledge of a topic is the justified true belief concerning the topic [34]. Because there is no way to directly ascertain the truth of anticipation before its defined realization time, knowledge concerning possible futures can be nothing else than well-justified or plausible beliefs. The future related knowledge shares the verification problem with all generalizing hypotheses. However, as especially the representatives of critical realism have stressed you can at least in principle always try to falsify a generalizing hypothesis [34]. This concerns also those causal processes that motivate the anticipations which have been made. Futures researchers typically anticipate the development of complex fields with various interacting causal processes. In practice, you cannot falsify such a complex system of interacting causal processes. However, a new recent fact – e.g., a weak signal – typically has an impact on the plausibility of many assumed causal processes and related scenarios. More plausible scenarios might also be contradicting scenarios. In scenario processes, it is reasonable to speak just about the increased plausibility of a single scenario and not about its increased validity or probability.^{Footnote 4} On the other hand, it is reasonable to assume that the total picture based on alternative scenarios or the Futures Map is more valid if it takes into account the new weak signal.

Let us illustrate the above reasoning with a case example of a typical scenario process that follows the stages suggested by Steinmüller [37]. Let us assume that the focus area of the scenario process (or the construction of the Futures Map) is the global energy market in 2030. We make first hypotheses concerning causal processes of the energy market based on past trends and actor strategies. Based on these hypotheses we construct possible scenarios. Let us assume that the Renewable Energy Sources scenario and the Counterattack of Oil Producers scenario belong to these draft scenarios or to this first draft of the Futures Map. Let us assume that we notice that a Chinese firm has started to supply very low price solar panels. We take this weak signal into account in the two very different scenarios. In the Renewable Energy Sources scenario, this might mean that the consumption of oil will be on a much lower level in 2030 than it is now. In the other scenario, Saudi Arabia increases its oil production in order to avoid the transformation of the market towards the use of solar energy. We can conclude that both scenarios are now more plausible and the validity of our Futures Map has increased based on the noticed weak signal.

The case example illustrates why it is not acceptable that some futures studies simply present a selection of recent facts (developments, trends, weak signals…) that justify a particular future without taking into account similar facts that point in another direction. According to our interpretation, these types of one-sided justifications do not deserve the name ‘futures research’ and even less the name ‘scientific futures research’. Futures research should always be based on the ‘whole picture‘of relevant futures. The validation of the ‘whole picture’ is also a solution to the problem mentioned by Grienitz et al. [15]. According to them “regarding all possible developments that may be relevant for a problem, each development has an infinitesimal probability of coming true”.

In the following, we will suggest six pragmatic criteria for the external validity of the futures map. They define practical criteria for the validation of the ‘whole picture‘. Concerning the suggested criteria, it is important to realize that the ‘whole picture’ is produced and adapted for certain customers or specific users. Thus, there are two basic challenges in futures research. The first main challenge is scientific rigor: Does the ‘whole picture’ meet scientific criteria? Does it deserve the name of ‘research’ or is it just based on trend searching and arbitrary interpretation in a ‘closed shop’ and in no transparent way? The other main challenge concerns the customers or the target group of users of the ‘whole picture’. Does the ‘whole picture’ serve their interests? Is it relevant for them? What is feasible from their point of view and what does the client or sponsor really want?

Wendell Bell [3] defined the purposes of futures studies as follows:

The purposes of the futures studies are to discover or invent, examine or evaluate, and propose possible, probable and preferable futures.

We consider that Bell’s definition characterizes the identity of futures studies though we would replace the word “discover” with the concept “construct” The purposes mentioned are well in line with the purposes of two main approaches of futures research: the scenario approach and Delphi surveys. The scenario approach and the related methods, first of all the morphological matrix- method (or the futures table-method), are used to construct or to propose possible futures. Scenario methods also provide systematic frames for the examination and evaluation of futures. Some versions of Delphi surveys can also be used to invent futures but most Delphi versions are mainly used for other purposes (e.g., sense-making, assessments, see [11] or [12]). Experts use Delphi studies to evaluate how probable and preferable or how important some specified futures are.

The Futures Map provides a conceptual frame that helps to evaluate how futures researchers have proceeded in the promotion of the purposes defined by Bell. In that way, it is also a suitable frame for the discussion about quality criteria of futures research, since the quality of research results should be measured not against some singled-out elements but the ‘whole picture’ that the futures study paints. For this ‘whole picture’ Malaska and Virtanen have introduced the concept of Futures Map^{Footnote 5} as follows [32]:

… map is a source of information about the scenery, a symbolic replica of some characters of it. There is a relationship between the map’s designs and symbols and the real scenery at some level of coarseness and vagueness…In geographical mapping the elementary symbols and patterns of the map represent different elements of the scenery, e.g. trees, lakes, meadows…In the same way a futures manifold (or map) is a symbolic representation of the future…

Based on the introduction of Malaska and Virtanen [32], we define:

A Futures Map is the comprehensive description of the outcomes of a futures research process. It comprises all relevant pictures of the future identified during the process and all relations between these pictures and between them and the present state as well as assessments about time frames, desirability and possibility of these pictures.

The Futures Map includes all possible futures (‘futuribles’ ^{Footnote 6}) as identified during the research process. Any picture of the future, when assessed according to time frame and desirability, has a definite place in the Futures Map. It may even occur that two pictures of the future occupy the same point in the two dimensions of time and desirability since they have identical assessments. Nevertheless, they can be completely different according to depth and content. We can place them differently in the third dimension. Pictures of the future are descriptions of a future state of affairs, sometimes rather lengthy narrative ones, sometimes rather short ones with only some key figures about society, economy etc. When placing the pictures of the future into the map, many dots and spaces in the Futures Map may be left empty since there is no corresponding picture of the future.

In our basic illustration of the Futures Map, any future or picture of the future is characterized by two parameters or dimensions: when it is assumed to realize (time of realisation, x-axis) and how the users of the map will appreciate its realization (desirability, preferred futures), see Fig. 1. The higher a point on the y-axis the more desirable the picture of the future is. That means normative as well as non-normative questions are considered. Another parameter (not shown in the illustration below) is possibility: Whether a picture of the future may – according to our best knowledge – realize or whether it is only an abstract option that does not fit reality (utopia, pipe dream…). We reserve the third dimension for the free arrangement of pictures of the future and pathways connecting them. This third dimension spans the ‘depth’ of the terrain.

Scenario paths, a trend, a road map, (Whereas a trend is just a continuous development that can be more or less desirable and is not defined by desirability, the roadmap leads towards a desirable future, one that is chosen actively for planning (goal, vision) a vision and acceptable futures on the Futures Map

Full size image

In this article, we will not describe in detail how we have integrated the basic concepts of futures research into our Futures Map frame. The detailed discussion is presented in an article in Futura [31]. Instead, we describe and motivate the Futures Map frame comparing the Map approach and the various versions of the scenario approach.

The descriptions of the scenario approaches are typically descriptions of scenario processes. For example Steinmüller [37], has described the basic steps of the scenario approach with eight basic steps. The first step is the structuring of the research field and the final stage is to look for suitable solutions to the issues or problems identified based on the scenarios and disturbing wild cards. From the point of view of the Futures Map frame, the scenario process of [37] can be seen as the integrated construction process of the Futures Map and the decision making based on it.

As in various scenario approaches, the “scenario” is the key concept of the Futures Map. In the Futures Map frame, the scenario is a specified path on the Futures Map connecting the present state to at least one picture of the future. In some scenario approaches, also broader contextual pictures of the future are called scenarios though this is a little confusing.^{Footnote 7} In Fig. 1, the straight lines with decision (bifurcation) points illustrate possible scenarios. The original definition of the term ‘scenario’ in futures research as given by Herman Kahn is a good way to describe the role of scenarios in the Futures Map [28].

Scenarios are hypothetical sequences of events constructed for the purpose of focusing attention on causal processes and decision-points. They answer two kinds of questions: (1) precisely how might some hypothetical situation develop, step by step and (2) what alternatives exist, for each actor, at each step, for preventing, diverting, or facilitating the process.

Besides Kahn, also many other prominent futures researchers e.g., Michel Godet [20] have defined the scenario similarly. Like most scenario approaches, we also suggest that any possible future requires a kind of looking backwards from the picture of the future, a kind of backcasting from the (picture of) the future to the present evidence. John Robinson [36] suggested backcasting as a normative and design oriented scenario approach that unlike the French prospective approach does not evaluate probability of scenarios. Recently, the backcasting approach implies a broader approach where all kinds of scenario paths starting from some picture (s) of the future are back to the present discussed.

Many scenario approaches use the concept of the scenario funnel. For example, this concept is especially important in the scenario approach of [39]. The basic idea of the scenario funnel is that the farther we gaze from today’s standpoint towards the future the more possibilities are open [29]. In Fig. 1, this idea of the funnel is taken into account in the scenario paths of the illustration. They all start from a “now” point and the range of the scenario paths increases with time. Scenario funnels are often used in a way that the most probable scenario is in the middle of the funnel. This idea does not work in Fig. 1 where the possible scenarios are presented in the order in which the most desirable scenarios are at the top.

The Futures Map is first of all the tool for the decision making of its customers. This feature is stressed in Peter Schwartz [38]’ scenario approach. According to him “Scenarios are tools for ordering one’s perceptions about alternative future environments in which today’s decisions might be played out.” You can see the similar point of view in the much-used statement: “There is no need to know the future, but to be prepared for the future” [5, 23]. The final position, the picture of the future where one arrives, depends on the choices the actors make on the way, on their purposes and capacities. Theodore Gordon [22] expressed this important starting point of futures research in the following way: ‘There is a future without action, and a different one with it. Thus futures research and predestination are, at least on the surface, antithetical’. In other words: people decide on the future path by action or inactivity – there is always an alternative. Of course, the futures map is often based on causal processes that do not function as the map assumes. Concerning any action based on the map, the actor has to be prepared that her or his expectations concerning the impacts of his or her action will not realize.

Other key concepts of the Futures Map that are also relevant for many scenario approaches are the vision, acceptable futures, the aspiration level of the actor (s), the accessibility and the trend or the business-as-usual scenario. Concerning their definitions in the Futures Map frame we refer to [31].

How can the relationship between the Futures Map frame and various scenario approaches be summarized? We consider that the Futures Map frame provides the well-working frame that defines in a consistent way key concepts used in various scenario approaches and in other kinds of futures research efforts.

In particular, the frame is useful for the classification of the time horizons used in futures research.

In futures research, the time horizon is one of the key issues. A rather common opinion among futurists seems to be that the time horizon of scenarios is longer than the time horizon of roadmaps or plans. Scenarios might have the time horizon of 100 years. The time horizon of the planning horizon/ the road map is nearly never longer than 20 years. From the point of view of futures research, the following statement of the English Wikipedia (“Time Horizon” 1.10.2015) is very questionable: A time horizon, also known as a planning horizon…

In the Futures Map frame it is possible to solve the conceptual problem concerning scenarios, roadmaps and plans using the distinction between the planning and the mapping horizons. The mapping horizon is the anticipation horizon of the Futures Map (of the possible futures). As in picture 1, a scenario path might de facto end already before the mapping horizon. For example in the Futures Map of a company, the bankruptcy of the company might be an end point for the individual company, but in fact, the scenario without this company might be going on, unconsidered by the managers of the company. However, the time horizon of most scenarios is defined by the mapping horizon. As in the picture, there might be many scenario paths to the same end point of the mapping horizon.

We connect the planning horizon of the Futures Map to the concept and method of the roadmap. In the picture, the dotted line that starts from the ‘now’ box (the present state of affairs) illustrates a roadmap. During the time frame of the planning horizon, the involved actors are committed to follow the specified road of the map – the roadmap. For example, a company decides that during the next 2 years all mobile phones they produce follow the same specified standards. A roadmap requires a vision or some other orienting picture of the future in the mapping horizon, planning horizon (time frame) and a planning horizon goal.

Typically, it is reasonable to create and design the road map for a shorter period than the mapping horizon of the Futures Map. In the illustration, when one has followed the road of the map to the end of the dotted line, the world is not the same as in the ‘now’ situation any more – it has changed and every new action starts under different conditions. At the end of the planning horizon, new evaluations of the vision and the acceptable futures are needed. Earlier adaptations are possible and recommended (e.g., in yearly strategic meetings adaptations in the roadmap can be made).

Basic dictionary meanings of ‘valid’ are sound, just or well-founded [40]. In logic it means that conclusions are based on established / agreed-on premises. In empirical sciences, the validity of generalizing theories or statements is, however, a matter of degree because you can never be sure that a general statement is true. Testing the theory, you can possibly falsify some of its conclusions but if no better theory is available you are justified to conclude that the theory is more valid than that of your competitors. The suitable definition of pragmatic validity in empirical sciences is the following: valid proposition, inference or conclusion is the best available approximation of the truth.^{Footnote 8}

Using the above definition of validity, we conclude that one of the key tasks of scientific futures research is to improve the pragmatic validity of Futures Maps. This focus is especially useful for the evaluation of the quality of the scenario process. The Futures Map approach looks at the quality of the scenario process from three points of view: the (internal) validity of the construction process of the Futures Map; the (external) validity of the constructed Futures Map; and the quality of the decision making process based on the Map. As we discussed above, the validation concerns the whole Futures Map. The validation increases or decreases the plausibility or probability of its single elements e.g., the scenarios.

Any empirical science defines its own criteria of pragmatic validity.^{Footnote 9} A common distinction in empirical sciences is the distinction between internal and external validity. Internal validity means that the research results are obtained using sound research methods. Besides the sound use of its various methods, we assume that the internal validity in futures research requires well-organized processes. Like many scenario process versions, they should be activity-oriented and contribute to the ‘shaping of the future‘as pragmatic and organizational approaches. The EFFLA Group [14] has suggested that foresight (in the sense of ‘applied futures research in the policy context’ but including open debates, see the definition above) should always be a process that integrates strategic intelligence, sense-making activities and the link to the policy cycle (see Fig. 2).

The elements of a strategy process (Kerstin Cuhls’ configuration of figure from [12])

Full size image

Like the EFFLA Group, we consider that a good way to promote the internal validity of the futures research process is to answer the following practical questions in the starting phase of the process ([14] p. 2–3). Answers to these questions help when planning the internally valid futures research process that also promotes externally valid results (the high quality Futures Map, see below).

1. a)

   What is the objective of the whole foresight activity? Are there hidden agendas?

2. b)

   What type of activity has to be considered for what type of issues/time spans/ knowledge?

3. c)

   What is the scope of foresight? What is the scope of relevant intelligence and sense-making? Is there specific strategic intelligence or are there sense-making projects to be launched? How focused or wide should their scope be?

4. d)

   What is an appropriate set of/ combination of/ methods to make use of specific actors’ strategic intelligence? And how can this be organized?

5. e)

   What are the intended outcomes of the different stages in the process? In general, reports are written but often the activity as such is an outcome. How are the results presented?

Campbell and Stanley [4] have presented a classic definition of the external validity in behavioral sciences:

External validity asks the question of generalizability: To what populations, settings, treatment variables and measurement variables can this effect be generalized?

In the case of futures research, the external validity means that there are sound reasons to generalize – or to make abduction^{Footnote 10} - from past and present facts to futures relevant conclusions. The Futures Map summarizes these generalizations or abductions. External validity therefore means that the research results, the Futures Map, is supported by facts and observations like existing trends, weak signals etc. and well-established theories concerning the relevant causal processes. The pragmatic validity of the Futures Map increases if relevant actors are able to use it. If most relevant actors do not get the map or do not understand its messages some opportunities, which are otherwise accessible and are identified in the map, might not be part of accessible possibilities. This concerns, of course, also the avoidance of bad possibilities.

For the pragmatic description of the validity of futures research results, we suggest six validity criteria that make the validity comparisons of different Futures Maps possible^{Footnote 11}:

1. Criterion 1

   The number or the scope of possible futures that might be relevant from the point of view of the vision or acceptable futures

2. Criterion 2

   The most relevant or important possible futures are identified

3. Criterion 3

   All kinds of causally relevant facts are covered by the identified futures

4. Criterion 4

   Causally relevant facts are effectively interpreted with as few scenarios as possible

5. Criterion 5

   Many kinds of users of the Futures Map are able to understand and use it

6. Criterion 6

   Key customers of the Futures Map are able to understand and benefit from the Map

An important idea of the suggested criteria is that no Futures Map can be a high quality map in all criteria. The researcher has to select those criteria of external validity in which his or her Futures Map tries to be especially good. From this follows that there is no absolute ranking order of the Futures Maps based on the criteria. You are just able to say that the Futures Map 1 (FM1) is better than FM2 in the external validity if it is better in at least one criterion and as good as FM2 in other criteria.

One key concept related to the criteria of the pragmatic external validity is the concept “relevancy”. In the Futures Map frame, the relevancy is the customer related concept. Some feature of the Futures Map is relevant if it includes information that is useful for the customer’s choices. Though the concept of “relevancy” is mentioned just in criteria 1–4, the Futures Map cannot be relevant if it is not understood by customers of the Map (the criteria 5 and 6).

What is the connection between the relevancy and the pragmatic external validity? A way to define the pragmatic validity is the criterion that the customer’s expectations based on the Map will realize. There is, however, a way to express the quality requirement of the Map so that both the validity and the relevancy aspects are present: If the content of the Map is both valid and relevant the customer is able to make choices that she or he will not regret. Because the customer might be happy also in the case of the unexpected development this criterion is more general and suitable than the realized expectations. For example, the result might be better than anticipated. Actually, [30] has suggested that the regretted or not-regretted choices can be seen as the general epistemological basis of the futures related learning and knowledge.

We can improve the external validity of the Futures Map in a way that does not depend on specified customers. Not depending on the specified customers, we can in principle test the possibility of any assumed hypothetical causal process of the Futures Map. Of course, in the case of complex processes with many kinds of interacting causal processes, this is not often possible in practice. On other hand, there is the customer specific aspect that combines the validity and the relevancy. If the Map includes irrelevant causal processes or futures it is less suitable for the not-regretted choices of the customers because of the information overflow. The difficulties in understanding and the information overflow are taken into account in criteria 5 and 6.

The simple way to classify causal processes / futures into relevant and not relevant is by asking the customers. However, this does not guarantee the fulfillment of the not-regretting requirement. The customer might have an opinion on the relevance that is based on lacking knowledge. For example, we are justified to assume that many of the people who were killed by lung cancer in the 1950s would have stopped smoking based on our recent knowledge concerning the dangers of smoking. However, as the case of smoking nicely illustrates, even very well-justified evidence does not necessarily change the perceived relevance evaluations and related choices. Even if a person is not ready to accept the valid evidence, the Futures Map that takes the evidence into account improves the Futures Map from his or her point of view ^{Footnote 12} We can conclude that focusing on the not-regretting criterion the objective or scientific validation of the futures map is possible both from the point of view of the not customer specific and the customer specific relevancy aspect. ^{Footnote 13}

For the identity of futures studies the first criterion is especially important. According to the Futures Research Methodology 3.0 of the Millennium Project, ‘perhaps the most commonly understood reason for the use of futures methods is to help identify what you do not know, but need to know, to make more intelligent decisions.’^{Footnote 14} . The identification of new possible scenario paths shows relevant gaps in our knowledge and in that way the validity of the Futures Map increases whenever it is able to evoke more (relevant) possible futures. ^{Footnote 15} The validity improves even without critical fact-based examination of the suggested new possible paths (criterion 1). However, if the map includes, ceteris paribus, the most relevant path it improves the validity of the map (criterion 2).

As Malaska and Virtanen [32] remarked, futures are just partly determined by the known facts. It is possible that just some weak signals give indications of the most relevant futures. So it is important to take into account aall causally relevant past or recent facts (criterion 3). But if you are able to construct a map of a few scenario paths that takes account of a given set of facts and observations and gives an interpretation of their effects, it is ceteris paribus better than a map that needs more scenarios for its interpretation (criterion 4).^{Footnote 16} A nice example of the use of criterion 4 (from another field) is linear regression analysis that interprets the variance of past evidence with a trend assuming that the trend will continue also in future.

Even if a futures researcher has identified causally relevant possible futures and has taken into account even weak future signals something more is needed for a valid Futures Map. The validity of the Futures Map requires more than the correspondence of the Futures Map and past or present facts. Assuming that users or customers of the Futures Map themselves are the best experts of choices that they will not regret (compare [30]), they have to understand its relevant messages (the criterion 5). Criterion 5 is especially relevant when the common understanding of possibilities and relevant past facts are important for common choices of actors. However, in some situations, it might be important that just key customers understand the map and those who have an interest in hindering most favorable futures of the map’s customers do not understand it (criterion 6).

There are connections between the suggested criteria of external validity and the internal validity of futures research processes. Typically, the identification of potentially important paths (criteria 1 and 2) belongs to the first phase of the futures mapping process. In Fig. 2, it is Strategic Intelligence. The collection of potentially futures-relevant facts belongs also to the strategic intelligence phase but the interpretation of their relevance in various scenarios belongs to Sense Making (criteria 3 and 4).

Criteria 5 and 6 are related to all stages but they are especially important in the phases Sense Making and Selecting Priorities. A good scenario is not just possible and consistent but it should also be believable, trustworthy and interesting. This is an especially difficult challenge concerning those possibilities that challenge recent values and beliefs of the Futures Map’s customers. In order to manage this challenge, the futures researcher has to make the customers understand their prejudices. In practice, this challenge is highly related to the use of interesting and convincing metaphors. To this end, Causal Layered Analysis is a suggested approach [25].

The validity criteria function pairwise so that criteria 1 and 2; 3 and 4; and 5 and 6 define basic dimensions in validity evaluations. These basic dimensions correspond to the “leading questions” of research as identified for typical research styles suggested by Jerome Ravetz [35]. According to Ravetz the leading question is ‘what/how’ for research, the outcome of which is a statement intended to be factual. What/how research combines substance and agency: ‘what is this made of’ or ‘how does this cause that’. These questions are relevant especially concerning the validity of criteria 3 and 4. Statistical forecasting methods - e.g., regression analysis - are useful in predictions based on causal connections between specified variables. In these kinds of anticipations, criterion 3 means that scenario paths of the Futures Map FM1 take into account more past trends of possibly relevant variables than the scenario paths of FM2.

The validity of criterion 4 means in these kinds of anticipations that FM1 causally interprets the past variance of facts with fewer trends about ‘how the future might develop’ than FM2. For example, think about weather forecasts. In the 1950s, the forecasts provided many possible scenarios for the next few days. Now we have a much more narrow scope of possible weather scenarios for the next 3 days, based on much richer past evidence and rather better models than in the 1950s. So recent weather Futures Maps are more valid based on both criterion 3 and criterion 4.

Ravetz’ second style of research is focused on ‘how/why’ questions. This style accepts concepts like ‘final course’, ‘function’ or ‘purpose’. According to Ravetz, here the point is the design that is able to perform a given function, to do its job. ‘How/why’ questions are reasonable only in the framework of actors with their interests (compare [30]). The questions of type “why” are especially relevant for the validity criteria 5 and 6. If an actor is not able to understand how his or her interests are connected to a possible future he or she is not interested in promoting that future.

Concerning the validity criteria 1 and 2, Ravetz’ third research style is the most relevant. According to Ravetz, ‘what if?’ expresses the spirit of creativity, of inventiveness, forays into an unknown that is passive and expectant. For Ravetz the role of ‘what if’ questions was first of all the management of treating ignorance related to the impacts of new technologies and environmental hazards e.g., possible hazards of new chemical plants. We can, however, generalize ‘what if’ questions to concern also positive futures that are possibly just based on weak signals. ‘What if?’ questions challenge prevailing anticipations and action routines and, actors realize new options for action based on them. The focus is actually at least as much on new questions as on the suggested answers. This idea is stressed by Michel Godet [21] who cited Woody Allen: ‘The answer is yes, but what was the question?’

Next we will compare the validity criteria of the Futures Map with other suggested quality criteria of futures research.The quality problem on the general level is discussed in many books that are used in academic teaching or as guides to practical work of futurists. A main theme of Wendell Bell’s Foundation of Futures Studies [2] is the nature of futures studies as a scientific activity.^{Footnote 17} For technology foresight directly linked to policy-making, the major reference for ‘practitioners’ internationally is “The Handbook of Technology Foresight’ [16].

The journals of the field, especially Technological Forecasting and Social Change, Futures, Foresight and The Journal of Futures Studies and the new European Journal of Futures Research or the Zeitschrift für Zukunftsforschung (published in German) have served as platforms for discussing good practice in futures research. The international Millennium Project has an acknowledged role in the development of shared understanding of the field. Though the Millennium Project is open for both scientific futures research and artistic futures studies, the quality issues are met especially in its Delphi related activities.

The quality criteria of futures research have been discussed on three basic levels:

• The philosophical basis

• Pragmatic general quality criteria

• Special quality criteria suitable for specific contexts/uses

On these three levels, the importance of the internal, external and customer specific quality criteria differ.

Up to now, futures researchers have found the philosophical basis for their research efforts mainly from two sources: critical realism and constructivism. Wendell Bell [3] has been the most prominent promoter of critical realism in futures studies. Some radical constructivists have connected critical realism to ‘positivist’,‘behaviorist’, ‘empiricist’, or ‘technocratic’ approaches. According to Jakil [26], many constructivists share their philosophical basis with ‘post-modernistic’, ‘post-positivistic’, ‘post-empiric’, ‘interpretative’, ‘discursive’ or ‘deliberative’ approaches. Less radical constructivists used to think that both critical realism and constructivism are feasible philosophical starting points for futures research. The common feature of constructivist epistemology and critical realism is the idea that human knowledge is based on continuously developing conceptual systems. Theories or constructions are “true” or “working” as long as they are not shown to be untrue or not-working. A philosophical approach that shares this common starting point of constructivism and critical realism is the learning based paradigm of the General Theory of Consistence (GTC) [30]. When the six criteria of the pragmatic external validity of the Futures Map were first suggested in 2011, they were presented in the frame of that philosophical approach.

On the level of special quality criteria suitable for specific contexts, specific quality criteria are developed for specific uses of the Map’s customers. For example [27] suggested three quality criteria suitable for quality evaluation of the Sustainability Governance Foresight (SGF) on the EU level that takes into account culturally sensitive frames of policy issues. High quality SGF:

• detects conflicts in framing policy issues;

• facilitates citizen deliberation; and

• brings to the fore the grass roots knowledge at all stages of policymaking and policy inquiry.

The criteria of the internal and external validity of the Futures Map belong to the level of pragmatic general quality criteria. In the present state of the art, they just provide a simple, theoretically sound and systematic heuristics that is suitable e.g., for the evaluation of student works in futures research. Another practical and more detailed portfolio of general quality criteria has been suggested by the Task Force Standards (TSF) of the German Netzwerk Zukunftsforschung.^{Footnote 18}

According to ([18] p.12), the standards of the TSF should inspire quality focused discussions in professional futures research. They aim to support foresight professionals and other practitioners in planning and realizing scientific futures studies in assessing their quality, they are also thought to support training and education in the field.

The criteria of the Task Force Standards fall into three groups:

• First, standards that ensure that futures research takes into account that the usual scientific procedures of verification or falsification can hardly be applied to future issues since their specific (future) subject is not accessible and requires therefore a special approach in knowledge creation. In short: The standards of this group make sure that futures research becomes really futures research, distinct from other scientific pursuits.

• Second, there are also well-established norms and standards of good scientific research that are effectively applicable to futures studies. In short: The standards of this group ensure that futures studies become really futures research, distinct from other activities generating statements about future issues (like clairvoyance or science fiction).

• Third, standards which guarantee that futures studies serve the purposes they are launched for, that the results of the studies have the best possible practical value. In short: The standards of this group ensure the relevance and effectiveness of futures research.

As we have already discussed previously, there is a close connection between procedure and product and therefore the two sets of criteria/ standards are interrelated. They are not identical but support each other – as will be discussed in the following.

The TFS portfolio of standards suggests the following standards that result from the specific character of futures studies in distinction to other forms of research:

1. 1.1

   Principle ‘Images of the Future” reflects that statements about the future are constructions

2. 1.2

   Modality aspect of futures research: possible, probable and preferable futures are not to be mixed up

3. 1.3

   Argumentative verification: images of the future have to be open to scrutiny in a debate about their ingredients and composition

4. 1.4

   Futures research processes have to be designed and assessed in view of the interests of the actors in shaping the future

5. 1.5

   Interdisciplinarity

6. 1.6

   Transdisciplinarity

How are these standards linked to the quality criteria of the construction process of the Futures Map, the six external validity criteria of the Futures Maps and the quality criteria related to the use of the Map in decision making?

The Futures Map frame shares the constructivist epistemology of the TFS standard 1.1 Also from the point of the validation of the Futures Map, the reasonable requirement is that futures researchers should always make clear that they construct their statements based solely on present knowledge, not on any direct ‘fore-knowledge’ of future objects and that their research results do not represent the future as it will be not true for some time, but present assumptions about the future (images, constructs).

The distinction between possible, probable and preferable futures (the standard 1.2.) is in many ways taken into account in the Futures Map frame and in its validity standards. In crude terms, the external validity criteria 1 and 2 are related to the possible; 3 and 4 to the probable or plausible; and 4 and 5 to the preferable futures from the point of view of the Map’s customers.¨

For scientific futures research the crucial question is under which conditions a statement concerning the future represents knowledge. As was remarked at the beginning of this article, futures researchers can just evaluate if the actor (e.g., an expert) has good justifications for the statement.^{Footnote 19} This is subject to the standard 1.3 ‘Argumentative verification’: The assumptions underlying the statement about the future and the methodological integration of these assumptions should, in principle, be open to discourse, to examination, to approval or refusal by others.

A justified belief ^{Footnote 20}concerning futures is related to all six criteria of the external validation of the Futures Map:

• Criterion 1: The coverage of possible futures (P _i i = 1,.., n) gives a good justification for the disjunction of the possibilities (P ₁ ∨ P ₂ ∨ … ∨ P _n ).

• Criterion 2: If two futures maps FM1 and FM2 identify the same possible futures (P₁…P_n) but if FM1 is able to classify the futures according to their relevancy it is a better justification than just (P₁ ∨P₂ ∨…∨P_n)

• Criterion 3: Let us assume that FM1 takes into account better than FM2 future impacts of all kinds of causal processes that explain a set of given facts (F_i i = 1, …, k). Possible futures (P₁…P_n) with explained facts derived from the past F₁ &F₂&…& F_k & (P₁ ∨P₂ ∨…∨P_n) is surely a better justification than possible futures that are not based on development hypotheses which also explain a given range of facts.

• Criterion 4: If FM1 and FM2 identify the same possible futures (P₁…P_n) and are based on the interpretation of the same facts (F₁,…, F_n) but FM1 is able to show better than FM2 which futures provide better interpretations of given facts than others, then FM1 is a better justification. A methodological application is e.g., the regression analysis that tries to find variables that explain the past variance of the explained variables in the best way. In practice, covering interpretation of facts and efficient interpretation of facts used to be competing choices.

• Criterion 5: The justification given by this criterion is based on impacts on decision making. Futures are described in the Futures Map so that many can be used in decision making. Understandable descriptions might, however, be too superficial because they have to use metaphoric comparisons and skip complicated but relevant issues. Actors might later regret their choices.

• Criterion 6: The justification given by this criterion is based especially on choices that actors do not need to regret. Based on the Futures Map, some actors understand possible futures so well that they understand causal chains behind the possible futures and impacts of their choices. Criteria 5 and 6 might contradict each other if the better informed customers of the Futures Map can benefit from the ignorance of others.

Futures Maps are generated for actors and their decision making. The validation procedure with Futures Maps serves in this way decision making and action focus of the standard 1.4.

An important choice of scientific Futures Research is the choice between interdisciplinary and transdisciplinary research approaches (standards 1.5 and 1.6). Both aspects are important from the point of view of scientific futures research. Transdisciplinarity suggests a research strategy that crosses many disciplinary boundaries to create a holistic approach and even goes beyond that. Transdisciplinary research, as a rule, includes non-scientific experts from relevant domains of practice, who bring in their perspectives and knowledge. Interdisciplinarity accepts different specific scientific perspectives in the pursuit of a common task.

More valid Futures Maps define a distinct transdisciplinary field of futures research. In this way, futures research is proceeding towards the target suggested by Bell. According to Bell [3], futures studies should not only ‘futurise’ existing fields of research but should incorporate more principles of scientific futures research into the perspectives, theories, teaching, and research of ‘old sciences’. Bell ([3] p.59) considered that without this type of distinct transdisciplinary field, the ‘futurizing’ of other fields of science will be difficult or even impossible.

The TFS portfolio of standards suggests the following standards that make scientific futures research different from other forms of futures thinking:

1. 2.1

   Explicit definition of aims and framework conditions

2. 2.2

   Transparency and comprehensibility: third parties should be able to follow each step of reasoning

   1. 2.3

      Theoretical foundation: a sound theoretical basis for the construction of images of the future

   2. 2.4

      Appropriate choice and combination of research methods

   3. 2.5

      Conceptual quality, including procedure according to the state of the art

   4. 2.6

      Scientific relevance

   5. 2.7

      Code of conduct

Any kind of scientific work requires explicit aims and framework conditions as well as transparency and comprehensibility. The standards 2.1 and 2.2 are, however, difficult to achieve without a sound theoretical basis (standard 2.3). This standard is related to the philosophical basis that we discussed above.

We consider that the prospects of scientific futures research will depend first of all on scientific methods that organize research processes well (the internal validity) and result in more valid Futures Maps (the external validity) (the standard 2.4). In the next chapter, we briefly evaluate how different methods are suitable for different purposes and what their strengths and weaknesses in the validation of the Futures Maps are. The standards 2.5, 2.6 and 2.7 - i.e., conceptual quality, including procedure according to the state of the art; scientific relevance and code of conduct - are closely related to the methodological choices.

The TFS portfolio suggests the following standards that ensure that futures studies serve the purpose they are launched for:

1. 3.1

   Practical relevance, usability and impact

2. 3.2

   Understanding of the addressees, their types, roles and peculiarities

3. 3.3

   Transfer and communication: results should have a format suited to transfer

4. 3.4

   Identification of (general) lines of action

5. 3.5

   Project and process management

Futures Maps are customer specific. If the Futures Map takes into account customers’ interests and key customers understand it, the validity of the Futures Map increases. In this sense, futures research is first of all an applied science serving actors in their decision making. Relevant actors have to understand the validated Futures Maps (validation criterion 5). In some cases, it is required that just key customers of the study understand the map (validation criterion 6). Sometimes these key customers might even require that e.g., their competitors are not able to use the map. This does not imply incomprehensibility of results for third parties, but usually confidentiality: the resulting intellectual property belongs to the organization that commissioned the study. Confidentiality makes applied research different to academic research. As a rule it excludes validation procedures like peer reviews and the integration of the results into the knowledge corpus of the field.

The standards 3.1–3.5 are especially important for those who are practitioners in applied futures research. However, besides good project management, any futures study has to take very seriously the standards 3.2 and 3.3 into account. If panellists of a Delphi study notice that the study has practical relevance, usability and impact they are typically more ready to provide high quality arguments and judgments.

The prospects of futures research as a field of science depend first of all on the research methods. The crucial test of the usefulness of the Futures Map frame is its usefulness in the evaluation of the research methods of the futures research and in the selection of the suitable method for a specific use. In [31] we have discussed rather extensively this topic. In this article, we just compare the Futures Map frame and the classification of the methods suggested by Cuhls [10].

Cuhls [10] classified research problems of technology foresight processes using the following dimensions of the research problems:

1. 1.

   Explorative vs. selective

2. 2.

   Long-term vs. short-term

3. 3.

   Participative vs. analytic foresight approaches

4. 4.

   Focused on general themes vs. on specific themes

There are some evident links between the six criteria of the pragmatic external validity of the Futures Map and the types of research problems identified by Cuhls. In explorative studies, the criteria 1 and 3 are especially important. In selective studies the criteria 2 and 4 as well as the understanding of key customers of the study (criterion 6) are especially relevant. In the long-term studies the rich description of the possibilities (criteria 1 and 3) is the asset but also the most relevant lines of development (criterion 4). In short-term choices, the evidence-base (criterion 3 and 4) as well as the most relevant options that key customers understand (criteria 2 and 6) are especially important.

For participation focused research problems, criterion 5 is especially important together with the open discussion about various possibilities (criterion 1). The analytic approach is based on the evidence base (criterion 3 and 4). Especially the focus is on criterion 4, on most effective explanations. When the theme is general, the rich view of the theme area is especially important (criteria 1 and 3). In wicked problems, in which very different kinds of stakeholders are involved, it is especially important to find a shared conceptual frame for communication and cooperation (criterion 5). In specific problems, the relevancy of the choices from the perspective of the key customers (criteria 2 and 6) and the search for best explanations based on data (criterion 4) are especially important.

When the above remarks and Cuhls’ classification of methods are compared, this results in the hypotheses concerning validity criteria, on which the different methods especially focus. Cuhls [10] considered that scenarios and monitoring (e.g., combined with technology mining) are most suitable for explorative studies. In line with the above conceptual frame of the Futures Map, she considered that road mapping belongs to selective methods and is even at the border between foresight and planning. We can conclude that all planning horizon methods are selective, e.g., also the SWOT and the trend impact analysis.

The planning horizon methods are observed to be more short-term (e.g., trend analysis) than mapping horizon methods, first of all the scenario methods. It is, however, important to realize that in the case of very rapidly changing environments the mapping horizon might be very short e. g., just 1 year. This means that also scenarios are performed just for 1 year. Delphi and simulation models are suitable both for explorative and selective as well as both short-term and long-term studies.

Various kinds of futures conferences – especially Robert Jungk’s futures workshop concept – are participative methods. Also large panel and especially open access Delphi surveys are ways to promote participation and for many stakeholders to hear of an issue [6, 9]. Small carefully selected expert Delphi panels of e.g., the Argument Delphi [30] represent, however, rather the analytic approach as well as the trend analysis, the economic analysis (especially the regression analysis) and simulation models.

The more open methods of futures research - especially the scenario methods and the Delphi – are able to challenge the statistical or economic methods, e.g., the regression analysis in complex issues. Complex issues are often broad issues, only in rare cases one finds sector specific issues. In principle, most futures research methods are, however, suitable also for sector-specific studies though especially trend analysis is especially suitable for short-term sector-specific studies.

In this article, we have interpreted basic concepts of futures research in the Futures Map frame and suggested quality criteria of futures research in that frame. It is possible to define basic concepts of futures research beside the Futures Map frame in many other kinds of frames e.g., in the frames of various versions of the scenario approach. Anyway, we consider that the Futures Map frame is one promising possibility for the definition of the basic concepts of futures research as well as its quality criteria.

Although we observe a lot of futures research, new projects and new actors that are coming up, we still have a high degree of uncertainty about classifications, criteria what is ‘good’ and what is ‘not good’ futures research in the sense of scientific rigor or transparency. Projects are just performed without much reflection, some are even evaluated [8, 17], but the way of evaluation is rather arbitrary and the criteria are chosen according to the case or according to individual preferences. Existing ways of evaluation, which are mainly summative evaluations do not fit and do not provide the essential criteria for assessing the “quality” of futures research or foresight processes. In order to know what ‘good foresight’ or ‘good futures research’ really is, how it can be measured and which criteria have to be observed in the beginning and planning phase of a study or project, we still need more discussion, agreement and definition work. This paper is an attempt to stimulate this discussion and make proposals for quality criteria for Futures Maps and criteria for the choice of a certain methodology in a certain case.

We need this discussion especially for those who organize processes in the futures fields – but also for education. We meanwhile have some lectures and chairs in futures research, especially in Germany and in Finland, Denmark, Italy but also in Russia or the U.S.A. and for them, a little more clarity in wording, criteria and methodological rigor would do the discipline an immense favor. We hope to contribute to this debate – otherwise, soon, the confusion will be perfect, quality of futures research will decrease and the – still fragile – field might disappear again. This would be a pity, because:

We need futures thinking,

we need it now,

we need it with a long-term view and

we need insistence and continuity.

1. 1.

   The peer referred article was published in Futura 1/2015. Futura is the scientific journal of the Finnish Society for Futures Studies that publishes mostly in Finnish language.

2. 2.

   Comp. de Jouvenel’s concept of conjecture as ‘justified speculation about the future” [13].

3. 3.

   Comp. Grünwald’s two aspects of a scientific approach to futures studies: validity in terms of ‘ingredients” and ‘composition” [24].

4. 4.

   As Steinmüller [37] suggests, a possibility is, however, to connect the plausibility concept to the subjective probability including its acceptability to the subject. Concerning future-oriented causal processes in which people are not involved, the probability concept is usually more suitable than the plausibility. Amara [2] has discussed those processes in the frame of his reproducibility criterion and Kuusi [30] as not-learning systems.

5. 5.

   The concept of futures map is not to be confused with general bibliometric or science maps.

6. 6.

   In the terminology of [13] and [34] “futurible”.

7. 7.

   According to von Reibnitz, Schwab and Cerutti [36, 39], the scenario approach involves developing future environment situations (scenarios) and describing the path from any given present to these future situations. Though we prefer to call just the path the scenario, one may distinguish diachronic (or developmental) scenarios (often also called roadmap, but we use this term only linked to planning) from synchronic (or static) scenarios that coincide with our concept of the picture of the future.

8. 8.

   Compare http://www.socialresearchmethods.net/kb/introval.php

9. 9.

   For example in psychology, validity encompasses the entire experimental concept and establishes whether the results obtained meet all of the requirements of the scientific practice e.g., there must have been randomization of the sample groups (https://explorable.com/validity-and-reliability)

10. 10.

    According to Charles Peirce, new ideas cannot be originated by deduction or induction but only by abduction; “[a] bduction furnishes all our ideas concerning real things, beyond what are given in perception” (www.commens.org/dictionary/sources/quotes/644).

11. 11.

    The first version of these criteria was suggested by Osmo Kuusi in a research seminar of the Finnish Futures Academy, Helsinki 11.10.2011. The name of his presentation was ‘The Identity of the Futures Research among Sciences and Its Empirical Implications”

12. 12.

    In the General Theory of Consistence (GTC) introduced in [30], Kuusi connects the poor-informed relevance evaluations to the ‘perceived interests’ of actors and the well-informed relevance evaluations to the ‘genuine interests’ of the actors.

13. 13.

    In the introduction of the Futures Research Methodology 3.0 of the Millennium Project Jerome Glenn argued that ‘futures research is not a science because it does not have controlled experiments like physics and chemistry and because two groups with different values, experience and knowledge using the same methods to explore the future of the same subject will produce different results”. We think that our approach makes it possible to avoid this conclusion.

14. 14.

    http://www.millennium-project.org/millennium/GFIS.html

15. 15.

    Metaphorically, a futures study focused on criterion 1 is like a panorama photograph in which a picture of the future offers a particular perspective.

16. 16.

    One has to notice that criterion 4 is in conflict with criterion 1 that suggests a preference to a futures map with a higher number of scenarios. Conflicts between criteria are not uncommon and sometimes helpful.

17. 17.

    Bell [3] p. 75–97 suggested nine major tasks of futures studies. The scientific approach is visible at least in the first four tasks.

18. 18.

    The Netzwerk Zukunftsforschung (NZF) is the association of futures studies professionals of German speaking countries. The starting point for establishing this portfolio was the wide-spread perception within the NZF that there is a lack of commonly accepted standards and quality criteria for futures research. In 2010 a group of NZF members united to initiate a discourse about ways to improve that situation. Seven professionals from academia and practice formed what they called”Task Force Standards” (TFS) and established a list of basic requirements futures studies should comply with. Ten more futures studies experts joined in to write a manual of standards, modeled on similar handbooks in social sciences. For an outline of the history and a short exposition of the standards see [19].

19. 19.

    Compare Bertrand de Jouvenel’s basic concept of ‘conjecture”: A conjecture is a justified hypothesis about the future.

20. 20.

    When expert judgments are the source of knowledge concerning futures it is important to certify that the expert really believes his or her judgments. This is a highly relevant issue e.g., related to Delphi processes. Sometimes an expert is not ready to provide her or his true belief. According to [30], the expert’s information policy defines how she or he informs others about his or her beliefs.

Affiliations

1. Aalto University, Helsinki, Finland

   • Osmo Kuusi

2. University of Turku, Turku, Finland

   • Osmo Kuusi

3. Fraunhofer Institute for Systems and Innovation Research ISI, Mannheim, Germany

   • Kerstin Cuhls

4. Z_punkt GmbH The Foresight Company, Köln, Germany

   • Karlheinz Steinmüller

Corresponding author

Correspondence to Osmo Kuusi.

Reprints and Permissions