Basic dictionary meanings of ‘valid’ are sound, just or well-founded [40]. In logic it means that conclusions are based on established / agreed-on premises. In empirical sciences, the validity of generalizing theories or statements is, however, a matter of degree because you can never be sure that a general statement is true. Testing the theory, you can possibly falsify some of its conclusions but if no better theory is available you are justified to conclude that the theory is more valid than that of your competitors. The suitable definition of pragmatic validity in empirical sciences is the following: valid proposition, inference or conclusion is the best available approximation of the truth.Footnote 8
Using the above definition of validity, we conclude that one of the key tasks of scientific futures research is to improve the pragmatic validity of Futures Maps. This focus is especially useful for the evaluation of the quality of the scenario process. The Futures Map approach looks at the quality of the scenario process from three points of view: the (internal) validity of the construction process of the Futures Map; the (external) validity of the constructed Futures Map; and the quality of the decision making process based on the Map. As we discussed above, the validation concerns the whole Futures Map. The validation increases or decreases the plausibility or probability of its single elements e.g., the scenarios.
Any empirical science defines its own criteria of pragmatic validity.Footnote 9 A common distinction in empirical sciences is the distinction between internal and external validity. Internal validity means that the research results are obtained using sound research methods. Besides the sound use of its various methods, we assume that the internal validity in futures research requires well-organized processes. Like many scenario process versions, they should be activity-oriented and contribute to the ‘shaping of the future‘as pragmatic and organizational approaches. The EFFLA Group [14] has suggested that foresight (in the sense of ‘applied futures research in the policy context’ but including open debates, see the definition above) should always be a process that integrates strategic intelligence, sense-making activities and the link to the policy cycle (see Fig. 2).
Like the EFFLA Group, we consider that a good way to promote the internal validity of the futures research process is to answer the following practical questions in the starting phase of the process ([14] p. 2–3). Answers to these questions help when planning the internally valid futures research process that also promotes externally valid results (the high quality Futures Map, see below).
-
a)
What is the objective of the whole foresight activity? Are there hidden agendas?
-
b)
What type of activity has to be considered for what type of issues/time spans/ knowledge?
-
c)
What is the scope of foresight? What is the scope of relevant intelligence and sense-making? Is there specific strategic intelligence or are there sense-making projects to be launched? How focused or wide should their scope be?
-
d)
What is an appropriate set of/ combination of/ methods to make use of specific actors’ strategic intelligence? And how can this be organized?
-
e)
What are the intended outcomes of the different stages in the process? In general, reports are written but often the activity as such is an outcome. How are the results presented?
Campbell and Stanley [4] have presented a classic definition of the external validity in behavioral sciences:
External validity asks the question of generalizability: To what populations, settings, treatment variables and measurement variables can this effect be generalized?
In the case of futures research, the external validity means that there are sound reasons to generalize – or to make abductionFootnote 10 - from past and present facts to futures relevant conclusions. The Futures Map summarizes these generalizations or abductions. External validity therefore means that the research results, the Futures Map, is supported by facts and observations like existing trends, weak signals etc. and well-established theories concerning the relevant causal processes. The pragmatic validity of the Futures Map increases if relevant actors are able to use it. If most relevant actors do not get the map or do not understand its messages some opportunities, which are otherwise accessible and are identified in the map, might not be part of accessible possibilities. This concerns, of course, also the avoidance of bad possibilities.
For the pragmatic description of the validity of futures research results, we suggest six validity criteria that make the validity comparisons of different Futures Maps possibleFootnote 11:
-
Criterion 1
The number or the scope of possible futures that might be relevant from the point of view of the vision or acceptable futures
-
Criterion 2
The most relevant or important possible futures are identified
-
Criterion 3
All kinds of causally relevant facts are covered by the identified futures
-
Criterion 4
Causally relevant facts are effectively interpreted with as few scenarios as possible
-
Criterion 5
Many kinds of users of the Futures Map are able to understand and use it
-
Criterion 6
Key customers of the Futures Map are able to understand and benefit from the Map
An important idea of the suggested criteria is that no Futures Map can be a high quality map in all criteria. The researcher has to select those criteria of external validity in which his or her Futures Map tries to be especially good. From this follows that there is no absolute ranking order of the Futures Maps based on the criteria. You are just able to say that the Futures Map 1 (FM1) is better than FM2 in the external validity if it is better in at least one criterion and as good as FM2 in other criteria.
One key concept related to the criteria of the pragmatic external validity is the concept “relevancy”. In the Futures Map frame, the relevancy is the customer related concept. Some feature of the Futures Map is relevant if it includes information that is useful for the customer’s choices. Though the concept of “relevancy” is mentioned just in criteria 1–4, the Futures Map cannot be relevant if it is not understood by customers of the Map (the criteria 5 and 6).
What is the connection between the relevancy and the pragmatic external validity? A way to define the pragmatic validity is the criterion that the customer’s expectations based on the Map will realize. There is, however, a way to express the quality requirement of the Map so that both the validity and the relevancy aspects are present: If the content of the Map is both valid and relevant the customer is able to make choices that she or he will not regret. Because the customer might be happy also in the case of the unexpected development this criterion is more general and suitable than the realized expectations. For example, the result might be better than anticipated. Actually, [30] has suggested that the regretted or not-regretted choices can be seen as the general epistemological basis of the futures related learning and knowledge.
We can improve the external validity of the Futures Map in a way that does not depend on specified customers. Not depending on the specified customers, we can in principle test the possibility of any assumed hypothetical causal process of the Futures Map. Of course, in the case of complex processes with many kinds of interacting causal processes, this is not often possible in practice. On other hand, there is the customer specific aspect that combines the validity and the relevancy. If the Map includes irrelevant causal processes or futures it is less suitable for the not-regretted choices of the customers because of the information overflow. The difficulties in understanding and the information overflow are taken into account in criteria 5 and 6.
The simple way to classify causal processes / futures into relevant and not relevant is by asking the customers. However, this does not guarantee the fulfillment of the not-regretting requirement. The customer might have an opinion on the relevance that is based on lacking knowledge. For example, we are justified to assume that many of the people who were killed by lung cancer in the 1950s would have stopped smoking based on our recent knowledge concerning the dangers of smoking. However, as the case of smoking nicely illustrates, even very well-justified evidence does not necessarily change the perceived relevance evaluations and related choices. Even if a person is not ready to accept the valid evidence, the Futures Map that takes the evidence into account improves the Futures Map from his or her point of view Footnote 12 We can conclude that focusing on the not-regretting criterion the objective or scientific validation of the futures map is possible both from the point of view of the not customer specific and the customer specific relevancy aspect. Footnote 13
For the identity of futures studies the first criterion is especially important. According to the Futures Research Methodology 3.0 of the Millennium Project, ‘perhaps the most commonly understood reason for the use of futures methods is to help identify what you do not know, but need to know, to make more intelligent decisions.’Footnote 14 . The identification of new possible scenario paths shows relevant gaps in our knowledge and in that way the validity of the Futures Map increases whenever it is able to evoke more (relevant) possible futures. Footnote 15 The validity improves even without critical fact-based examination of the suggested new possible paths (criterion 1). However, if the map includes, ceteris paribus, the most relevant path it improves the validity of the map (criterion 2).
As Malaska and Virtanen [32] remarked, futures are just partly determined by the known facts. It is possible that just some weak signals give indications of the most relevant futures. So it is important to take into account aall causally relevant past or recent facts (criterion 3). But if you are able to construct a map of a few scenario paths that takes account of a given set of facts and observations and gives an interpretation of their effects, it is ceteris paribus better than a map that needs more scenarios for its interpretation (criterion 4).Footnote 16 A nice example of the use of criterion 4 (from another field) is linear regression analysis that interprets the variance of past evidence with a trend assuming that the trend will continue also in future.
Even if a futures researcher has identified causally relevant possible futures and has taken into account even weak future signals something more is needed for a valid Futures Map. The validity of the Futures Map requires more than the correspondence of the Futures Map and past or present facts. Assuming that users or customers of the Futures Map themselves are the best experts of choices that they will not regret (compare [30]), they have to understand its relevant messages (the criterion 5). Criterion 5 is especially relevant when the common understanding of possibilities and relevant past facts are important for common choices of actors. However, in some situations, it might be important that just key customers understand the map and those who have an interest in hindering most favorable futures of the map’s customers do not understand it (criterion 6).
There are connections between the suggested criteria of external validity and the internal validity of futures research processes. Typically, the identification of potentially important paths (criteria 1 and 2) belongs to the first phase of the futures mapping process. In Fig. 2, it is Strategic Intelligence. The collection of potentially futures-relevant facts belongs also to the strategic intelligence phase but the interpretation of their relevance in various scenarios belongs to Sense Making (criteria 3 and 4).
Criteria 5 and 6 are related to all stages but they are especially important in the phases Sense Making and Selecting Priorities. A good scenario is not just possible and consistent but it should also be believable, trustworthy and interesting. This is an especially difficult challenge concerning those possibilities that challenge recent values and beliefs of the Futures Map’s customers. In order to manage this challenge, the futures researcher has to make the customers understand their prejudices. In practice, this challenge is highly related to the use of interesting and convincing metaphors. To this end, Causal Layered Analysis is a suggested approach [25].
The validity criteria function pairwise so that criteria 1 and 2; 3 and 4; and 5 and 6 define basic dimensions in validity evaluations. These basic dimensions correspond to the “leading questions” of research as identified for typical research styles suggested by Jerome Ravetz [35]. According to Ravetz the leading question is ‘what/how’ for research, the outcome of which is a statement intended to be factual. What/how research combines substance and agency: ‘what is this made of’ or ‘how does this cause that’. These questions are relevant especially concerning the validity of criteria 3 and 4. Statistical forecasting methods - e.g., regression analysis - are useful in predictions based on causal connections between specified variables. In these kinds of anticipations, criterion 3 means that scenario paths of the Futures Map FM1 take into account more past trends of possibly relevant variables than the scenario paths of FM2.
The validity of criterion 4 means in these kinds of anticipations that FM1 causally interprets the past variance of facts with fewer trends about ‘how the future might develop’ than FM2. For example, think about weather forecasts. In the 1950s, the forecasts provided many possible scenarios for the next few days. Now we have a much more narrow scope of possible weather scenarios for the next 3 days, based on much richer past evidence and rather better models than in the 1950s. So recent weather Futures Maps are more valid based on both criterion 3 and criterion 4.
Ravetz’ second style of research is focused on ‘how/why’ questions. This style accepts concepts like ‘final course’, ‘function’ or ‘purpose’. According to Ravetz, here the point is the design that is able to perform a given function, to do its job. ‘How/why’ questions are reasonable only in the framework of actors with their interests (compare [30]). The questions of type “why” are especially relevant for the validity criteria 5 and 6. If an actor is not able to understand how his or her interests are connected to a possible future he or she is not interested in promoting that future.
Concerning the validity criteria 1 and 2, Ravetz’ third research style is the most relevant. According to Ravetz, ‘what if?’ expresses the spirit of creativity, of inventiveness, forays into an unknown that is passive and expectant. For Ravetz the role of ‘what if’ questions was first of all the management of treating ignorance related to the impacts of new technologies and environmental hazards e.g., possible hazards of new chemical plants. We can, however, generalize ‘what if’ questions to concern also positive futures that are possibly just based on weak signals. ‘What if?’ questions challenge prevailing anticipations and action routines and, actors realize new options for action based on them. The focus is actually at least as much on new questions as on the suggested answers. This idea is stressed by Michel Godet [21] who cited Woody Allen: ‘The answer is yes, but what was the question?’