Developing a framework for responsible innovation. Research

The governance of emerging science and innovation is a major challenge for contemporary democracies. In this paper we present a framework for understanding and supporting efforts aimed at ‘responsible innovation’. The framework was developed in part through work with one of the ﬁrst major research projects in the controversial area of geoengineering, funded by the UK Research Councils. We describe this case study, and how this became a location to articulate and explore four integrated dimensions of responsible innovation: anticipation, reﬂexivity, inclusion and responsiveness. Although the framework for responsible innovation was designed for use by the UK Research Councils and the scientiﬁc communities they support, we argue that it has more general application and relevance.


Responsibility, science and innovation
Responsible innovation is an idea that is both old and new. Responsibility has always been an important theme of research and innovation practice, although how it has been framed has varied with time and place. Francis Bacon's imperative to support science 'for the relief of man's estate', the institutionalisation and professionalisation of science from the 17th century onwards, Vannevar Bush's (1945) 'Endless Frontier', JD Bernal's (1939) arguments for science in the service of society and Michael Polanyi's (1962) 'Republic of Science' counter-argument have all contained particular notions of responsibility.
Science has been conventionally invoked by policy as emancipatory. This has allowed scientists and innovators considerable freedom from political accountability. From this perspective, the role responsibilities of scientists -to produce reliable knowledge -and their wider moral responsibilities to society are imagined to be conflicted. The perceived high value of knowledge to society means that such role responsibilities typically trump any wider social or moral obligations (Douglas, 2003). Although frequent objections from university scientists suggest a permanent assault on their autonomy, much of the constitution of Polanyi's (1962) self-governing 'Republic of Science' survives to this day.
In the second half of the 20th century, as science and innovation have become increasingly intertwined and formalised within research policy (Kearnes and Wienroth, 2011), and as the power of technology to produce both benefit and harm has become clearer, debates concerning responsibility have broadened (Jonas, 1984;Collingridge, 1980;Beck, 1992;Groves, 2006). We have seen recognition and negotiation of the responsibilities of scientists beyond those associated with their professional roles (e.g. Douglas, 2003;Mitcham, 2003). We have seen scientists' own ideas of 'research integrity' change in response to societal concerns (Mitcham, 2003;Steneck, 2006). In the 1970s, biologists in the nascent field of recombinant DNA research sought to 'take responsibility' for the possible hazards their research might unleash, with a meeting at Asilomar in 1975 and a subsequent moratorium. 2 Concerns about the 'dual use' of emerging technologies and the limits of selfregulation, visible in physicists' agonising about nuclear fission prior to the Manhattan project (Weart, 1976), resurfaced in 2012 with the recent controversy over the publishing of potentially dangerous research on flu viruses (Kaiser and Moreno, 2012). The negotiation of responsibility between practicing scientists, innovators and the outside world remains an important and contested area of debate to this day.
Research in Science and Technology Studies (STS) suggests that conceptions of responsibility should build on the understanding that science and technology are not only technically but also socially and politically constituted (e.g. Winner, 1977). Latour (2008) suggests that science does not straightforwardly reveal reality through techniques of simplification and purification aimed at further mastery. As Callon et al. (2009) point out, science and technology can, paradoxically, add to our sense of uncertainty and ignorance. They tend to produce a "continuous movement toward a greater and greater level of attachments of things and people at an ever expanding scale and at an ever increasing degree of intimacy" (Latour, 2008, p. 4, italics in original). These observations suggest that unforeseen impacts -potentially harmful, potentially transformative -will be not just possible but probable (Hacking, 1986).
Responsibility in governance has historically been concerned with the 'products' of science and innovation, particularly impacts that are later found to be unacceptable or harmful to society or the environment. Recognition of the limitations of governance by market choice has led to the progressive introduction of post hoc, and often risk-based regulation. This has created a well-established division of labour that reflects a consequentialist framing of responsibility, as accountability or liability (Pellizzoni, 2004;Grinbaum and Groves, 2013). With innovation, the past and present however do not provide a reasonable guide to the future (Adam and Groves, 2011), so such retrospective accounts of responsibility are inherently limited. We face a dilemma of control (Collingridge, 1980), in that we lack the evidence on which to govern technologies before pathologies of path dependency (David, 2001), technological lockin (Arthur, 1989), 'entrenchment' (Collingridge, 1980) and closure (Stirling, 2007) set in . We have (pre-)cautionary tales of risks whose effects did not materialise for many years, where potential threats were foreseen but ignored or where only certain risks were considered relevant (Hoffmann-Riem and Wynne, 2002;EEA, 2001EEA, , 2013. Governance processes, often premised on formal risk assessment, have done little to identify in advance many of the most profound impacts that we have experienced through innovation, with the 2008 financial crisis being the most disruptive recent example (Muniesa and Lenglet, 2013). Bioethics, another major governance response, has drawn criticism for privileging individual ethical values such as autonomy over those such as solidarity that might lead to a genuine 'public ethics' (Nuffield Council on Bioethics, 2012; also Prainsack and Buyx, 2012) and, in its consequentialist version, serving to bolster the narrow instrumental expectations of innovators in some areas (Hedgecoe, 2010). Callon et al. (2009) use the metaphor of science and technology 'overflowing' the boundaries of existing scientific regulatory institutional frameworks. They point to the need for new 'hybrid forums' that will help our democracies to be "enriched, expanded, extended and. . . more able to absorb the debates and controversies surrounding science and technology" (Callon et al., 2009, p. 9). Such controversies have demonstrated that public concerns cannot be reduced to questions of risk, but rather encompass a range of concerns relating to the purposes and motivations of research (Grove-White et al., 2000;Wynne, 2002;Grove-White et al., 1997;Macnaghten and Szerszynski, 2013;Stilgoe, 2011), joining a stream of policy debate about the directions of innovation (Smith et al., 2005;Stirling, 2008;Morlacchi and Martin, 2009;Fisher et al., 2006;Flanagan et al., 2011). Yet, despite efforts at enlarging participation (see, for example, RCEP, 1998; House of Lords, 2000; Wilsdon and Willis, 2004) current forms of regulatory governance offer little scope for broad ethical reflection on the purposes of science or innovation.

A new scientific governance?
One alternative to a consequentialist model of responsibility has been to succumb to moral luck (Williams, 1981), to hope that an appeal to unpredictability and an inability to 'reasonably foresee' will allow us to escape moral accountability for our actions. Dissatisfaction with both this approach and risk-based regulation has moved attention away from accountability, liability and evidence towards those future-oriented dimensions of responsibility -care and responsiveness -that offer greater potential to accommodate uncertainty and allow reflection on purposes and values (Jonas, 1984;Richardson, 1999;Pellizzoni, 2004;Groves, 2006;Adam and Groves, 2011).
Emerging technologies typically fall into what Hajer (2003) calls an 'institutional void'. There are few agreed structures or rules that govern them. They are therefore emblematic of the move from old models of governing to more decentralised and open-ended governance, which takes place in new places -markets, networks and partnerships as well as conventional policy and politics (Hajer and Wagenaar, 2003). A number of multi-level, non-regulatory forms of science and innovation governance have taken this forward-looking view of responsibility, building on insights from STS that highlight the social and political choices that stabilise particular innovations (Williams and Edge, 1996;Pinch and Bijker, 1984;Winner, 1986). New models of anticipatory governance (Barben et al., 2008;Karinen and Guston, 2010) Constructive, Real-Time and other forms of technology assessment (Rip et al., 1995;Guston and Sarewitz, 2002;Grin and Grunwald, 2000), upstream engagement (Wynne, 2002;Wilsdon and Willis, 2004), value-sensitive design (Friedman, 1996;van den Hoven et al., 2012) and socio-technical integration (Fisher et al., 2006;Schuurbiers, 2011) have emerged. These have been complemented by policy instruments such as normative codes of conduct (see, for example, European Commission, 2008), standards, certifications and accreditations, running alongside expert reports, technology assessments and strategic roadmaps. Such initiatives have, to varying degrees, attempted to introduce broader ethical reflection into the scientific and innovation process, breaking the existing moral division of labour described above. They have attempted to open up science and innovation (Stirling, 2008) to a wider range of inputs, notably through the creation of new spaces of 'public dialogue' (Irwin, 2006).
The other important aspect of a forward-looking view of responsibility in science and innovation is that it is shared (Richardson, 1999;Mitcham, 2003;Von Schomberg, 2007). The unpredictability of innovation is inherently linked to its collective nature. Following Callon's account of innovation as 'society in the making' (Callon, 1987), we can see that implications are 'systemic', coming from the interplay of the technical and the social (Hellström, 2003). This suggests that scientists, research funders, innovators and others have a collective political responsibility (Grinbaum and Groves, 2013) or co-responsibility (Mitcham, 2003). This reflects understanding that while actors may not individually be irresponsible people, it is the often complex and coupled systems of science and innovation that create what Ulrich Beck (2000) calls 'organised irresponsibility'. 3 We can point to 'second-order' (Illies and Meijers, 2009) or 'metatask' responsibilities (van den Hoven, 1998;van den Hoven et al., 2012) of ensuring that responsible choices can be made in the future, through anticipating and gaining knowledge of possible consequences and building capacity to respond to them. This reframing of responsibility and the approaches aimed at opening up scientific governance described above provide important foundations for responsible innovation. The phrase, sometimes lengthened to 'responsible research and innovation', is starting to appear in academic and policy literature (Guston, 2006;Hellström, 2003;von Schomberg, 2011avon Schomberg, , 2011bLee, 2012;Sutcliffe, 2011;Owen and Goldberg, 2010;Owen et al., 2012;Randles et al., 2012), but it is still lacking conceptual weight. Around nanotechnology and other emerging areas of science and technology, Rip (2011) identifies a move from a discourse of responsible science to one of 'responsible governance'. US nanotechnology debates have tended to use the phrase 'responsible development' (Kjølberg, 2010). But the meaning of such terms remains contested. Rather than representing a clear novel governance paradigm, we might instead see responsible innovation as a location for making sense of the move from the governance of risk to the governance of innovation itself (Felt et al., 2007). In the following sections we develop these concepts and associated literatures to articulate a framework for responsible innovation. This has been informed by a geoengineering research project in which we were involved. Finally, we offer some conclusions on how this framework might be taken forward, based in part on our experiences within this case study of technoscience-in-the-making.

Four dimensions of responsible innovation
von Schomberg (2011a) offers the following definition of Responsible Research and Innovation: "A transparent, interactive process by which societal actors and innovators become mutually responsive to each other with a view to the (ethical) acceptability, sustainability and societal desirability of the innovation process and its marketable products (in order to allow a proper embedding of scientific and technological advances in our society)." This definition is anchored to European policy processes and values. As we will discuss in the final section of this paper, our framework has similar elements but emerges from a different context. We offer a broader definition, based on the prospective notion of responsibility described above: "Responsible innovation means taking care of the future through collective stewardship of science and innovation in the present." The dimensions that make up our framework originate from a set of questions that have emerged as important within public debates about new areas of science and technology. These are questions that public groups typically ask of scientists, or would like to see scientists ask of themselves. Table 1 draws on Macnaghten and Chilvers' (forthcoming) analysis of cross-cutting public concerns across 17 UK public dialogues on science and technology and categorises these questions as to whether they relate to the products, processes or purposes of innovation. Conventional governance focuses on product questions, particularly those of technological risk, which can obscure areas of uncertainty and ignorance about both risks and benefits (Hoffmann-Riem and Wynne, 2002;Stirling, 2010). Tools of ethical governance and research integrity move into questions of process, especially when human volunteers and animals are involved in experimentation. Approaches to responsible innovation extend the governance discussion to encompass questions of uncertainty (in its multiple forms), purposes, motivations, social and political constitutions, trajectories and directions of innovation.
If we take these questions to represent aspects of societal concern and interest in research and innovation, responsible innovation can be seen as a way of embedding deliberation on these within the innovation process. The four dimensions of responsible innovation we propose (anticipation, reflexivity, inclusion and responsiveness) provide a framework for raising, discussing and responding to such questions. The dimensions are important characteristics of a more responsible vision of innovation, which can, in our experience, be heuristically helpful for governance. We will go on to describe one application of our framework at a project level, where the main actors were the project scientists, research funders, stakeholders and ourselves. However, the framework may be applicable at other levels, such as with the development of policy or thematic programmes (see Fisher and Rip, 2013). Each dimension demands particular explanation, but the lines between them are blurred. We therefore end this section by discussing the importance of integration. For each dimension, we explain the conceptual and policy background, give meaning to the term, describe some mechanisms and approaches that might articulate the dimension in practice and offer criteria and conditions for effective innovation governance.

Anticipation
The call for improved anticipation in governance comes from a variety of sources, from political and environmental concerns with the pace of social and technical change (e.g. Toffler, 1970), to scholarly (and latterly, policy) critiques of the limitations of top-down risk-based models of governance to encapsulate the social, ethical and political stakes associated with technoscientific advances (amongst others, see Wynne, 1992Wynne, , 2002RCEP, 1998;Jasanoff, 2003;Henwood and Pidgeon, 2013). The detrimental implications of new technologies are often unforeseen, and risk-based estimates of harm have commonly failed to provide early warnings of future effects (European Environment Agency, 2001Hoffmann-Riem and Wynne, 2002). Anticipation prompts researchers and organisations to ask 'what if. . .?' questions (Ravetz, 1997), to consider contingency, what is known, what is likely, what is plausible and what is possible. Anticipation involves systematic thinking aimed at increasing resilience, while revealing new opportunities for innovation and the shaping of agendas for socially-robust risk research.
The attempt to improve foresight in issues of science and innovation is a familiar theme in science and innovation policy (Martin,  How should risks and benefits be defined and measured? Are these motivations transparent and in the public interest? How might these change in the future?
Who is in control? Who will benefit? What don't we know about?
Who is taking part? What are they going to gain? What might we never know about?
Who will take responsibility if things go wrong? What are the alternatives? How do we know we are right? 2010). This is not to say there is a shortage of future-gazing. Indeed, there is a growing literature in STS concerned with scientists' and innovators' 'imaginaries' of the future ( van Lente, 1993;Brown et al., 2000;Fortun, 2001;Brown and Michael, 2003;Hedgecoe and Martin, 2003;Fujimura, 2003;Borup et al., 2006;Selin, 2007). These expectations work not just to predict but also to shape desirable futures and organise resources towards them (te Kulve and Rip, 2011). Research in genomics and nanotechnology has, for example, been shown to carry highly optimistic promises of major social and industrial transformation, suggesting a need for what Fortun (2005) calls 'an ethics of promising' to instil some form of responsibility in disentangling present hype from future reality (Brown, 2003). Any process of anticipation therefore faces a tension between prediction, which tends to reify particular futures, and participation, which seeks to open them up.
Upstream public engagement (Wilsdon and Willis, 2004) and Constructive Technology Assessment (Rip et al., 1995) are two techniques that involve anticipatory discussions of possible and desirable futures. Guston and Sarewitz's (2002) 'Real-Time Technology Assessment' is another model of what they call 'anticipatory governance' (see also Barben et al., 2008;Karinen and Guston, 2010). Anticipation is here distinguished from prediction in its explicit recognition of the complexities and uncertainties of science and society's co-evolution (Barben et al., 2008). Methods of foresight, technology assessment, horizon scanning or scenario planning can be important techniques, although used narrowly they risk exacerbating technological determinism. Scenarios (Selin, 2011;Robinson, 2009) and vision assessment (Grin and Grunwald, 2000) have been used in various settings. Some scholars (e.g. Miller and Bennett, 2008) have also suggested that socio-literary techniques drawing on science fiction may be powerful ways to democratise thinking about the future.
Much of the academic literature here makes the point that successful anticipation also requires understanding of the dynamics of promising that shape technological futures (Borup et al., 2006;Selin, 2011;van Lente and Rip, 1998). Anticipatory processes need to be well-timed so that they are early enough to be constructive but late enough to be meaningful (Rogers-Hayden and Pidgeon, 2007). The plausibility of scenarios is an important factor in their success (Selin, 2011;von Schomberg, 2011c) and we should not underestimate the work involved in building robust tools for anticipation (Robinson, 2009). We must also recognise institutional and cultural resistance to anticipation. As Guston (2012) points out, a lack of anticipation may not just be a product of reductionism and disciplinary siloes. It may, at least in part, be intentional as scientists seek to defend their autonomy (Guston, 2012).

Reflexivity
Responsibility demands reflexivity on the part of actors and institutions, but this is not straightforwardly defined. Lynch (2000) unpacks the word 'reflexivity' to reveal its multiple meanings and modes of engagement with social worlds. Social theorists (Beck, 1992;Beck et al., 1994) have argued that reflexivity is a condition of contemporary modernity. Scientists' own version of reflexivity often echoes Popper's (1963) argument that selfreferential critique is an organising principle of science (Lynch, 2000). We would argue, following Wynne (1993), that there is a demonstrated need for institutional reflexivity in governance. Reflexivity, at the level of institutional practice, means holding a mirror up to one's own activities, commitments and assumptions, being aware of the limits of knowledge and being mindful that a particular framing of an issue may not be universally held. This is second-order reflexivity (Schuurbiers, 2011) in which the value systems and theories that shape science, innovation and their governance are themselves scrutinised. Unlike the private, professional self-critique that scientists are used to, responsibility makes reflexivity a public matter (Wynne, 2011).
Mechanisms such as codes of conduct, moratoriums and the adoption of standards may build this second-order reflexivity by drawing connections between external value systems and scientific practice (Busch, 2011;von Schomberg, 2013). Recent attempts to build reflexivity have tended to focus at the laboratory level, often with the participation of social scientists or philosophers. The argument is that in the bottom-up, self-governing world of science, laboratory reflexivity becomes a vital lever for opening up alternatives through enhancing the "reflections of natural scientists on the socio-ethical context of their work" (Schuurbiers, 2011, p. 769; also see Schuurbiers and Fisher, 2009). Approaches such as 'midstream modulation' (Fisher et al., 2006;Fisher, 2007) and 'ethical technology assessment' (Swierstra et al., 2009) give familiar ethnographic STS laboratory studies an interventionist turn (see Doubleday, 2007 for a similar approach). Rosalyn Berne's (2006) account of her interviews with nanoscientists suggests a similar intention. The conversation becomes a tool for building reflexivity. Wynne (2011) concludes that, while this work has been demonstrably successful in beginning to build reflexivity at the laboratory level, such concepts and practices need to be extended to include research funders, regulators and the other institutions that comprise the patchwork of science governance (a conclusion that has also surfaced from public dialogues in areas of synthetic biology and beyond (e.g. TNS-BRMB, 2010)). These institutions have a responsibility not only to reflect on their own value systems, but also to help build the reflexive capacity within the practice of science and innovation.
Building actors' and institutions' reflexivity means rethinking prevailing conceptions about the moral division of labour within science and innovation (Swierstra and Rip, 2007). Reflexivity directly challenges assumptions of scientific amorality and agnosticism. Reflexivity asks scientists, in public, to blur the boundary between their role responsibilities and wider, moral responsibilities. It therefore demands openness and leadership within cultures of science and innovation.

Inclusion
The waning of the authority of expert, top-down policy-making has been associated with a rise in the inclusion of new voices in the governance of science and innovation as part of a search for legitimacy (Irwin, 2006;Felt et al., 2007;Hajer, 2009). Over the last two decades, particularly in Northern Europe, new deliberative forums on issues involving science and innovation have been established, moving beyond engagement with stakeholders to include members of the wider public (e.g. RCEP, 1998;Grove-White et al., 1997;Wilsdon and Willis, 2004;Stirling, 2006;Macnaghten and Chilvers, forthcoming).
These small-group processes of public dialogue, usefully described as 'mini-publics' by Goodin and Dryzek (2006), include consensus conferences, citizens' juries, deliberative mapping, deliberative polling and focus groups (see Chilvers, 2010). Often under the aegis of quasi-governmental institutions such as Sciencewise-ERC in the UK or the Danish Board of Technology, these can, according to the UK government, "enable [public] debate to take place 'upstream' in the scientific and technological process" (HM Treasury/DTI/DfES, 2004, p. 105; see also Royal Society/Royal Academy of Engineering, 2004). Additionally, we can point to the use of multi-stakeholder partnerships, forums, the inclusion of lay members on scientific advisory committees, and other hybrid mechanisms that attempt to diversify the inputs to and delivery of governance (Callon et al., 2009;Bäckstrand, 2006;Brown, 2002).
The practice of these exercises in inclusive governance and their impact on policymaking has been uneven, and has attracted substantial critique (among others, see Horlick-Jones et al., 2007;Kerr et al., 2007;Rothstein, 2007). Public engagement practitioners can be accused of following an emerging orthodoxy, with an assumed reasoning that "the technical is political, the political should be democratic and the democratic should be participatory" (Moore, 2010, p. 793). In response, STS scholarship has begun to problematise public dialogue as a public good in itself (see Chilvers, 2009). The proliferation of participatory approaches activities has led to arguments for greater clarity about the methods of participation, the purposes for which they are used and the criteria against which they might be evaluated Frewer, 2000, 2005). In addition, a growing body of critique has developed, drawing attention to, among other things: framing effects within dialogue processes which can reinforce existing relations of professional power and deficit understandings of the public (Wynne, 2006;Kerr et al., 2007), thus constituting, at times, a new "tyranny" with questionable benefits (Cooke and Kothari, 2001); the ways in which engagement processes construct particular kinds of publics that respond to contingent political imaginaries (Lezaun and Soneryd, 2007;Macnaghten and Guivant, 2011;Michael and Brown, 2005); and the diverse, occasionally competing motivations that underpin dialogue (see Fiorino, 1989;Stirling, 2008;Macnaghten and Chilvers, forthcoming).
Irwin and colleagues suggest, however, that "the (often implicit) evocation of the highest principles that engagement might ideally fulfil can make it difficult to acknowledge and pay serious attention to the varieties of engagement that are very much less than perfect but still somehow 'good"' (Irwin et al., 2013, p. 120). The importance of public dialogue in "opening up" (Stirling, 2008) framings of issues that challenge entrenched assumptions and commitments has been emphasised (Lövbrand et al., 2011). And while there has been a resistance to attempts to proceduralise public dialogue for fear that it becomes another means of closure (Wynne, 2005;Stirling, 2008) or technocracy (Rose, 1999;Lezaun and Soneryd, 2007), there have been efforts to develop criteria aimed at assessing the quality of dialogue as a learning exercise. On the latter, Callon et al. (2009, p. 160) offer three criteria: intensity -how early members of the public are consulted and how much care is given to the composition of the discussion group; openness -how diverse the group is and who is represented; and quality -the gravity and continuity of the discussion. In relation to what actually is at stake in the advance of new science and technology, Grove-White et al. (2000) argue that public dialogue needs to open up discussion of future social worlds (building on the dimension of anticipation) in ways that critically interrogate the 'social constitutions' inherent in technological options -that is, the distinctive set of social, political and ethical implications that their development would likely bring into being (see Macnaghten, 2010 for an articulation of this approach with respect to nanotechnology and Macnaghten and Szerszynski, 2013 on geoengineering).
Processes of inclusion inevitably force consideration of questions of power. Agencies commissioning such exercises, facilitators and public participants may all have different expectations of the instrumental, substantive or normative benefits of dialogue (Stirling, 2008). There should be room therefore for public and stakeholder voices to question the framing assumptions not just of particular policy issues (Grove-White et al., 1997;Jasanoff, 2003), but also of participation processes themselves (van Oudheusden, 2011). Observed bottom-up changes within innovation processes may engender greater inclusion. User-driven (von Hippel, 1976, open (Chesbrough, 2003), open source (Raymond, 1999), participatory (Buur and Matthews, 2008) and networked innovation (Powell et al., 1996) all suggest the possibility of including new voices in discussions of the ends as well as the means of innovation, although it remains to be seen whether, first, these trends are as widespread and disruptive as their proponents claim and second, whether they in reality resemble outsourcing rather than genuine forms of 'collective experimentation' (Callon et al., 2009, p. 18).
It is far from clear whether current or past attempts at public engagement, taken together, can be said to constitute a new governance paradigm. Rather, they might be regarded as a process of 'ongoing experimentation' (Lövbrand et al., 2011, p. 487), a symptom of changes in governance rather than a centrepiece, mixing old and new governance assumptions (Irwin, 2006). Such processes might therefore be considered legitimate if their ambitions are modest and if the STS scholars who advocate dialogue are willing "to put their own normative commitments through the test of deliberation" (Lövbrand et al., 2011, p. 489). Attention has also been drawn to the "institutional preconditions for deliberation" (Lövbrand et al., 2011, p. 491). Dryzek (2011) argues that deliberative processes are only part of the 'deliberative systems' that are required to confer legitimacy (see also Goodin and Dryzek, 2006).

Responsiveness
There exist a range of processes through which questions of responsible innovation can be asked (see Table 2). Some of these processes focus questioning on the three dimensions of responsible innovation above. A few approaches, such as Constructive Technology Assessment (Rip et al., 1995), Real-Time Technology Assessment (Guston and Sarewitz, 2002), midstream modulation (Fisher et al., 2006) and anticipatory governance (Barben et al., 2008), seek to interrogate multiple dimensions. However, for responsible innovation to have purchase, it must also seek to respond to such questions.
Responsible innovation requires a capacity to change shape or direction in response to stakeholder and public values and changing circumstances. The limited capacity for empowering social agency in technological choice and the modulation of innovation trajectories has been a significant criticism of the impact of public engagement (e.g. Stirling, 2008;Macnaghten and Chilvers, forthcoming). We must therefore consider how systems of innovation can be shaped so that they are as responsive as possible. Pellizzoni describes responsiveness as "an encompassing yet substantially neglected dimension of responsibility" (Pellizzoni, 2004, p. 557). Drawing an explicit link to inclusion, he suggests that responsiveness is about adjusting courses of action while recognising the insufficiency of knowledge and control (with echoes of Collingridge's aspiration of 'corrigibility' (Collingridge, 1980)). Its two aspects relate to the two meanings of the word respond -to react and to answer (Pellizzoni, 2004). Responsiveness involves responding to new knowledge as this emerges and to emerging perspectives, views and norms.
For responsible innovation to be responsive, it must be situated in a political economy of science governance that considers both products and purposes. In the UK, Europe and perhaps more broadly, we can point to growing policy interest in 'grand challenges' (Lund Declaration, 2009). von Schomberg (2013) contends that the central challenge of responsible innovation is to become more responsive to societal challenges. But such challenges are not preordained, nor are they uncontested.
There are various mechanisms that might allow innovation to respond to improved anticipation, reflexivity and inclusion. In some cases, application of the precautionary principle, a moratorium or a code of conduct may be appropriate. Existing approaches to technology assessment and foresight may be widened to engender improved responsiveness (von Schomberg, 2013). Value-sensitive design (Friedman, 1996) suggests the possibility of designing particular ethical values into technology. As we describe in the next section's case study, techniques such as stage-gating can also create new, responsive governance choices. Diversity is an important feature of productive, resilient, adaptable and therefore responsive innovation systems (Stirling, 2007). Responsible innovation should not just welcome diversity; it should nurture it. This may require active policies of, for example, niche management (Schot and Geels, 2008). It certainly demands explicit scrutiny of the tensions and governance mechanisms within processes of research funding, intellectual property regimes and technological standards, which often act to close down innovation in particular ways, and other norms, pressures and expectations that reinforce particular path dependencies and lockins. These will differ across countries, disciplines and contexts, but this 'de facto governance' (Kearnes and Rip, 2009) is likely to follow what Pellizzoni (2004) calls 'a logic of unresponsiveness' (p. 558) in which, if responsibility is considered in any depth, retrospective accountability takes precedence.
Empirical research with governance actors in the UK, scrutinising their receptivity to substantive public concerns about science governance, suggests some important mediating factors that are likely to improve institutional responsiveness. These include: a deliberative science policy culture, emphasising reflexive learning and responsiveness; an open organisational culture, emphasising innovation, creativity, interdisciplinarity, experimentation and risk taking; top-level leadership and commitment to public engagement and to taking account of the public interest; and commitments to openness and transparency (Macnaghten and Chilvers, forthcoming). Responsiveness is therefore linked to reflexive capacity.
We can see the societal embedding of technologies as requiring a process of alignment (te Kulve and Rip, 2011;Fujimura, 1987) (or 'enrollment', following Latour (1987)). Actors and interests are arranged such that they are dependent on one another, so stabilising a particular sociotechnical system. The project of responsible governance requires understanding this 'alignment work' (te Kulve and Rip, 2011). The midstream modulation approach described in Fisher et al. (2006) differentiates between three levels of decision-making: de facto, reflexive and deliberate, with the aim of iterating governance through these levels to make assumptions more explicit and decisions more deliberate. 4 Making innovation more responsive also requires attention to metagovernance -the values, norms and principles that shape or underpin policy action (Kooiman and Jentoft, 2009). STS approaches have highlighted how policy discourses shape the governance of emerging technologies (Hilgartner, 2009;Lave et al., 2010). These governance principles may be explicit, as in the case of the Bayh-Dole act in the US; implicit, as with the growing policy focus on the relevance and 'impact' of research (Hessels et al., 2009); or nascent, as with the 'grand challenges' approach (Kearnes and Wienroth, 2011;Lund Declaration, 2009). At an overarching level, the insistence, particularly in European policy, that Research and Development should be increased in order to spur economic growth, with no question of which research or what developments, provides, it has been argued, a powerful policy discourse that limits responsiveness (von Schomberg, 2013;Felt et al., 2007).

Integrating the dimensions of responsible innovation
Moving beyond the range of processes described above that seek to advance single or multiple dimensions, responsible innovation demands their integration and embedding in governance. The dimensions therefore do not float freely but must connect as an integrated whole. It is necessary to draw connections both between the dimensions and with the contexts of governance in which they sit. The dimensions may in practice be mutually reinforcing. For example, increased reflexivity may lead to greater inclusion or vice versa. But, as illustrated in the case study in Section 3, these dimensions may also be in tension with one another and may generate new conflicts. 5 Anticipation can encourage wider participation, but, as Guston (2012) argues, it may be resisted by scientists seeking to protect their autonomy, or prior commitments to particular trajectories (see also te Kulve and Rip, 2011). The surfacing and subsequent negotiation of such tensions is central to making responsible innovation responsive. For this reason, institutional commitment to a framework that integrates all four dimensions (with no a priori instrumental conditioning) becomes vital, rather than relying on piecemeal processes that highlight particular dimensions and not others.
Public dialogue, bioethics, research integrity, codes of conduct, risk management and other mechanisms may target parts of the governance of science, but they do not offer an overarching, coherent and legitimate governance approach unless we consider how they are aligned with one another. Approaches that build on Constructive Technology Assessment (Rip et al., 1995) have recognised this need for integration at multiple levels of governance. For example, the US project of Socio-Technical Integration Research (STIR) taking place around nanotechnology seeks to explore: "what counts as responsible innovation at the macro-level of public policy, the micro-level of laboratory research, and the meso-level of institutional structures and practices that connect them" (Fisher and Rip, 2013).
The integration of the dimensions described above provides a general framework, but attention to the responsiveness dimension in particular demands that such a framework be embedded in particular institutional contexts and adjusted to take account of their idiosyncrasies. In the following section, we describe the application and further development of our framework within a UK Research Council and a particular research project as a case study of this.

Responsible innovation in action: a case study of 'technoscience in the making'
While we were working with the Research Councils to develop the framework described above, we were presented with an opportunity to work alongside a particular science and engineering project. This case study allowed us to embed and deepen our thinking. In this section we first describe the particulars of the project, its broader socio-political context, and the de facto governance arrangements in place. We then describe how our emerging responsible innovation framework was applied within this context. Finally we critically reflect on this case study: on the framework's dimensions, its implementation, impact and legitimacy, including, importantly, whether it offered a means to genuinely empower social agency in technological decision-making (Stirling, 2008).
The case was the Stratospheric Particle Injection for Climate Engineering (SPICE) project, funded by three UK research councils (the Engineering and Physical Sciences Research Council, the Natural Environmental Research Council and the Science and Technology Facilities Council). The aim of this project was to investigate whether the purposeful injection of large quantities of particles into the stratosphere could mimic the cooling effects of volcanic eruptions and provide a possible means to mitigate global warming (SPICE, 2010). The SPICE project was funded to answer three broad questions: First, what quantity of which type(s) of particle would need to be injected into the atmosphere (and where), to effectively manage the climate system? Second, how might we deliver it there? Third, what are the likely impacts associated with deployment? In response to the second question, a test was proposed of a scaled down delivery system, a 1-km high hose attached to a tethered balloon. Although the testbed would not be a geoengineering test per se -the trial would spray only a small amount of water -the testbed nevertheless constituted the UK's first field trial of a technology with geoengineering potential (Macnaghten and Owen, 2011), and was as such deeply symbolic, even though this symbolism was not initially apparent to many of those involved.

Socio-political context for the case study
Geoengineering has been defined as the "deliberate large-scale manipulation of the planetary environment to counteract anthropogenic climate change" (Royal Society, 2009, p. 1). Within the space of a few years, geoengineering has become a powerful policy discourse, offering a new class of response to anthropogenic climate change, alongside mitigation and adaptation (American Meteorological Society, 2009;Royal Society, 2009;Bipartisan Policy Centre Task Force, 2011). Those geoengineering approaches classed as solar radiation management, which are intended to reduce the amount of sunlight reaching the Earth's surface, have received particular attention because initial estimates suggest that they could be both effective and relatively cheap compared to the cost of implementing greenhouse gas mitigation policies (Boyd, 2008;Caldeira and Keith, 2010;SRMGI, 2011).
Although unformed and uncertain, solar radiation management introduces a range of significant social, political and ethical questions. These include: whether international agreement and buy-in for such a planetary-wide technology is plausible; whether research into or deployment of solar radiation management geoengineering will create a moral hazard, diverting political attention away from climate mitigation efforts; whether the impacts of solar radiation management can be fully understood before deployment; whether solar radiation management can be accommodated within democratic institutions; and whether the technology would be used for other purposes, opening up the potential for new geopolitical conflicts (for various accounts of the social and ethical dimensions of social radiation management, see Corner et al., 2011;Hamilton, 2013;Ipsos-MORI, 2010;Owen, 2011;Macnaghten and Szerszynski, 2013). Such questions have informed governance initiatives aimed at the responsible conduct of geoengineering (see the 'Oxford Principles', Rayner et al., 2013) and solar radiation management research (see the 'Solar Radiation Management Governance Initiative', SRMGI, 2011).

The SPICE project: history
Following the publication of the Royal Society's geoengineering report in July 2009 and in response to a specific recommendation that UK Research Councils co-fund "a 10 year geoengineering research programme at the level of the order of £10 M per annum" (Royal Society, 2009, p. xii), the Research Councils convened a scoping workshop in October 2009 aimed at informing a programme of geoengineering research. The aim was to "fund research which will allow the UK to make informed and intelligent assessments about the development of climate geoengineering technologies" (EPSRC/NERC/LWEC, 2009, p. 1). A number of themes relating to governance, ethics, public acceptability and public engagement were discussed at the scoping workshop but these were not considered a top priority.
The subsequent March 2010 'sandpit', conducted 15-19 March 2010, was set up by the Research Councils with the aim of "bringing together researchers from numerous backgrounds and to encourage and drive innovative thinking and radical approaches to addressing research challenges in this area" (EPSRC/NERC/LWEC, (2) a commitment to two-way communication; and (3) a 'sticky questions' briefing 4. Applications and impacts described and mechanisms put in place to review these Anticipation, reflexivity Pass pending Additional work is required: (1) more information on the envisaged milestones and associated questions, that will need to be addressed before deployment of the testbed; (2) a literature review of risks, uncertainties and opportunities of solar radiation management including social and ethical dimensions 5. Mechanisms identified to understand public and stakeholder views Inclusion, reflexivity Pass pending Additional work is required: (1) stakeholder mapping exercise; (2) engagement with stakeholders and (3) ensuring that key stakeholders are aware of the testbed 2010, p. 2), although involvement of social scientists was limited.
The SPICE project was one of two projects funded from this sandpit and did not include ethics or social science competency. Aware of at least some of the wider ethical and socio-political dimensions of solar radiation management (a point stressed by a presenter at the beginning of the sandpit from an environmental Non Governmental Organisation (NGO)), the UK Research Councils were sensitive to the potential for the SPICE project to be the subject of external scrutiny, particularly given that its proposed testbed moved beyond laboratory tests or simulations and thus could be defined as a "small field trial" (see SRMGI, 2011, p. 26). Perhaps unsurprisingly, the SPICE project passed through the ethics procedures at the Universities concerned with little or no comment: the research did not involve human volunteers or animals and the research was unlikely to have a direct effect on the environment. Nevertheless, given the evident sensitivities involved, the Research Councils decided upon a 'stage-gate' review process, which we used to incorporate our own emerging ideas of responsible innovation.

Embedding the dimensions of responsible innovation within SPICE
Stage-gating is a well-established mechanism for developing new products (Cooper, 1990) by splitting R&D into discrete stages. Decision gates use certain criteria for progression through the stages. Conventionally the inputs to the decision gate have been based on technical considerations and market potential. In the case of the SPICE project, the stage-gate was constructed to include a set of responsible innovation criteria (see Table 3), based on the dimensions we have described in Section 2. The decision gate involved an independent panel evaluating the SPICE team's response to the criteria and recommending to the Research Councils whether the testbed should proceed and, if so, under what conditions. Two authors of this paper were involved in the development and implementation of this governance approach: Owen was the architect; Macnaghten was chair of the independent stage-gate panel. Other members of the stage-gate panel included a social scientist, a representative of a civil society organisation, an atmospheric scientist and an aerospace engineer.
Five criteria were developed for the stage gate. Criteria 1 and 2 were related to the issues and potential impacts directly associated with the testbed itself: that the research was conducted in ways that were assessed to be safe and compliant with existing legislation. These were not particularly related to the prospective notion of responsibility as developed in this paper and were responded to comprehensively. Criteria 3-5 were however concerned with wider issues and potential impacts, associated with how the research was framed and issues relating to future deployment.
Criterion 3 concerned framing, communication and dialogue. It asked SPICE researchers to develop a communications approach informed by dialogue with diverse stakeholders, acknowledging areas of uncertainty and ignorance. It built on dimensions of inclusion (a commitment to base communications on genuine dialogue with stakeholders) and reflexivity (a commitment for communications to demonstrate reflection on SPICE's own embedded assumptions, commitments and framings). Criterion 4 required SPICE researchers to assess future applications and impacts, broadening their visions of application and impact, drawing on the dimensions of anticipation and reflection. It requested a review of the risks and uncertainties of solar-radiation management as well as reflection on the questions (social, ethical and technical) that would have to be considered between the testbed and eventual deployment of a working full-size system. Criterion 5 incorporated the dimensions of inclusion and reflexivity, asking researchers to engage directly with stakeholders and wider publics and to reflect upon their own tacit understandings, assumptions, uncertainties, framings and commitments.
These criteria are described in Table 3, with reference to the dimensions of responsible innovation, along with the panel recommendations and Research Council responses.
The stage-gate panel reviewed responses by the SPICE team to the five criteria in June 2011. Criteria 1 and 2 were passed, but more work was requested to meet Criteria 3-5.
The stage-gate itself was a process of responsiveness. While the panel assessment was independent, the criteria were discussed in advance of the stage gate between EPSRC officials and the SPICE team and some support provided to enable them to identify what inputs they should consider in order to respond (for example, on public engagement, see Pidgeon et al., 2013). EPSRC created the institutional conditions for this new governance mechanism and were willing, with leadership from senior staff, to interrogate their own institutional responsibilities. There was a visible degree of the institutional reflexivity demanded by Wynne (1993). Nevertheless, the stage-gate review had to fit within a wider governance landscape, partly de jure, partly de facto. The state-gate review was introduced after the project had been funded, with little scope for deliberation on the motivations for the research or whether the research should have been funded at all. There were conventional appeals to scientific autonomy and to the authority of the principal investigator. There were over-riding assumptions that Research Council decision-making should be science-led, in the service of national competitiveness. Outside the Research Councils, policy bodies such as the European Parliament and the United Nations Convention on Biological Diversity were urging caution on field tests of geoengineering. In addition, the 'Oxford Principles' (Rayner et al., 2013) on geoengineering research, including specific recommendations for early public participation in decision-making, had recently been endorsed by the UK House of Commons Science and Technology Committee. The stage-gate became a forum for the Research Councils' negotiation of internal and external demands, blending a substantive motivation towards greater responsibility with an instrumental imperative to protect reputations and relationships.
In September 2011, following the advice of the stage-gate panel, EPSRC postponed the testbed to allow the team to undertake the additional work requested. At the same time a vocal debate was taking place in the media. Following an earlier announcement by the SPICE team that the testbed would go ahead imminently (SPICE had decided to continue to prepare for the testbed experiment alongside the extra work requested by the stage-gate review panel), EPSRC received a letter in September 2011, copied to the then UK Secretary of State for Energy and Climate Change and signed by more than 50 NGOs, demanding that the project be cancelled. The NGOs saw the testbed as symbolic, sending the wrong signal to the international community, deflecting political and scientific attention from the need to curb greenhouse gas emissions (HOME, 2011).
There is a risk with any new governance mechanism that it gives the illusion of control. The stage-gate's ambitions were modest. The process was a useful 'hybrid forum' (Callon et al., 2009) within which to open up a complex governance discussion, surfacing tensions, framings, tacit assumptions, areas of contestation and, importantly, commitments. As such, it prompted a discussion of some particular issues that would turn out to be vital, even if they were not made explicit in the criteria, and were not predicted at the start of the process.
While responding to Criterion 3, by drawing up a 'sticky questions' briefing, the SPICE project leader was made aware of the existence of a prior patent application (Davidson et al., 2011) on the concept of a tethered balloon stratospheric particle delivery mechanism. This had been submitted by one of the mentors at the 'sandpit' prior to this meeting. The patent application included one of the SPICE project investigators as a co-author. Although there was no evidence that Research Council rules such as those on vested interests had been broken, given the sensitivities of the project, an independent external review was commissioned by EPSRC to investigate the sandpit and funding process. Later, in May 2012, after discussions between the Research Councils, the SPICE project leader and one of this paper's authors (Owen), the SPICE team decided to cancel the testbed, citing the lack of rules governing geoengineering research and the fact that the patent application represented "a potentially significant conflict of interest" (Cressey, 2012, p. 429).

Reflections on the SPICE project
In the case of the SPICE project, the responsible innovation approach introduced reflection, anticipation, inclusive deliberation and responsiveness, materially influencing the direction of a contentious, charged and highly uncertain area of emerging technoscience. Specifically, the framework helped the research scientists, the stage-gate panel and Research Council officials to anticipate previously unexplored impacts, applications and issues. Those involved were asked to reflect on SPICE's embedded commitments, assumptions, promissory statements, uncertainties and areas of ignorance. As the project developed, there was evidence of a more reflexive and deliberative research culture within SPICE and the Research Councils, not least through on-going dialogue on the project from the EPSRC's advisory Societal Issues Panel (of which Owen and Macnaghten were members), set up to help EPSRC Council to take account of public opinion. The SPICE principle investigator's blog -'The reluctant geoengineer' (Watson, 2011) -reveals an emerging appreciation of the social and ethical dilemmas associated with the project and a growing reflexivity in relation to his own responsibilities. In terms of inclusion, the SPICE stage gate was informed, first, by a public dialogue exercise, with results that suggested at best a highly qualified public support for the project Pidgeon et al., 2013) and, second, by a programme of stakeholder engagement (Stilgoe et al., submitted for publication). Further reflection and deliberation between the SPICE team, Research Councils and others were important in the SPICE team's decision not to proceed with the testbed, but it is important to note that this decision was one made by the SPICE team itself.
The proposed SPICE testbed originally attracted the attention of NGOs because of its potential to set a precedent for governance. The subsequent debate and change of direction -what some have called 'the SPICE experience' (Nature, 2012; Olson, 2012) -have had a discernible impact on geoengineering research and governance discussions for the same reason. Although it is unclear how these discussions will continue, the precedent set by the SPICE project and its funders in at least acknowledging wider complexities and responsibilities looks set to remain a talking point.
This case study of responsible innovation in action has also highlighted some important limitations and constraints. It became apparent that the framework should have been in place earlier, before the project's conception, and articulated more clearly. The responsible innovation framework had been separately funded and then embedded into the SPICE project once the latter was underway. The framework had no influence on the constitution of the project within the sandpit, the framing of the sandpit itself or the scoping workshop that informed this. It was therefore open to instrumental conditioning. Nevertheless, it opened up the SPICE project and its wider socio-political context to broader reflection and deliberation, providing a hybrid forum to support decisions by the funders and scientists. It also served an important function as a location for a wider ethical discussion concerning solar radiation management research through a tangible example. The case highlights the potential for a framework to inform decision-making in a field with limited governance, even if this was restricted by the nature and timing of its intervention.

Discussion
The framework for responsible innovation that we have described starts with a prospective model of responsibility, works through four dimensions, couples anticipation, reflection and deliberation to agency and action and makes explicit the need to connect with cultures and practices of governance. For this reason, the case study that we set out above was an important site for the framework's development. In using the framework, actors and institutions were challenged to go beyond compliance with established regulation, in ways that challenged conventional role and institutional responsibilities.
The eventual outcome of the case study -in which the testbed was postponed and subsequently cancelled -was unexpected, but this was an important feature of the framework. The outcome was a product of the reflexive process itself. The framework sought not to instrumentally legitimise any particular framing or commitment. Instead it served to guide, prompt and open up space for essential governance discussions aimed at supporting, but not dictating, decisions about the framing, direction, pace and trajectory of contentious and innovative research. Although the case study was limited in scope, the adoption of a responsible innovation approach prompted unconventional and, as it turned out, important governance discussions. We should not make assumptions about the applicability or validity of our framework across all issues or at all levels of decision-making, but we believe that our framework may at least provide a basis for discussions as policy and research enthusiasm for ideas of responsible innovation grows.
The framework we have suggested does not pretend to be an offthe-shelf quick fix for responsible governance. It joins and seeks to constructively inform an emerging debate on responsible research and innovation. Our framework draws on insights and experiences from the recent governance developments (Real-Time Technology Assessment, Constructive Technology Assessment, upstream engagement, midstream modulation etc.) we describe above. It seeks to shape a constructive engagement between questions of responsibility and innovation that interrogates the purposes of innovation alongside the more conventional preoccupation with the products of innovation. The framework allows scientists and decision makers to build on past lessons rather than reinventing responsibilities for each particular emerging technology.
Recent institutional inclinations towards responsible innovation, though under-conceptualised at present, can be seen as part of a move towards a new governance of science. Responsible innovation is seen by some as a response to a particular authority gap. It is therefore important to interrogate the legitimacy of our framework (see also Randles et al., 2012). Lövbrand et al. (2011) point to a legitimacy gap in deliberative engagement on science and technology issues. There are problems of both input legitimacy -how processes are set up and run -and output legitimacy -the efficacy of governance. They claim that: "The science and technology studies literature still offers little guidance on institutional design. . . [and is] often weary of institutional realities." (Lövbrand et al., 2011, p. 480). We would hope that our framework provides a counterexample to this assertion. In addition, we heed Lövbrand et al.'s (2011;see also Chilvers, 2012) call for self-reflection on the grounds on which legitimacy is based. 6 We could, as von Schomberg (2011a) has done in the European context, anchor responsible innovation to the pursuit of particular values: in his case the values that drive European Union policy. But in different areas of innovation, and in different cultural contexts, different values will be more or less pertinent, and they may be conflicted. In our analysis, we have therefore been reticent to explicitly define the normative ends of responsible innovation (what von Schomberg calls the 'right impacts'). Our approach, in line with the concepts of metagovernance described above, has concentrated on the means of governance such that an improved -more democratic or more legitimate -consideration of ends becomes possible, and in ways that are attentive to the distinctive social and ethical stakes that are associated with particular scientific and technological developments. In this sense, we have second-order normative commitments to democratisation, which we see as vital for the good governance of science and innovation. We support the feasibility and desirability of shaping or steering science and innovation, as opposed to letting the future take care of itself. It is not the purpose of this paper to explore the first order normative question of desirable ends, although we would argue that such a discussion is important.
Our aims are modest and incremental. We are providing neither a toolkit nor a manifesto, but rather one input into a broader discussion that is highly likely to shape research policy (particularly in the European Union) in the coming years. Responsible innovation will inevitably be a dynamic concept enacted at multiple levels (see Fisher and Rip, 2013). We have considered it at a 'meso-level', emphasising the leadership role of Research Councils in developing and implementing the framework that we describe. The legitimacy of our framework is perhaps therefore best imagined in the spirit of experimentation suggested by Lövbrand et al. (2011;see also Stilgoe, 2012). We see the suggested framework as a way to guide governance developments in order to enable social learning and empower social agency. We would suggest that the framework goes beyond previous deliberative experiments so that governance institutions and structures become part of the experimental apparatus.
Ongoing experiments (including our own) should not be taken as evidence of implementation, and the ease with which 'responsible (research and) innovation' can be inserted into policy documents should remind us of the risks of instrumentalising the phrase (see Owen et al., 2012 for more discussion). Chilvers (2012) has argued that, while there may be substantive enthusiasm for an opening up of debates around science and emerging technologies among individual governance actors, institutional and governance pressures typically close down such processes such that they are used in an instrumental way (following Fiorino's (1989) definition).
We have discussed elsewhere some features and underlying policy motivations of the evolving national and European discussions of responsible innovation in research policy (Owen et al., 2012). Reflections on the 'responsible' in responsible innovation are prompting new discussions about remit, role, division of labour and how trans-disciplinary programmes of science and innovation within, for example, the European Research Area should be configured and resourced. These discussions not only re-ignite an older debate about scientific autonomy but also offer new opportunities for creating value. The ways in which the concept of responsible innovation is being constituted should themselves be opened up to broad anticipation, reflection and inclusive deliberation, with the aim of making policy more responsive.