What counts as success in public engagement with science?

By Noah Weeth Feinstein, Rainer Bromme, Sarit Barzilai, and Ayelet Baram-Tsabari

Public engagement with science is widely perceived to be a source of many benefits to society, including benefits to the everyday decision making of its citizens, the quality of its democratic discourse, and the vitality of its scientific research institutions. Examples for such benefits would be the adherence to science based recommendations regarding nutrition or drug use, the willingness to accept scientific evidence in political debates about socio-scientific issues like climate change or the public appreciation of and interest in the work of scientists. In practice, people who promote and study public engagement with science rarely measure these outcomes directly. Instead, they rely on other indicators, e.g., participation measures, which may or may not be connected to the anticipated long-term benefits of public engagement with science. Even when there is no formal evaluation or official definition of quality, research and practice in public engagement with science are guided by implicit ideas about what constitutes good public engagement with science, and how we know when public engagement is successful. These goals and ideals of success, whether stated outright or merely implied, are important because they shape what activities are sponsored and how the public’s spontaneous engagement with science is measured and perceived.

Notably, not all of the goals converge, and observers with different ideas about what counts as good public engagement with science might draw entirely different conclusions about an event or interaction. For example, imagine two people responding to a blog post that expresses doubt about the scientific consensus on climate change. One person bluntly asserts her belief in climate change because “everyone knows that summers are getting hotter,” while the other person offers a detailed justification for her dissent from the scientific consensus, supporting her argument with scientific papers and even primary analysis of satellite data (this may seem implausible, but is drawn from the actual experience of one of the authors). If you define success in terms of the accuracy of the conclusions reached by the two participants, you might consider the first person successful. If your definition focuses on the depth of engagement with scientific literature and scientific data, you might consider the second person successful. If your definition focuses on the motivation to voluntarily engage with science-related issues and information, you could claim both were successful. However, if your definition focused on the capacity of laypeople to appropriately account for both expertise and evidence in their engagement with complex science topics, you might call both people unsuccessful.

Although there have been numerous attempts to categorize public engagement activities, we are not aware of a published account that focuses specifically on the different ways in which success in public engagement with science can be conceptualized, measured, or evaluated. In this blog post, we present a simple framework for classifying various dimensions of success, as well as related goals for public engagement activities. Our analysis is intended to be descriptive rather than normative: we are not arguing that successful public engagement should be measured in all of these different ways, merely that it might be measured thus. Readers will not find any measurable criteria for success below. On the other hand, our analysis may help them understand their own goals and criteria for success, and the ways in which their goals differ from others. In other words, clarifying the different dimensions of success can help readers reflect on the normative assumptions that underlie their current definitions of success, ultimately enabling more thoughtful research as well as more precisely targeted interventions.

Our framework consists of three dimensions :

1. Is success evaluated internally, by the people engaging with science, and/or externally, by observers or sponsors? 

Relatively little research considers whether people engaging with science consider their own efforts successful or are satisfied with the outcomes of their engagement. Yet it is their evaluation of their own success, perhaps more than anything else, which determines whether they will continue engaging with science, whether their engagement with science will deepen or come to an abrupt end, and if/how they will engage with science in the future. We expect that individuals and groups have a wide range of criteria for evaluating their own success, from the detailed and explicit to the intuitive and implicit. These criteria may overlap with the criteria adopted by scientific communities or differ from them. For this dimension of our analysis, the key question is not how people evaluate their success, but whether and to what degree their own evaluation (vs. an observer’s or researcher’s evaluation) is identified and accepted as meaningful.

2. Is success conceptualized in terms of the quality of the engagement process and/or by the outcome or result of this process?

Some research that addresses public engagement with science emphasizes the quality or integrity of the engagement process, whether that process is understood as individual (e.g. evidence-informed decision-making) or social (e.g., seeking out particular sorts of expertise and assistance). Other research emphasizes the outcome: Does a person or group of people know more about a scientific topic, or accept a particular scientific consensus? Do they adopt or avoid a particular health-related behavior? Do they better understand a science-related issue, or have a more positive attitude toward scientists and scientific research? Although each of these outcomes can certainly be interpreted as part of a larger process, they are often interpreted to be the results of engagement.

The processes and outcomes of public engagement with science are conceptualized in many different ways. For this dimension, the key question is not which process or outcome is chosen as a hallmark of success, but the degree to which the evaluation of success focuses on process and/or outcomes. People who place greater value on the process may judge a particular instance of public engagement to be good or successful regardless of the particular conclusions that participants form. People who place greater value on results may judge a particular instance of public engagement to be good or successful regardless of the reasoning or social processes that led to these results.

3. Is success evaluated in epistemic terms, related to the quality of knowledge and the processes that produce it, and/or in non-epistemic terms, related to things other than knowledge (e.g., advancing desirable behaviors, attitudes, or emotional states)?

Because science is often conceptualized in terms of advancing knowledge and knowing, public engagement with science is frequently evaluated with regard to quality of knowledge production, acquisition, and communication. Such evaluations focus on how people who engage with science attempt to know, what they learn, how they interact with scientific knowledge (whether they are contributing to it or questioning it), and how they understand the nature of scientific knowledge, as well as how the sponsors of public engagement attempt to communicate knowledge or engage people in scientific ways of knowing. These epistemic processes and outcomes differ from other non-epistemic ways one might characterize the quality of processes and outcomes of public engagement with science. In health and environmental contexts, for example, the value of knowledge is often eclipsed by actions. For example, the success of particular interventions might be measured in terms of increases in recycling or declines in drug use. Finally, epistemic and non-epistemic purposes can interact, both in the work of scientists and in lay engagement with science. For example, parents may seek valid knowledge about the effects of WiFi radiation in schools because they are concerned about their children’s health. Hence, public engagement with science may be evaluated based on both epistemic and non-epistemic criteria.

By characterizing this dimension as “epistemic and/or non-epistemic,” we have revealed something about our own inclinations and disciplinary biases. A behavioral theorist or health educator would probably put knowledge in the background and frame this dimension as “behavioral and/or non-behavioral.” This dimension, perhaps more than the others, could be characterized in many ways. We believe that our focus on knowledge is in keeping with the scholarly emphasis of many fields that study public engagement with science (as well as many of the agencies that promote it); an argument could be made that advancing knowledge has special importance in public engagement with science, compared to other spheres of public engagement (e.g., public engagement with art). That argument is beyond the scope of this blog post, however.

At its most simplistic level, this framework can be thought of as a 2x2x2 matrix that may help sort out the explicit as well as the implicit aims for public engagement with science. It could be applied to evaluate programs, plan new public engagement activities, or study the aims and goals of actors engaged in these types of activities. The three dimensions of the framework might also be useful for identifying aspects of success that have been ignored or overlooked in studies of people’s engagement with science. Yet the framework and its dimensions should be understood as analytic simplification with characteristic weaknesses. Other dimensions, or other ways of conceptualizing these dimensions, are easily imagined. There is vast heterogeneity within each dimension. Perhaps most importantly, we have outlined each dimension in terms of a contrast – between processes and outcomes, for example – but in no case are the two “ends” of a dimension mutually exclusive. Indeed, we believe that the richest and most fruitful evaluations of success in public engagement with science will mingle process and outcome, internal and external judgments, epistemic and non-epistemic outcomes.

Nevertheless, we offer this framework as a useful starting point, a tool that we believe will be valuable for revealing unspoken/ hidden aims of public engagement with science, for researching its live enactments, and for critiquing programs and policy documents. We hope that others will test, refine, dispute, and improve it further in the years ahead.

---

This blog post is a product of PESO2017 – Public Engagement with Science Online, an interdisciplinary, international research workshop of the Israel Science Foundation, exploring interactions between sciences, publics and social media. It took place at the Technion – Israel Institute of Technology June 25th to June 28th, 2017.

Noah Weeth Feinstein is Associate Professor in the departments of Curriculum & Instruction and Community & Environmental Sociology at the University of Wisconsin- Madison. He studies how people understand and use science as they confront health and sustainability challenges in their personal, social, and political lives. His current projects focus on climate change adaptation and the capacity of science museums to enrich democratic societies by enhancing engagement with science.

Rainer Bromme is Professor for Educational Psychology at the University of Munster. His research interests include: Communication and trust between Science and the Public, the psychology of interdisciplinary work and the development of knowledge and understanding in Science and Mathematics.

Sarit Barzilai is a lecturer (assistant professor) at the Department of Learning, Instruction and Teacher Education at the University of Haifa’s Faculty of Education. Her primary research interests focus on learners’ epistemic thinking (i.e. thinking about knowledge and knowing) and digital literacy. She is particularly interested in studying learners’ epistemic thinking in digital media contexts such as learning from multiple online information sources and from digital games.

Ayelet Baram-Tsabari is an Associate Professor at the Faculty of Education in Science and Technology at the Technion – Israel Institute of Technology, and head of the Science Communication research group. Her research interests include: Bridging science education and science communication scholarship; Identifying people’s interests in science; Building on these authentic interests to teach and communicate science in more meaningful and personally relevant ways.