By Elyse Amend and David M. Secko
About the authors
David M. Secko is an Assistant Professor in the Department of Journalism at Concordia University in Montreal (7141 Sherbrooke St. West, CJ-3.245, Montreal, Quebec, H4B 1R6). He currently runs the Concordia Science Journalism Project (www.csjp.ca) and his research covers the analysis of new models of science journalism. email@example.com.
Elyse Amend is an MA student in the Department of Journalism at Concordia University and a member of the Concordia Science Journalism Project. Her research interests include science journalism, examining how news media “translate” scientific reports and the effects science journalism has on the public’s attitudes and behaviours. firstname.lastname@example.org.
In this paper, we report on the research method used to produce a qualitative meta-synthesis of the experiences of journalists covering health and science. Journalists covering health and science are often bombarded with critique. The literature that aims to study their practices abounds with allegations of inaccuracy, sensationalism and a failure to engage publics in meaningful dialogue. Amongst this mass of research are small islands of study that aim to better understand the experiences of science journalists in the face of such critiques, yet these qualitative studies have so far been left adrift, with little effort directed at generalizing the findings. This paper explores the process of collecting, selecting and synthesizing such qualitative studies so as to remain faithful to the lives narrated in them.
Qualitative research synthesis comprises a family of methods that have been at once championed as a way to improve the ability of qualitative studies to make an impact on their field of study (Bondas & Hall, 2007) and rebuked for simply being critical literature reviews in disguise or undertaken by the novice to avoid “the time-consuming and messy business of original data collection with human participants” (Thorne, Jensen, Kearney, Noblit & Sandelowski, 2004, p. 1343). Paterson, Thorne, Canam and Jilings have further argued that such meta-study approaches “are not for the faint-hearted and are not for the beginner” (2001, p. ix) since they require combined theoretical, methodological, and contextual analysis of a body of literature. Nevertheless, interest continues to grow in approaches to research synthesis that combine systematic review of qualitative literature with the interpretative integration of findings from identified studies. This has given rise to diverse approaches, such as meta-ethnography, thematic synthesis, critical interpretive synthesis, meta-study, meta-narrative and qualitative meta-synthesis (see, for example, Noblit & Hare, 1988; Sandelowski, Docherty & Emden, 1997; Paterson et al., 2001; Sandelowski & Barroso, 2003; Thorne et al., 2004; Dixon-Woods, Agarwhal, Jones, Young, Sutton, 2005; Walsh & Downe, 2005; Bondas & Hall, 2007; Barnett-Page & Thomas, 2009). However, when we consider this evolution of divergent methods and views on their application against the above cautions, an important question emerges: How can the interested, but inexperienced, research synthesist undertake such a project and emerge faithful to the primary studies?
In this paper, we draw on a qualitative meta-synthesis project on the experiences of journalists covering health and science to enact a deeper conversation with this question. Qualitative meta-synthesis is seen as one approach to pushing forward the emergence of disciplinary knowledge in journalism studies and to developing mid-range theory from journalistic experiences in the hotly debated field of science and health communication. Such research synthesis approaches are by no means a replacement for primary qualitative research, but instead are meant to further it by the interpretative clarification of concepts and patterns in aggregated data. But exactly how such a unique endeavour is to be accomplished is a challenging question, and one that is magnified for the novice synthesist. Such factors provide a strong motivation for the reflexive production of an audit trail.
This paper describes such an audit trail from a qualitative meta-synthesis undertaken over the past 11 months. Our particular interest is in science and health journalism, which as a profession is often approached practically as opposed to informed by evidence-based or theoretical research. This is despite inherent debates about the practice of science and health journalism that have been difficult to resolve (Secko & Smith, 2010; Bubela, Nisbet, Borchelt, Brunger, Critchley, et al., 2009). Here, we focus on our methodology and a few illustrative examples, with the wider results of the qualitative meta-synthesis to be reported elsewhere (Amend & Secko, 2010). In outlining our methodology more fully than would be possible in a paper focusing on results, we work to take seriously the considerations of the above methodologists and their concerns over novice synthesists. This is our first meta-synthesis project, so it is with some humility that we enter into the discussion over qualitative meta-synthesis in this regard, but we hope this endeavour will help other novices engage with this exciting approach to knowledge development.
Focus on Method
As noted, a diversity of methods has developed since Noblit and Hare’s (1988) classic text on meta-ethnography, methods that seek to enable qualitative research synthesis (Barnett-Page & Thomas, 2009). Here we focus on an adaption of the qualitative meta-synthesis and meta-study approaches of Sandelowski and Barroso (Sandelowski & Barroso, 2003a; Sandelowski & Barroso, 2003b; Sandelowski & Barroso, 2003c; Barroso, Gollop, Sandelowsk, Meynell, Pearce, et al., 2003) and Paterson et al. (2001), respectively. However, before describing our adaptation of these approaches to journalism studies, it is important to briefly set the context for our work in both the science/health journalism and methodological literature.
Considerations of science and health journalism
There is a rich body of literature studying science and health journalism, much of which is outside the scope of our focus on methodology in this paper (for those interested, see Logan, 2001, or Weigold, 2001). It is, however, worth noting that a renewed urgency has recently emerged over the need to more fully and openly discuss the fields of science and health journalism (Bubela et al., 2009; Dentzer, 2009). In our mind, this renewed urgency can in part be read as a desire for actionable knowledge, however provisional and modest, that meaningfully speaks to long-standing debates over journalistic practice (Secko & Smith, 2010; Russell, 2006) and rapidly evolving contexts such as the online environment (Secko, 2009; Brumfiel, 2009).
Scholars, scientists and journalists have long argued for the importance of science and health journalism, particularly as one form of communication that can enable people to keep apprised of scientific advancements, assess the appropriateness of research, and make judgments related to their environment, health and wellbeing (Nelkin, 1995). Health and science journalists perform this task by turning scientific research into understandable, engaging and entertaining stories accessible to audiences that often do not have the scientific backgrounds to make sense of research in its original form. To do this, journalists often heavily rely on scientists and academic journals. They may not have degrees in science, but instead have educational backgrounds in the arts, social science and in journalism. Thus, science and health journalists may occupy specialist positions at their news organizations, but can be seen as generalists in health and science (Conrad, 1999). Indeed, even those with advanced degrees in genetics cannot be considered experts in specialties such as microfluidics or paleobiology, although they may be more familiar with the scientific enterprise.
In this context, some studies have shown science and health coverage to be accurate (Bubela & Caulfield, 2004; Caulfield, 2004). But more often the literature has alleged inaccuracy, sensationalism and a failure to engage publics in a meaningful dialogue about scientific issues (see, for example, Dentzer, 2009; Bubela et al., 2009; Racine et al., 2006; Russell, 2006; Holtzman et al., 2005; Schwartz & Woloshin, 2004; Cassels et al., 2003; Nisbet and Lewenstein, 2002; Logan, 2001; Weigold, 2001; Hartz & Chappell, 1997; Nelkin, 1995). Science and health journalists have also been critiqued for being complicit in the production of a “cycle of hype” (Caulfield, 2004), which for journalists is argued to be linked to a desire to make dry science more newsworthy, as well as driven by passionate researchers, university funding pressures and the reality of profit-driven media organizations.
Amongst the mass of research exist small islands of qualitative study that aim to better understand the experiences of science and health journalists in the face of such critiques. This type of research often relies on in-depth, open-ended interviews, and the results are typically detailed first-person reactions, opinions and narratives that provide a subjective perspective on a topic. Such qualitative studies are undertaken for numerous reasons, but can be argued (for example) as important for understanding of what counts as science and health journalism, unearthing the logic behind the seemingly ad hoc judgments of journalists, and helping researchers gain first-person narratives on the complex relationship between journalists and the scientific actors and enterprises they cover. Furthermore, as Hinnant and Len-Rios (2009, p. 85) suggest, such qualitative inquiry can help unearth journalists’ tacit knowledge or the “unexpressed, but closely adhered to, ideas of how to do their work.”
However, the qualitative studies aimed at investigating the lived experiences of health and science journalists have so far been left adrift, with little effort directed at generalizing their findings. We therefore do not have comprehensive summaries of what has been found through such methods of inquiry. Thus, we set out to produce a qualitative meta-synthesis relating to the experiences of health and science journalists. We were aware of no previous attempts to complete such a study.
Considerations of qualitative meta-synthesis
Qualitative meta-synthesis is an approach to building more generalize-able knowledge from a set of qualitative literature on a particular topic. Many of the current discussions surrounding it are rooted in the qualitative health research community and nursing science (Paterson et al., 2001; Bondas & Hall, 2007), but several authors also locate its prior origins in a desire to systematically analyze sociological knowledge from distinct perspectives (Zhao, 1991), to synthesize diverse field research in anthropology (Noblit & Hare, 1988), and as derived from the context of quantitative meta-analysis (Thorne et al., 2004). Yet the current focus on health research means that, based on our knowledge, the use of qualitative meta-synthesis has not significantly penetrated the journalism studies literature.
Nevertheless, qualitative meta-synthesis is well suited to the theory/practice tensions that exist in journalism studies. In fact, interest in developing approaches to qualitative meta-synthesis was not only to enable researchers to go beyond a set of qualitative literature and thereby aid theoretical or methodological development, but also to answer increasing calls for how qualitative studies could better inform policy and practice in a field of inquiry. For instance, Bondas and Hall (2007, p. 113) argue that while qualitative studies are well recognized in their field of nursing science, they have had little impact for evidence-based practice. We carry a similar concern for the field of journalism studies related to health and science, where despite insightful past qualitative research, there exists a lack of connecting threads between studies. Hence, we risk studying the same health and science journalism topics over and over again—essentially “reinventing the wheel” (Sandelowski et al., 1997, p. 366) with each new study—while having little impact on journalistic practice. And herein lies one assumption about this approach: that qualitative meta-synthesis can help address these critiques and advance knowledge. In the words of Thorne, the motivation is to meet the charge “not only to publicly account for what we know, but also to account for how we know it” (2004, p. 572).
To meet such a charge, qualitative meta-synthesis has been envisioned as an analytic process by which primary studies are combined, compared and contrasted to generate meaning that extends beyond any individual study. This analytic process is held in contrast to a critical literature review, a quantitative meta-analysis that is used to aggregate quantitative results, or a secondary analysis of qualitative data sets (Paterson et al., 2001). Instead, as Sandelowski and Barroso (2003a) outline, a qualitative meta-synthesis is an interpretative product that involves three basic steps: (1) a systematic search and collection of qualitative studies; (2) a focus on, and extraction of, the results of the collected studies; and (3) the use of qualitative methods to synthesize the identified results. Importantly, such synthesizing has the goal of amplifying the results of multiple studies, as opposed to data reduction. In this way, qualitative meta-synthesis is “research of research” (Paterson et al., 2001, p. 5) where the “reviewers,” to borrow this term from Sandelowski (2006), must be aware that they are synthesizing constructions of constructions (i.e. they are several steps removed from the primary data and lived experience).
Such distance from data and lived experience, and the issues of representation this raises between “lived and narrated lives” (Sandelowski, 2006), is only one of the challenges and complexities of qualitative meta-synthesis. Other examples include the following: (a) how to comprehensively locate qualitative studies on a topic (Barroso et al., 2003); (b) how to choose studies, how many studies to choose, and whether to separate studies utilizing differing qualitative methods (Estabrooks, Field & Morse, 1994; Sandelowski et al, 1997); (c) how to determine what is to be considered “data” in a primary study and how to extract this data along a common scale (Thorne et al., 2004); (d) whether to set a broad or narrow scope to a meta-synthesis and how to thereby frame the use of meta-analytic techniques to ensure insight, manageability and transferability (Sandelowski & Barroso, 2003a; Walsh & Downe, 2005); (e) how to avoid recreating the aggregative logic that a qualitative meta-synthesis aims to replace through interpretive synthesis (Reid, Sinclair & Barr, 2009); (f) whether to assess the quality of studies (Sandelowski & Barroso, 2003b); (g) how to preserve and analyze the context, methods and theoretical stances of studies and researchers (Paterson et al., 2001); and (h) questions of epistemology and whether we need to balance empirical/analytical and critical/discursive readings of studies (Sandelowski, 2006; Barnett-Page & Thomas, 2009), among other issues.
While it is outside the scope of this paper to discuss all of these challenges, we now turn to our approach to addressing some of them from a journalism studies perspective. We frame this discussion along the three basic steps to a qualitative meta-synthesis as outlined by Sandelowski and Barroso (2003a).
Systematic searching and collection of qualitative studies
Since a qualitative meta-synthesis takes primary literature as its “data” source, the identification of appropriate qualitative studies is of key concern. While it is difficult, if not impossible, to locate every single study related to the experiences of science and health journalists, we nevertheless attempted to build a comprehensive strategy that would maximize our search results. Bondas & Hall (2007) have noted the wisdom of making use of a librarian, suggesting that search specialists often find 50 per cent more references than researchers. We therefore consulted with a subject librarian in journalism and communication at Concordia University while developing the search strategy described below.
We selected five core databases that would provide us with the most relevant articles for our purposes. The core databases searched were Communication and Mass Media Complete (CMMC), Communication Abstracts, Academic Search Premier, PubMed and Web of Science. The use of bibliographic database searches was deemed appropriate due to the decision to focus only on the peer-reviewed literature. This decision was initially made for two reasons: (1) so as to provide a clearly focused and manageable data set, and (2) to make use of the “quality” screening employed by journals in the publication of qualitative studies. We are aware that the non-peer reviewed literature and, in particular, dissertations and theses (see Estabrooks et al., 1994), may form an additional set of useful documents that can be searched in future work. A third retrospective reason for not expanding beyond the peer-reviewed literature was the conceptual richness of the studies identified, which were deemed sufficient to provide insight into our research questions.
With these points in mind, the data set uncovered by our search strategy should be viewed as a representative sample of the studies relating to the experience of health and science journalists. Paterson et al. (2001) note that researchers are unlikely to move beyond the capture of such a representative sample due to the difficulty inherent to the process (cf. Barroso et al., 2003), so instead must document their diligence so that others can assess the breadth of their data retrieval.
In July 2009, we began developing our search terms, which needed to be tailored to each individual database. We began with the CMMC database and used a “building block” approach to develop our search terms for three main concepts: “journalism,” “science,” and the “qualitative filter.” The CMMC thesaurus system was used to locate specific subject terms. For the “journalism” concept, for example, subject terms such as “journalism,” “mass media,” “news,” “newspaper” and “broadcast” were used to isolate studies that focused on journalism and journalists. The same approach was taken for the “science” concept, which for the purposes of this study, included science, health/medical and environmental subjects. CMMC’s thesaurus was used to find appropriate subject terms, such as “mass media and the environment,” “science,” “health,” and “medical.” Lastly, the two concepts were combined with the “qualitative filter,” a concept made up of numerous keywords related to qualitative research. The purpose of combining the “qualitative filter” with the “journalism” and “science” concepts in the final search was to exclude quantitative studies (e.g. surveys), qualitative studies with no human beings as participants (e.g. content analysis and discourse analysis), and qualitative studies with no journalists as participants (e.g. audience focus groups).
Our final search strategy for CMMC yielded 390 results. Both the CMMC and Academic Search Premier databases are hosted by EBSCO, and we were therefore able to use the same search strategy for Academic Search Premier. This second database search yielded 733 results. See Appendix 1 for the complete CMMC and Academic Search Premier search strategy. The search strategies for the final three databases were developed in a manner similar to those used for CMMC, but with the unique indexing system of each database taken into account. These searches yielded 45 (Communication Abstracts), 218 (PubMed) and 504 (Web of Science) results. In total, 1,890 studies were identified by this method.
To further increase the comprehensiveness of our strategy, additional searches were performed on Google Scholar and in the online databases of key journals (e.g. Journalism, Journalism Studies, Science Communication, Public Understanding of Science, Health Communication, the Canadian Journal of Communication, and J&MC Quarterly) to retrieve articles that might not have been found in the databases. We also performed citation, footnote and author searches as related to identified studies (see Barroso et al., 2003, for a fuller description of these techniques). Lastly, since bibliographic databases are not fixed entities but evolve over time, we searched the above databases a second time in January, 2010.
Selecting studies: Inclusion/exclusion criteria
After the primary searches were completed, we proceeded to review the results in order to exclude articles that were irrelevant to our purpose. As one of the goals was to identify the peer-reviewed literature that explores the lived experience of science journalists, we undertook several inclusion/exclusion phases to ensure the final studies selected for this study met this objective. Inclusion criteria for this process included studies that: (a) made use of qualitative research methods, such as qualitative interviews, focus groups, ethnographic work, etc.; (b) sought to obtain the experiences of science, health and environmental journalists by including journalists as research participants; and (c) were peer-reviewed.
In turn, we excluded the following kinds of studies: (a) studies with no human participants; (b) studies not dealing with science and health; (b) studies not dealing with journalism or not involving journalists; (c) qualitative studies with no identifiable results; (d) mixed methods studies where qualitative results could not be differentiated from other results; (e) studies where the reporting on a mixture of participants (journalists and others) did not allow the separation of these different groups; (f) non-peer reviewed studies or other non-research accounts.
The first inclusion/exclusion phase involved scanning the articles’ titles and abstracts, focusing on whether or not they involved health and science journalism or journalists. Articles that had nothing to do with journalism or science were excluded. At this stage, we did not exclude articles based on whether they were qualitative studies. This first exclusion phase considerably reduced our pool of articles, reducing the CMMC results from 390 to 104, the Academic Search Premier results from 733 to 66, the Communication Abstract results from 45 to 17, the PubMed results from 218 to 17, and the Web of Science results from 504 to 43, for a total set of 247 articles.
A second exclusion phase involved scanning the full articles to see if they were valuable for the purposes of this study, excluding those that did not involve qualitative research methods. “Qualitative research” was generously defined as empirical research with human participants that used commonly identified qualitative methods within any theoretical orientation (Sandelowski & Barroso, 2003a). There is debate in the methodological literature over whether a quality appraisal should also form part of the inclusion/exclusion criteria during a meta-synthesis (Paterson et al., 2001; Reid et al., 2009). On this point, we agree with Sandelowski and Barroso (2002) that it is problematic to exclude for reasons of quality since consensus does not exist on quality criteria for qualitative research, and for this reason, such exclusions risk removing valuable data that only some would view as containing methodological mistakes. This second exclusion left us with 41 articles.
The final exclusion phase involved in-depth reads of each of the 41 articles, carried out independently by the two researchers, who recorded their reviews of the studies. The researchers then met to discuss their reviews and come to agreements on which articles to include and exclude. This left us with 21 articles included in the final data set for analysis (See Appendix 2).
There is little direction provided in existing literature on exactly how many studies should be included in such a meta-synthesis, or alternatively, how many studies is enough to produce a saturated and transferable product of synthesis. Sandelowski et al. (1997) and Paterson et al. (2001) suggested a minimum of 10 to 12 primary studies, but shied away from strict rules since each study can provide a wealth of information. We see the question of sample size as related to both the comparability of identified studies and matters of narrow versus broad research interest. Comparability relates to asking whether findings in the identified studies can be compared to allow synthesis (Bondas & Hall, 2007), which can help set sample size parameters through the removal of incompatible studies. It has been suggested that comparing research questions across studies can help decide comparability (Sandelowski et al., 1997). The second element relates to a desire for saturation or depth, in that, more studies many lead to a broader or higher level of theorizing, while a large sample can nevertheless diminish the ability to create a narrow or deep analysis of themes (Bondas & Hall, 2007). We regarded our 21 articles as providing a wealth of comparable information via common research questions that would allow a deep analysis of themes.
Data extraction from ccollected studies
Use of the Meta-study approach
To extract qualitative data from the 21 included studies, we chose to use the meta-study approach developed by Paterson et al. (2001), which extracts and analyzes data on three levels—meta-method, meta-theory and meta-data—for subsequent synthesis. We decided upon this approach for two main reasons: (1) the meta-study approach requires that the contexts and underlying assumptions behind each included study be taken into account, as well as their possible effects on the findings, and (2) the approach is comprehensive and complex in nature, thus ensuring each included article is examined in-depth and that generalizations and conclusions are drawn with caution (Thorne et al., 2004, p. 1354). The goal of meta-study is to produce deeper understanding about the phenomenon in question by using the three components of the “analytic sequence” to produce the meta-synthesis (Bondas & Hall, 2007, p. 115). The meta-method analysis investigates the methodological processes used to collect data and the research practices that are driving the field. It seeks to shed light on the impacts these methodological features may have on the research findings. Meta-theory analysis explores the theoretical frameworks and underlying assumptions in the research, linking them to the studies’ findings and conclusions. Finally, the meta-data analysis aggregates the actual findings from included studies and re-analyzes this “processed data.” This step allows the data from individual studies to be compared and makes its reinterpretation possible (Paterson et al., 2001).
Throughout the meta-study process, we remained aware of two important limitations. First, the process de-contextualizes data, removing it from the context in which it was originally created. Our access to this original context is also severely limited by the primary research report which may omit important contextual factors. Second, as a meta-synthesis of this type does not analyze original data, the synthesis is only as good as the articulation given to the data by the primary researchers. Additionally, as previously mentioned, a qualitative meta-synthesis is a construction of a construction (Paterson et al., 2001) and its interpretive nature means it cannot be thought of as the only possible synthesis that could be created from the studies.
How data was extracted
In order to ensure the preservation of the context of each study, we began the data extraction process by producing structured summaries of the studies across a common scale. This was accomplished with the use of a reading grid inspired by Paterson et al.’s (2001, p. 135-139) “primary research appraisal tool” (see Appendix 3). The reading grids recorded details for discussion, such as the studies’ research questions and approach, the participants, the major findings, as well as conclusions drawn, implications and limitations. This extraction of data was descriptive and not evaluative.
Structured summaries were used to discuss what was to be considered a “unit of data,” and to ensure that we remained faithful to the meaning and context of the data as presented in the original study. We then undertook a formal data extraction phase, recording processed data into spreadsheets along Paterson et al.’s (2001) meta-method, meta-theory and meta-data categories. For the purposes of this study, we defined “processed data” as text, from one word to full paragraphs, contained in each study on the experiences of journalists (specialist and general) covering health and science. Claims as to what was a finding by the primary researcher were subject to our interpretation.
What follows in this section is a brief illustration of the results of this process. More extensive reporting of these results will appear elsewhere (Amend & Secko, 2010).
In order to shed light on how the qualitative data in the included studies were collected, we recorded data in the meta-method category that revealed what research questions the studies sought to address, how many participants were used, what types of journalists were involved (specialist, generalists, print, broadcast, experienced, new to the field, etc.), what geographic location or locations the study was conducted in, what year the research was carried out, and what descriptions the studies’ authors used to explain their research methods. Briefly, the meta-method analysis revealed qualitative research on the lived experiences of health and science journalists to be largely collected from a North American perspective, with the majority of studies originating in Canada and the United States. The main research method driving this field of inquiry was qualitative interviews (structured, semi-structured and open-ended) with participant pool sizes mostly ranging between 10 and 50. However, most studies used a mixed-methods approach, for example by complementing surveys with qualitative interviews. This does allude to qualitative inquiry in this field being seen as a supporting method, which is undertaken to add a personal perspective to quantitative data.
Data in the meta-theory category was used to investigate the studies’ underlying research designs and structures. In this category, we recorded the major paradigms and schools of thought present in the studies, the underlying assumptions (implicit or explicit) of the researchers, and our interpretation of theories used in the studies. Here it became apparent that few studies explicitly noted a theoretical orientation. However, three implicit orientations did emerge. First, many studies were descriptive in nature, investigating the lived experiences of health and science journalists to gain information that could help improve certain practices, such as enhancing accuracy in health and science journalism or improving journalists’ relationships with their sources. Descriptive studies simply reported the contents of interviews and focus groups with the assumption that doing so could lead to improvements in practice, while often ignoring the discussion of underlying contexts that drive/impact these practices. In contrast, a number of studies fell into a second category that we termed social construction. These studies aimed at understanding the inherent subjectivity in journalistic practice and how journalists construct versions of reality. These studies paid close attention to how culture drives journalism and science/health. Thirdly, a small number of studies took on more of an advocacy role, promoting a specific change in practice or agenda (such as focusing more on the social determinants of health in journalistic articles) and aimed at understanding how best to make such changes. We deemed this category social change.
The goal of the meta-data analysis was to investigate what the research revealed about the lived experiences of health and science journalists. We were particularly interested in how the included studies were similar and tied certain phenomena together, and in whether individual research reports presented divergent interpretations of experiences and practices of health and science journalists. As many of the studies were descriptive in nature, much of the text extracted contained the reported quotes of interviews and focus groups. Extracted data also included research findings presented in the researchers’ “own language” and qualitative interpretations thereof. This extraction produced 40 pages of processed data, supporting the notion that the selected studies contained enough data richness to amplify the results of each study and to provide the first available synthesis on the topic, which we turn to in the last section.
Qualitative methods used to synthesize results
The final step in the qualitative meta-synthesis was the reconstruction and synthesis of the extracted data, a process that is not simply aggregative but interpretive. For the novice synthesist, this is by far the most daunting step. But it is an essential component if the researcher wants to move beyond a critical literature review to synthesize the findings of identified studies and work to profit from their combined power to inform and improve journalistic theory and practice. As Thorne et al. note, this process requires not only attention to interpretive matters, but also embracing the “creative nature of any attempt to build generalizations from other people’s material” (2004, p. 1362). For us, this creativity involved moving from individual studies to a “terrain” perspective, first via the use of meta-summaries that provide a thematic summary of re-occurring themes, and then via exploring taxonomies and the use of sustained comparisons (Sandelowski & Barroso, 2003a).
Use of meta-summary
Sandelowski and Barroso have argued that qualitative meta-summaries are “useful end products of systematic reviews of qualitative research because they allow the inclusion of studies with valuable findings that some researchers would argue either are weak examples of qualitative research or not qualitative research at all” (2003, p.156). However, such thematic summary can also provide an important step in the creation of a qualitative meta-synthesis by summarizing studies so that re-occurring themes can be identified. In our process, units of meta-data were freely coded and organized into themes according to the topics addressed in each unit. This process did not transform the data. Instead, it organized the meta-data into 14 distinct meta-themes, with the majority of these themes also further split into a number of sub-themes (Table 1).
Through this process, it became apparent that the included studies shared a number of major themes, such as sourcing practices and story selection processes, as well as a number of common themes, such as journalistic tools and routines, constraints, and training and education. Thus, we were able to assess the frequency of these themes in comparison to the entire data set (see Table 1). With this process, the qualitative meta-summary is “equal to the sum of its parts” (Thorne et al., 2004, p. 1358) and remains true to the original research reports. It also reveals the frequency of themes across the data set, which some have argued to be indicative of the validity (i.e., a higher frequency indicating higher validity of the finding; Barnett-Page & Thomas, 2009).
Overall, the qualitative meta-summary provides a “map” of the qualitative data contained in the included studies. This serves as an empirical foundation for further interpretive tasks, and in particular, as a clear summary from which further linkages can emerge (Thorne et al., 2004).
Table 1: Select meta-summary themes from 21 qualitative studies on science and health journalism
|Meta-theme||Frequency (N = 21)|
|Sourcing Practices||14 out of 21|
|Story Selection||13 out of 21|
|Journalistic Tools and Routines||6 out of 21|
|Journalistic Constraints||4 out of 21|
|Audience||4 out of 21|
|Journalistic Training/Education||4 out of 21|
|Autonomy||4 out of 21|
|Science Literacy||4 out of 21|
|Journalist/Journalism’s Role||3 out of 21|
|Specialist vs. Generalist||3 out of 21|
Use of taxonomies
With the meta-summary “map” in hand, we proceeded to create a taxonomy of the findings. As the purpose of this meta-synthesis was to reveal similarities and differences in existing research and inform future theory so as to improve practice, the goal of the taxonomy was to transform the findings into a conceptual form (Sandelowski & Barroso, 2003c) in order to aid in further synthesis. To do this, we clustered themes from the meta-summary that were similar or went together (LeCompte, 2000) by constantly moving between the meta-summary and an evolving taxonomy that compared themes and sub-themes. Through taxonomic analysis, as described by Spradley (1979), we grouped together themes according to their “semantic relations” (Spradley, 1979, p. 117-118), specifically “X is part of Y” (X being the themes that emerged in the meta-summary, and Y being the taxonomic names evolving out of our interpretative groupings).
Four taxonomic categories emerged through this process: Story construction elements, external factors, journalistic identity and science literacy. These categories served to group the meta-summary themes according to their similarities and differences, essentially providing interpretive categories that described the organization of the data in our set of studies. For example, meta-summary themes such as sourcing practices, story selection and journalistic tools and routines emerged as connected to each other under the category of story construction elements (Table 2). In contrast, themes such as audience, journalistic constraints and autonomy congregated as part of the external factors that may affect journalistic practice. Journalists’ training and education, perceived roles and position as specialist health and science or generalist reporter were part of the factors that determine journalistic identity. Science literacy was the only taxonomic category that contained one sole theme, that of science literacy itself.
Table 2: Taxonomic Grouping for Story Construction Elements
|Taxonomic names (Y items)||Grouped Themes (X items)|
|Story construction elements||Language|
|Tools and routines|
|Reporting conflicts of interest|
Unlike the meta-summary previously undertaken, the aim of the taxonomy was not to investigate which themes were most prevalent in the included literature, but instead to investigate the relationships between the themes as well as the principal concepts that may not be evident or expressed in the original findings (Sandelowski & Barroso, 2003a).
Synthesis of meta-method, meta-theory and meta-data
To give one example of how the taxonomy helps in drawing deeper connections between the included studies to synthesize data, we end by considering the “journalistic constraints” theme, which was present in four of the 21 studies (Table 1). Through the use of the meta-summary and taxonomy results as related to these four studies, in combination with meta-method and meta-theory results, we were able to consider the reported journalistic constraints as a whole (Sandelowski & Barroso, 2003a) to better understand how we know what we do about this topic from the qualitative literature.
The extracted data from the papers in this meta-summary theme revealed five sub-themes that health and science journalists experienced as constraints: time; lack of space; finding/reaching reliable sources; industry realities; competition and commercialization. However, the meta-method results pointed out that these constraints were drawn largely from interviews with Canadians and Australians. Furthermore, meta-theory results showed that three of the studies were descriptive in their theoretical orientation, simply reporting the content of the interviews and focus groups under the lens that documenting these constraints could lead to the improvement of practice. However, these studies did not consider the forces that drive these constraints, leaving a reader of these studies with little interpretive insight into them.
One study was oriented to a social construction lens, stressing the importance of power relationships and the power of scientific discourse in society. In part, it noted how these constraints were discussed in the interviews as things that weakened the ability of journalists to wield power to improve their craft. This distinction makes it clear how the journalistic constraints theme is related taxonomically to other themes in external factors category. Thus, combining the data from the four studies via our taxonomy provides a richer dataset that supports the theory that journalistic constraints are maintained by external factors, such as the power structures related to scientific discourse. For example, the scientific community may exert influence over journalists by controlling expert discourse, thus leading to feelings of lacking time, space and access to sources. This taxonomy also highlights that autonomy and the needs of audiences are connected to the constraints that journalists report in interviews and focus groups.
Overall, this final process made use of sustained comparisons (Sandelowski & Barroso, 2003a) that can generate hypotheses for future research or theories for further development. While novice synthesists are wise to continually reflect on their methodology and the literature throughout a meta-synthesis project, it is in this final synthesis step that additional care is needed to meet the goals of such a project. Thorne et al. (2004) have argued that it is here that the synthesist must strive to come to a point and present at set of inferences, conclusions, or generalizations that seek to advance knowledge on their chosen topic. “Of what use is a qualitative meta-synthesis in the practice disciplines if it does not advance a synthesis, however tentative and humble?” (Thorne et al., 2004, p. 1360). While an important goal, it is one that must be approached with “representational humility” (Sandelowski, 2006, p. 13), so as to avoid an excess of what Sandelowski terms both un-reflexive inquiry (findings being seen as non-problematic reflections of reality) and hyper-reflexive inquiry (in which nothing is studied but the research process). We must balance contact with empirical findings and our interpretive product.
In this paper, we have outlined an audit trail for a qualitative meta-synthesis project on the lived experiences of journalists covering science and health. In doing so, our purpose was methodological. We attempted to add to the discussion currently surrounding qualitative meta-synthesis by exploring some of the details of our method, and we plan to report more complete details of our results in a future paper. In addition, we wanted to encourage those in journalism studies to consider meta-synthesis as a way to advance the disciplinary knowledge in a young field.
It is clear that there are many challenges to overcome to ensure the rigour of such studies. But as journalism scholars interested in qualitative studies, we are struck by Paterson et al.’s observation that “although each study may be interesting, informative, and thought-provoking, the body of qualitative research from the insider perspective provides many individual pieces of a jigsaw puzzle” (2001, p. 4). We believe drawing connecting threads between the studies included in this research—the “pieces of the jigsaw puzzle”—can help illuminate underlying issues in science and health journalism that may not have been directly addressed by the existing research. Thus, the combined data obtained through this meta-synthesis provides a map of what we already know about health and science journalists’ lived experiences, as well as gaps that should be addressed in future research. Indeed, a key element of the qualitative data analyzed is that it alerts us to how much we know about story construction elements used in health and science journalism and many of the external factors that influence and affect journalists’ work, specifically print journalists. Conversely, it also alerts to how little we know about other elements. In saying this, we must return to a final awareness that this conclusion is draw from our contact with research reports, and not the lives lived by science and health journalists. Nevertheless, we continue to work to remain true to interpreting the lives they have narrated to others.
The authors would like to thank Sonia Poulin (Concordia University) for helping to develop our database search strategy and Linda Slater (University of Alberta) for developing the original qualitative filter. This work was supported by Genome Canada and Genome Quebec as part of the GE3LS component of the “Genozymes for Bioproducts and Bioprocesses Development” project.
Appendix 1: CMMC and Academic Search Premier Search Strategy
(SU journalis* OR mass media OR “reporting&reporters” OR “news” OR newspaper* OR press OR broadcast* OR blog* OR investigative reporting OR media spillover OR digital media OR photojournalis*) AND (SU mass media & the environment or SU ( scien* or health or medic* )) AND (TX ( qualitative OR ethnol* OR ethnog* OR ethnonurs* OR emic OR etic OR leininger OR noblit OR field OR fieldnote* OR participant observ* OR participant observation* OR hermaneutic* OR phenomenolog* OR lived experience* OR heidegger* OR husserl* OR merleau-pont* OR colaizzi OR giorgi OR ricoeur OR spiegelberg OR van kaam OR van manen OR grounded theory OR constant compar* OR theoretical sampl* OR (glaser AND strauss) OR content analy* OR thematic analy* OR narrative* OR unstructured categor* OR structured categor* OR interview* OR maximum variation* OR snowball OR audio* OR tape* OR video* OR metasynthes* OR meta-synthes* OR metasummar* OR meta-summar* OR metastud* OR meta-stud* OR meta-ethnograph* OR metaethnog* OR meta-narrative* OR metanarrat* OR meta-interpretation* OR metainterpret* OR qualitative meta-analy* OR qualitative metaanaly* OR qualitative metanaly* OR purposive sampl* OR action research OR focus group* )
Appendix 2: Bibliographic Sample for the Metasynthesis Project with Findings Related to the Experiences of Science and Health Journalists
- Balasegaram, Mangai, Sooria Balasegaram, Denis Malvy and Pascal Millet. “Neglected Diseases in the News: A Content Analysis of Recent International Media Coverage Focussing on Leishmaniasis and Trypanosomiasis.” PLoS Negl Trop Dis. 2.5 (2008): e234.
- Chew, Fiona, Judith Mandelbaum-Schmid and Sue Kun Gao. “Can Health Journalists Bridge the State-of-the-Science Gap in Mammography Guidelines?” Science Communication. 27.3 (2006): 331-351.
- Conrad, Peter. “Uses of Expertise: Sources, Quotes and Voices in the Reporting of Genetics in the News.” Public Understanding of Science 8 (1999): 285-302.
- Gasher, Mike, Michael Hayes, Robert Hackett, Donald Gutstein, Ian Ross and James Dunn. “Spreading the News: Social Determinants of Health Reportage in Canadian Daily Newspapers.” Canadian Journal of Communications 32 (2007): 557-574.
- Geller, Gail, Barbara A. Bernhardt, Mary Gardner, Joann Rodgers and Neil A. Holtzman. “Scientists’ and Science Writers’ Experiences Reporting Genetic Discoveries: Toward an Ethic of Trust in Science Journalism.” Genetics in Medicine 7.3 (2005): 198-205.
- Hansen, Anders. “Journalistic Practices and Science Reporting in the British Press.” Public Understanding of Science 3(1994): 111-134.
- Henderson, Lesley, and Jenny Kitzinger. “Orchestrating a Science ‘Event’: The Case of the Human Genome Project. New Genetics and Society 26.1 (2007): 65-83.
- Hijmans, Ellen, Alexander Pleijter, and Fred Wester. “Covering Scientific Research in Dutch Newspapers.” Science Communication. 25.2 (2003): 153-176.
- Hinnant, Amanda, and Marie E. Len-Rios. “Tacit Understanding of Health Literacy: Interview and Survey Research with Health Journalists.” Science Communication. 31.1 (2009): 84-115.
- Hodgetts, Darrin, Kerry Chamberlain, Margaret Scammell, Rolinda Karapu and Linda Waimarie Nikora. “Constructing Health News: Possibilities for a Civic-Oriented Journalism.” Health 12.1 (2007): 43-66.
- Larsson, Anne, Andrew D. Oxman, Cheryl Carling and Jeph Herrin. (2003). “Medical Messages in the Media—Barriers and Solutions to Improving Medical Journalism.” Health Expectations 6 (2003): 323-331.
- Maillé, Marie-Ève, Johanne Saint-Charles and Marc Lucotte. “The Gap Between Scientists and Journalists: The Case of Mercury science in Quebec’s Press.” Public Understanding of Science. 19.1 (2010): 70-79.
- Reed, Rosslyn.. “(Un-)Professional discourse?” Journalism 2.3 (2001): 279-298.
- Roy, Stephanie C., Guy Faulkner and Sara-Jane Finlay. “Fit to Print: A Natural History of Obesity Research in the Canadian News Media.” Canadian Journal of Communication 32 (2007): 575-594.
- Saari, Mary-Anne, Candace Gibson and Andrew Osler. “Endangered Species: science Writers in the Canadian Daily Press.” Public Understanding of Science 7 (1998): 61-81.
- Treise, Debbie, and Michael F. Weigold. “Advancing Science Communication: A Survey of Science Communicators.” Science Communication 23.3 (2002): 310-322.
- Trumbo, Craig W., Kim J. Spencer, Rebecca J. Dumlao, Gi Woong Yun and Shearlean Duke. “Use of E-Mail and the Web by Science Writers.” Science Communication. 22.4 (2002): 347-378.
- Van Trigt, Anke M., Lolkje T. W. Jong-van den Berg, Flora M. Haaijer-Ruskamp, Jaap Willems and T. (Dirk) F. J. Tromp. “Medical Journalists and Expert Sources on Medicines.” Public Understanding of Science 3(1994): 309-321.
- Waddel, Charlotte, Jonathan Lomas, John N. Lavis, Julia Abelson and Cody A. Shepherd. “Joining the Conversation: Newspaper Journalists’ Views on Working with Researchers.” Healthcare Policy 1.1 (2005): 123-139.
- Ward, Stephen J. A., and Eric Jandciu. “Challenges in Communicating Science to Canadians.” Media Development 3(2008): 12-16.
- Wilkinson, Clare, Stuart Allan, Alison Anderson and Alan Peters. “From Uncertainty to Risk? Scientific and News Media Portrayals of Nanoparticle Safety.” Health, Risk & Society. 9.2 (2007): 145-157.
Appendix 3: Reading Grid Used for Initial Data Extraction
|Study Number: Date grid was completed on:|
|Reference: (Author Names, Year, Title, Journal Vol. X: Page Numbers)|
|A. Question Investigated and Approach to the question
|B. Nature of the Sample used in the Study
|↑Crossover Points ↓||ã Crossoverä
å Points æ
|↑Crossover Points ↓|
|C. Major Findings
Tools and approaches:
Role of journalists:
|D. Conclusions, Implications & Limitations
|Key Quotes and other Considerations/thoughts
Amend, E., & Secko, D. (2010). In the face of critique: A qualitative metasynthesis of the experiences of journalists covering health and science. [In preparation].
Barnett-Page, E., & Thomas, J. (2009). Methods for the synthesis of qualitative research: A critical review. BMC Medical Research Methodology, 9, 59.
Bondas, T., & Hall, E. (2007). Challenges in approaching metasynthesis research. Qualitative Health Research, 17(1), 113-121.
Barroso, J., Gollop, C. J., Sandelowski, M., Meynell, J., Pearce, P. F., & Collonis, L. J. (2003). The challenges of searching for and retrieving qualitative studies. Western Journal of Nursing Research, 25, 153-178.
Brumfiel, G. (2009). Science journalism: Supplanting the old media? Nature, 458, 274-277.
Bubela, T. & Caulfield, T. (2004). Do the print media ”hype” genetic research? Canadian Medical Association Journal, 170, 1399-1407.
Bubela, T., Nisbet, M.C., Borchelt, R., Brunger, F., Critchley, C., Einsiedel, E., Geller, G., Gupta, A., Hampel, J., Hyde-Lay, R., Jandciu, E.W., Jones, S.A., Kolopack, P., Lane, S., Lougheed, T., Nerlich, B., Ogbogu, U., O’Riordan, K., Ouellette, C., Spear, M., Strauss, S., Thavaratnam, T., Willemse, L. & Caulfield, T. (2009). Science communication reconsidered. Nature Biotechnology, 27(6), 514-8.
Cassels, A., Hughes, M. A., Cole, C., Mintzes, B., Lexchin, J., & McCormack, J.P. (2003). Drugs in the news: An analysis of Canadian newspaper coverage of new prescription drugs. Canadian Medical Association Journal, 168(9), 1133-37.
Caulfield, T. (2004). Biotechnology and the popular press: Hype and the selling of science. Trends in Biotechnology, 22(7), 337-339.
Conrad, P. (1999). Uses of expertise: Sources, quotes and voices in the reporting of genetics in the news. Public Understanding of Science, 8, 285-302.
Dentzer, S. (2009). Communicating medical news – Pitfalls of health care journalism. New England Journal of Medicine, 360(1), 1-3.
Dixon-Woods, M., Agarwhal, S., Jones, D., Young, B., & Sutton, A. (2005). Synthesising qualitative and quantitative evidence: A review of possible methods. J Health Serv Res Pol, 10(1), 45-53b.
Estabrooks, C. A., Field, P. A., & Morse, J. M. (1994). Aggregating qualitative findings: An approach to theory development. Qualitative Health Research, 4, 503-511.
Hartz, J., & Chappell, R. (1997). Worlds apart: How the distance between science and journalism threatens America’s future. Nashville, TN: First Amendment Center.
Hinnant, A., & Len-Rios, M. E. (2009). Tacit understanding of health literacy: Interview and survey research with health journalists.” Science Communication, 31(1), 84-115.
Holtzman, N.A., Bernhardt, B. A., Mountcastle-Shah, E., Rodgers, J. E., Tambor, E., & Geller, G. (2005). The quality of media reports on discoveries related to human genetic diseases. Community Genetics, 8, 133-144.
LeCompte, M. D. (2000). Analyzing qualitative data.” Theory into Practice, 39(3), 146-154.
Logan, R.A. (2001). Science mass communication: Its conceptual history. Science Communication, 23(2), 135-163.
Nelkin, D. (1995). Selling science: How the press covers science and technology. New York: Freeman.
Nisbet, M.C., & Lewenstein, B.V. (2002). Biotechnology and the American media: The policy process and the elite press, 1970 to 1999. Science Communication, 23(4), 359-391.
Noblit, G.W., & Hare, R.D. (1988). Meta-Ethnography: Synthesizing qualitative studies. London: Sage Publications.
Paterson, B.L., Thorne, S. E., Canam, C., & Jillings, C. (2001). Meta-study of qualitative health research: A practical guide to meta-analysis and meta-synthesis. London: Sage Publications.
Racine, E., Gareau, I., Doucet, H., Laudy, D., Jobin, G., & Schraedley-Desmond, P. (2006). Hyped biomedical science or uncritical reporting? Press coverage of genomics (1992–2001) in Quebec. Social Science and Medicine, 62, 1278–1290.
Reid, B., Sinclair, M., & Barr, O. (2009). Reflections on the methodological challenges of undertaking a meta-synthesis: A response to Heyman. Social Science and Medicine, 69, 1577–1579.
Russell, C. (2006). Covering controversial science: Improving reporting on science and public policy. Joan Shorenstein Center on the Press, Politics and Public Policy. URL: http://www.hks.harvard.edu/presspol/publications/papers/working_papers/2006_04_russell.pdf [March 1, 2010].
Sandelowski, M. (2006). “Meta-Jeopardy”: The crisis of representation in qualitative metasynthesis. Nursing Outlook, 54, 10-16.
Sandelowski, M., & Barroso, J. (2002). Reading qualitative studies. International Journal of Qualitative Methods, 1(1), Article 5. URL: http://ejournals.library.ualberta.ca/index.php/IJQM/article/view/4615/3764 [Retrieved May 1, 2010]
Sandelowski, M., & Barroso, J. (2003a). Toward a metasynthesis of qualitative findings on motherhood in HIV-positive women. Research in Nursing and Health, 26, 153-170.
Sandelowski, M., & Barroso, J. (2003b). Classifying the findings in qualitative studies. Qualitative Health Research, 7, 905-923.
Sandelowski, M., & Barroso, J. (2003c). Creating meta-summaries of qualitative findings. Nursing Research, 52, 226-231.
Sandelowski, M., Docherty, S., & Emden, C. (1997). Qualitative metasynthesis: Issues and techniques. Research in Nursing and Health, 20, 365-371.
Schwartz, L. M., & Woloshin, S. (2004). The media matter: A call for straightforward medical reporting. Annuals of Internal Medicine, 140, 226-228.
Schwitzer, G. (2003). How the media left the evidence out in the cold. British Medical Journal, 326, 1403-04.
Secko, D. (2009). The unfinished science story: Reflections on journalist-audience interactions in the online environment. Journal of Media Practice, 10(2&3), 259–266.
Secko, D., & Smith, W. (2010). Health journalism: Fracturing concerns and building reflective capacity with a deliberative lens. Canadian Journal of Communication [in press].
Spradley, J.P. (1979). The ethnographic interview. New York: Holt, Rinehart & Winston.
Thorne, S., Jensen, L., Kearney, M.H., Noblit, G. & Sandelowski, M. (2004). Qualitative metasynthesis: Reflections on methodological orientation and ideological agenda. Qualitative Health Research, 14(10), 1342-1365.
Thorne, S. (2004). The role of qualitative research within an evidence-based context: Can metasynthesis be the answer? International Journal of Nursing Studies, 46, 569–575.
Treise, D., & Weigold, M.F. (2002). Advancing science communication: A survey of science communicators. Science Communication, 23(3): 310-322.
Walsh, D., & Downe, S. (2005). Meta-synthesis method for qualitative research: A literature review. Journal of Advanced Nursing, 50(2), 204–211.
Weigold, M.F. (2001). Communicating science. Science Communication, 23, 164-193.
Zhao, S. (1991). Metatheory, metamethod, meta-data-analysis: What, why, and how? Sociological Perspectives, 34, 377-390.