Running head: QUALITY CRITERIA UNDER PLURALISM

Quality Criteria under Methodological Pluralism:
Implications for Conducting and Evaluating Research

Chris Barker and Nancy Pistrang
University College London, London, England

Correspondence concerning this article should be addressed to Dr Chris Barker, Sub-Department of Clinical Health Psychology, University College London, Gower Street, London WC1E 6BT, England; e-mail: c.barker@ucl.ac.uk

Abstract
Many community psychologists adhere to a methodological pluralist orientation to research; however, it is often unclear what such a position means in practice. This paper draws out the practical implications of methodological pluralism for community research. It proposes 4 sets of criteria for how research might be appraised under a pluralistic ethos: criteria applicable to all research, research-relevant community psychology values and principles, criteria specifically applicable to quantitative research, and criteria specifically applicable to qualitative research. The paper also addresses how pluralistic community research may be conducted, at each of 3 levels: integrating methods within a single study, using different approaches within a research program, and pluralism in the field as a whole.

Quality Criteria under Methodological Pluralism:
Implications for Conducting and Evaluating Research
	It is well accepted that, as a discipline, community psychology has adopted a pluralist orientation to research. Several prominent community psychologists have forcefully argued the case for pluralism (for example, Kelly, 1990, 2003; Tolan, Keys, Chertok, & Jason, 1990; Tebes & Kraemer 1991; Trickett, 1996), and even a cursory examination of any of the field’s main English-language journals reveals articles using a variety of methods, approaches, and paradigms. Furthermore, one of the Society for Community Research and Action’s four principles explicitly mentions using “multiple methodologies” (SCRA website, n.d.). So it would therefore perhaps seem that further discussion of the pluralist approach in community psychology is now superfluous.
However, there are two qualifications to this statement. First, the field’s long-standing self-examination on the place of traditional scientific approaches to enquiry, which this special issue continues, is evidence that there is still heat in the debate about which way the field ought to head. Second, although almost all community psychologists would support a pluralistic position in principle, there has been little attempt in the literature to detail its implications for how community research is conducted and appraised.
	The present paper therefore seeks to develop further the implications of the methodological pluralist approach for research in community psychology. First, it briefly examines the characteristics of the pluralist approach. Second, it outlines some possible criteria for how research might be evaluated. Finally, it addresses different models of pluralism within the field. It attempts to draw together ideas from several other sources, both within and outside of community psychology. Its intended contribution is one of synthesis: to integrate methodological ideas drawn from disparate areas with the particular concerns and values of community psychology.
Some Preliminary Definitions
As Kelly (2003) observes, the place and role of  “science” within community psychology has been debated since the founding of the field, and the term itself has become loaded with much surplus meaning. It is therefore useful in the interests of precision to briefly address the usage of this term and our other key notion of “pluralism.”
Science
“Science” is a natural language term, of which no precise demarcation is possible. It may be helpful to distinguish two senses of the term as commonly used in the literature. The first, more restrictive sense emphasizes systematic observation and experimentation, and the formulation of general theories or laws of nature. Commonly associated with such laboratory-based disciplines as physics and biology, it is the image of science at the heart of the positivist program. The second, more inclusive sense is the “human science” definition: that the ambit of science can be widened to encompass many different kinds of systematic ways to advance knowledge. It is in this vein that we have political science, social science, and, indeed, community science (Wandersman, 2003). These two senses of the term are sometimes blurred. In case of ambiguity, this paper will indicate the first sense by the qualification “traditional science.”
Debates about an area’s scientific standing, or lack thereof, are of course not unique to community psychology. They have permeated the whole field of psychology (Driver-Linn, 2003) and allied disciplines, such as sociology. This debate has deep historical roots in the epistemological divisions between rationalists and empiricists. However, what gives it a unique dimension in the present context is the value-laden nature of our field. Community psychologists evaluate research not only by academic criteria, but also by a template of social, political, and moral criteria (Prilleltensky, 2001).
The debate over the role of science matters because it has implications for the research and development of the field: what knowledge can practicing community psychologists draw upon to support their activities, and what activities do researchers in community psychology carry out in order to advance the field’s knowledge base? In other words, the debate concerns the nature of evidenced-based practice, and the relevance of practice-based evidence.
Methodological Pluralism
The core idea behind methodological pluralism is that knowledge accumulates from a variety of sources in a variety of ways. Pluralism is an embracing of those possibilities: a finding of value in methodological diversity (Fiske & Shweder, 1986). The pluralist stance is that traditional scientific approaches (usually quantitative, often experimental) and their alternatives (e.g., qualitative, narrative, post-modern) all have their place and are all to be valued. Rather than pitting one approach against another, it is more productive to follow a strategy of fitting the method to the research question (Campbell, 1974, cited in Tebes & Kraemer, 1991).
Interest in the possibilities of pluralist approaches was crystallized by Fiske and Shweder’s (1986) edited volume, to which a number of leading thinkers in the epistemology of social science contributed. Within community psychology, Kelly (1990), Trickett (1996), and Tebes and Kraemer (1991) were early advocates of pluralistic methods in what was at the time a more traditional quantitative discipline. Tolan et al.’s (1990) community research methods volume helped to move the balance more towards non-traditional methods. The 1990s saw a number of social science publications advocating combining quantitative and qualitative approaches, from a pragmatic, mixed methods position (e.g., Steckler, McLeroy, Goodman, Bird, & McCormick, 1992; Tashakkori & Teddlie, 1998). The first editions of some pluralistically oriented social research methods textbooks were also published in the 1990s: Bryman (2004) and Robson (2002) being two noteworthy British examples (current editions cited). We will examine these ideas about putting pluralism into practice more fully in the final section of this paper.
	We have elsewhere defined methodological pluralism as the belief “that no single approach to research is best overall, rather, what is important is that the methods be appropriate for the questions under investigation. No single research method is inherently superior to any other: all methods have their relative advantages and disadvantages” (Barker, Pistrang, & Elliott, 2002, p. 245). It is the converse of the belief that certain research methods are inherently more scientifically valid or respectable than others.
	The remainder of this paper attempts to develop the implications of the pluralist position for the conduct and evaluation of community research.
Quality Standards under Pluralism
Riger et al. (2004), concluding their commentary on issues in participatory action research, pose the question “Whose standards are used to evaluate [research] quality, and what are those standards?” (p.237). This section attempts to address these issues of research quality. It will focus on the second of the questions (what standards?), but it is worth being explicit that, in line with the existing literature, our responses to the first question (whose standards?) mostly concern the standards of academic audiences. We believe that there is a substantial overlap between these standards and those of people outside academia (e.g., research stakeholders, research participants), but it is beyond the scope of the present paper to address this.
The practical issue of how to make judgments about the quality of diverse styles of research is constantly faced by journal editors and reviewers appraising manuscripts and grant proposals, and by community organizations who are prospective stakeholders in research activities. One common criticism of a pluralistic stance is that it is not possible to maintain standards across different genres of research. What standards can be used to evaluate a piece of ethnography as well as a randomized experiment? If no such comprehensive standards exist, is pluralism thus equivalent to anarchy, a way of saying that that everyone just goes ahead and “does their own thing?”
In our view, there is a difference between methodological pluralism and methodological anarchy. Pluralism is distinct from the relativistic “anything goes” stance which Feyerabend (1975) proposes in his aptly titled book Against Method. Pluralism and rigor are not necessarily mutually antithetical.
There is a vast literature on standards in research, including a number of recent publications on qualitative research. We draw on Elliott, Fischer, and Rennie’s (1999) paper on “evolving guidelines” for publishability standards for qualitative research. This is well known in the clinical and counseling fields, but less so elsewhere. Elliott et al.’s approach is first to describe a set of core standards that apply to all research genres, and then to describe a set of criteria that specifically apply to qualitative research. We extend this approach to the pluralist position towards community research, attempting to outline first common criteria and then criteria specific to each of the two main research genres, qualitative and quantitative.
Perhaps an analogy from music may be useful. Music can be divided into different genres (classical, jazz, rock etc.). A critic judging the quality of a musical piece will apply both criteria that transcend genres (e.g., pitch, rhythm, expressiveness), but also criteria specific to each genre (e.g. classical musicians will stay exactly on the beat; jazz musicians may not always). The pluralist position is to celebrate high quality performances across genres -- Schubert and Stravinsky as well as Sonny Rollins or Bruce Springsteen -- while accepting that the way each performance is appraised may require partially different criteria.
At this point, it is useful for heuristic purposes to make a large simplification: namely that there are fundamentally two main genres of research, which, following Tebes & Kraemer (1991) we will label quantitative and qualitative. We acknowledge that this dichotomy is a huge approximation, and not to the liking of some commentators (e.g., Reicher, 2000), who object to the presumption that qualitative approaches form a unitary genre. However, we believe that in this case the virtues of simplicity are more important than those of complexity, and hope that our broad brush does not sacrifice too much of the important detail.
We should add the disclaimer that, like Elliott et al.’s (1999) “evolving guidelines,” our framework is offered in a tentative spirit, as a contribution to the debate about what criteria might be appropriate for judging pluralistic research, realizing that such an enterprise is almost certainly contentious. We do not claim to have original or definitive answers, but we hope to achieve a new integration of existing ideas.
We have divided our criteria into four domains: (1) criteria applicable to all research, (2) research-relevant principles and values in community psychology, (3) criteria specifically applicable to quantitative research, and (4) criteria specifically applicable to qualitative research. We assume that most readers will be reasonably familiar with the first three sets of criteria, but possibly less so with the fourth; for completeness they are all briefly outlined here, with greater emphasis on the last. All the criteria are summarized in Table 1.
____________________
Insert Table 1 about here
____________________
Criteria Applicable to All Research
The first set of criteria is intended to apply across all genres of research. They are global concepts addressing fundamental issues, such as the importance of the investigation and the appropriateness of the procedures used. 
Explication of context and purpose. The conceptual background and context of the investigation need to be clearly and explicitly described, in order that the reader can appraise the impetus for, and necessity of, the study, and also for evaluating its procedures and making sense of the researchers’ interpretations of their findings. This means that the authors situate the study in the existing literature, out of which flow its aims, research questions, or hypotheses. The context is not necessarily limited to the purely academic or research background: the sociopolitical context and the historical background may also be explicated, and the values embodied in the research made manifest. For example, Bond, Belenky, and Weinstock’s (2000) Listening Partners program for isolated, rural, young mothers was conceptualized from a feminist, empowerment perspective. Stating this at the outset helps the reader evaluate the research questions addressed, the methods adopted and the sense made of the data. Qualitative researchers may also disclose aspects of their own personal background relevant to the study (see the section below on qualitative research criteria).
Use of appropriate methods. A fundamental criterion is that the study’s methods are appropriate for the research questions under investigation. This covers both the general research strategy that is employed (for example, experimental, correlational, ethnographic, or narrative) and also how that strategy is put into practice: the nature and number of the participants, the way the data are collected (e.g., semi-structured interviews, standardized questionnaires or participant observation), the research design (e.g., longitudinal or cross sectional) and how the data are analyzed. In other words, the research is question-driven, not method-driven. The phrase “appropriate methods” has intentional echoes of Schumacher’s concept of “appropriate technology” in Small is Beautiful (Schumacher, 1973). That is, it implies that the researcher will use the simplest possible methods for the particular set of research questions (Elliott, 2000) -- the simplest conceptually that is, but not necessarily the easiest for the researcher: the field seems overreliant on monomethod self-report studies, at the expense of the behavioral or even the biological domain.
Transparency of procedures. Exactly what the researchers did should be clearly described and specified. This includes measures, design, and analytic strategies. Transparency is essential so that readers may understand enough about how the study was conducted to be able properly to appraise its conclusions. It also enables other researchers to replicate the study.
Ethical treatment of participants. The study must conform to established ethical criteria (see also the following section on values and principles). In other words, participants are treated with respect, dignity, and constant regard for their welfare. This is put into practice by providing full information, freedom of choice, and protection from harm (which also includes not withholding potential benefits, often an issue in randomized trials).
Importance of findings. The final general criterion is that the investigation makes a contribution to knowledge (either theoretical or applied). Clearly, if the study is trivial, there is little point in doing it, or in publishing it once it is done (and it is unethical to ask participants to take part). At the other end of the scale, it is crucial to recognize when a study has produced important findings; these may need to be weighed against technical flaws in the procedures of the study. However, this criterion is easier stated that realized: the importance of a finding is not always easy to assess, and may only become apparent with the passage of time.
Research-Relevant Community Psychology Values and Principles
Community psychology is characterized by its adherence to a set of social and moral values, arising out of its founding spirit of 1960s social activism (Jason, et al., 2004). Some are shared by other disciplines and fields, but we would argue that community psychology is unique in its value orientation. Some of these values are explicitly stated in the core principles of the field (SCRA website, n.d.); others have been articulated by various authors within the community psychology literature (e.g., Fryer & Fagan, 2003; Langhout, 2003; Orford, 1992; Prilleltensky, 2001; Rappaport, 1990). The list below presents one possible integration of those values and principles applying specifically to research.
In contrast to the first set of criteria, in which all criteria would be expected to apply to all studies, this set represents guiding principles, and not every study or research program would be expected to realize all of them simultaneously. Furthermore, some criteria may be irrelevant to particular research programs. However, most studies would realize at least two or three of them (if not, it is hard to see how they could be regarded as relevant to community psychology).
Sensitivity to people’s contexts. The contextual nature of behavior is a fundamental community psychology principle: Orford (1992) uses the term “person-in-context” to denote this. In research terms, this principle implies conducting research in naturalistic settings, and being conscious of the variability of behavior across social contexts (Moos, 2003; Trickett, 1996). The pioneering ecological psychology studies of Roger Barker and colleagues are classic examples of how researchers might study contextual influences on behavior (Barker, Wright, Schoggen, & Barker, 1978).
Respect for diversity among people and settings. It is almost a corollary of the contextual principle that researchers value and respect diversity. One obvious implication is to investigate phenomena cross-culturally, including cultural groups other than the dominant one in the investigator’s own society or country. Examples of research putting this principle into practice are numerous: one good set of examples was collected in a special issue of this journal on minority issues in prevention (Roosa & Gonzales, 2000).
Addressing competencies (as well as problems). Whereas clinical psychology has tended to view people primarily in terms of their deficits or problems, community psychologists have made a strong argument for focusing on people’s competencies, and of working towards wellness rather just the absence of deficit. This principle has various historical antecedents in other branches of psychology and a contemporary resurgence in the positive psychology movement (Seligman & Csikszentmihalyi, 2000). Cowen was one of the earliest community psychologists to advocate this focus, and his extensive research in primary prevention reflected his efforts to put this into practice (Cowen, 1991, 1994).
Promoting empowerment. Although the term empowerment is somewhat hard to pin down, in essence it refers to conducting research that results in participants’ having greater “influence over decisions that affect their lives” (Rappaport, 1990, p.52). The case for conducting research from “an empowerment agenda” has been powerfully argued by Rappaport (1990), who placed empowerment at the top of community psychology’s research agenda, and viewed it as subsuming many of the other principles delineated here. The empowerment principle also extends to the research setting itself, where it translates into establishing collaborative relationships between researchers and participants.
Giving voice to traditionally underrepresented populations. Strongly related to empowerment is the principle that the research process be used to enable the voices of oppressed or marginalized social groups to be heard. A moving example of this, although arguably not strictly within community psychology, is the life story interviews of Bogdan and Taylor (1976) with “Ed Murphy,” a man with severe intellectual disability. The composite narrative that Bogdan and Taylor (1976) constructed gives a rare glimpse into the thoughts and feelings of a member of a hidden and highly marginalized group -- people in long-stay institutions for those with severe intellectual disabilities.
Promoting social justice. Several authors (e.g., Albee, 1986; Prilleltensky, 2001) have argued that the mission of community psychology extends beyond the level of promoting individual wellness, to promoting social justice for the population as a whole. Research from this perspective is often conducted within a participatory action research framework. For example, Stringer’s (1999) action research text describes a project aimed at promoting positive social change for Aboriginal people in Western Australia.
Research using multiple methodologies. The final principle, of conducting research using multiple methodologies, is the theme of this present paper. We will discuss below how this might be implemented.
Criteria Specifically Applicable to Quantitative Research
There is some debate about the compatibility between the usual methods of quantitative research and community psychology’s values and principles. For example, some authors take the view that experimental manipulation is inherently disempowering, or that standardized questionnaires impede participants’ voices being heard (see for example, Fox & Prilleltensky, 1997). We have some sympathy with these criticisms; for example, qualitative, semi-structured interview methods clearly are easier to use in a context-sensitive, collaborative way, which enables participants to express themselves fully. However, we contend that it is possible to do quantitative research in a way that does not violate these values and principles: were it not, the whole basis of our discipline as a pluralistic one would collapse.
Two classic studies used randomized controlled experiments in a way that is consonant with the above values and principles. In Goodman’s (1976) study of a non-professional helping program (which would be called mentoring today), matched pairs of troubled boys were randomly assigned to a “companionship therapy” intervention or a control group. Despite its traditional research design, the values driving the entire study were drawn from a humanistic orientation to community psychology. Similarly, Fairweather’s pioneering work on creating community settings that empower people with serious mental illness also utilized experimental designs (Fairweather & Davidson, 1986; Hazel & Onaga, 2003). Yet few would argue that this work contravened community psychology principles. 
The principal criteria for evaluating quantitative research have been widely agreed upon, so we describe them only briefly here. They are encapsulated in the four concepts of validity articulated by Cook and Campbell (1980) and later elaborated by Shadish, Cook, and Campbell (2001):
Reliability and validity of measurement. Reliability is concerned with replicability of measurement, or put another way, the amount of random error in the measurement. Validity, a more complex consideration, is concerned with the meaning of the measurement. Each concept has many subtypes, e.g., inter-rater reliability and criterion validity.
Internal validity is a design criterion which assesses the degree to which causal relationships can be inferred from the study. Cook and Campbell (1980) systematically delineate the various threats to internal validity that arise in a number of commonly used designs. From this point of view, the randomized controlled experiment with pre-post measures is a design with high internal validity.
External validity assesses the degree to which the findings of a study can be generalized over setting, place, and time. Studies that take place in more naturalistic community settings have greater external validity than those that take place in more artificial research laboratory settings.
Statistical conclusion validity. The fourth of Cook and Campbell’s criteria concerns the soundness of the statistical procedures which may be used to infer that variables are associated with each other.
Cook and Campbell (1980) acknowledge the difficulty of meeting these four criteria simultaneously, especially in field settings. There is often a trade-off between internal and external validity: the more controlled a study is, the less generalizable it is likely to be. Clearly many community studies will take an action-oriented approach and therefore place a much greater value on external rather than internal validity.
Criteria Specifically Applicable to Qualitative Research
In the last decade a number of authors have addressed and debated the issue of what sort of criteria are appropriate for guiding and evaluating qualitative research (e.g., Creswell, 1998; Elliott et al., 1999; Miles & Huberman, 1994; Reicher, 2000; Stiles, 1993; Whittemore, Case, & Mandle, 2001; Willig, 2001; Yardley, 2000). Part of this debate concerns the extent (if any) to which the traditional concepts of reliability and validity can, or should, be applied to qualitative research. This debate has arisen in part because of some of the defining features of qualitative research, which set it apart from traditional approaches. Qualitative data collection procedures are often highly flexible (e.g., semi-structured interviews and field notes), and the person of the researcher is an intrinsic part of the conduct of the inquiry. Qualitative data analysis procedures similarly involve a process of interpretation by the researcher (although the extent of interpretation varies across qualitative genres) and multiple “readings” of the data are therefore possible and expected. Thus, the concepts of reliability and validity do not easily translate to qualitative approaches.
The fundamental principle underlying the criteria below is that the study be conducted in a systematic and rigorous way. The criteria are intended to capture what we mean by rigor in this context: that is, how qualitative researchers can demonstrate the “trustworthiness” of their methods and findings (Lincoln & Guba, 1985; Schwandt, 1997). Our list is a subset of that proposed by Elliott et al. (1999), and overlaps considerably with those of other authors. As mentioned above, they are intended to apply across different genres of qualitative research: in particular, both to realist and relativist approaches (Schwandt 1997; Willig, 2001), although we acknowledge that this may be more difficult -- and is more contentious -- with radical relativist approaches.
Disclosure of perspective. Given that the person of the researcher influences the conduct of the inquiry and the interpretations of the data, it becomes important that the researcher discloses relevant personal background, so that the reader can better evaluate the conclusions drawn. This criterion overlaps somewhat with the first criterion applicable to all research, that of giving the context and purpose of the study. However, it has been recommended that qualitative researchers describe not just the conceptual background to the study, but also any personal characteristics that might have influenced how the data were collected or analyzed. These may be demographic characteristics, such as gender or ethnic background, as well as personal characteristics, such as first-hand experience with the phenomenon under investigation. (One could argue that such disclosure should also be applicable to quantitative research, in that the researcher’s background also exerts an influence, for example on the framing of hypotheses or the selection of measures.)
Our impression, from informally surveying current community journals, is that such disclosure is still relatively rare (for a recent exception, see the papers collected in Primavera & Brodsky, 2004). This is in contrast to the clinical and counseling field, where it is more standard practice. A nice example of researcher disclosure, in a psychotherapy journal, is found in Bolger’s (1999) study of the nature of emotional pain experienced by the adult children of alcoholics. She gives an extensive account of her personal background, including the fact that she herself is also the adult child of an alcoholic, which clearly had some bearing on how she conducted the study and made sense of the data. It is important to note that the influence of personal characteristics is not necessarily problematic (traditional researchers would say that there is a problem of “bias”); for example, Bolger’s personal background was probably an asset, both in terms of gaining the trust of participants and also in terms of being more finely attuned to the nuances of the phenomenon under study.
Grounding interpretations in the data. It is important that research papers provide sufficient examples of the actual observations (be they quotations from participants or field notes) to make the researcher’s understanding of the data explicit. The rationale is twofold. First, readers need to be able to judge the fit between the data and the interpretations drawn. That is, the researcher needs to demonstrate a link between the primary data, as the point of departure, and the conclusions, as the conceptual destination. Second, providing examples of the primary data allows readers to consider other “readings,” i.e. to generate alternative explanations or apply alternative conceptual frameworks. Furthermore, providing primary data allows the participants’ words to be directly conveyed to the reader: often they are the most powerful aspects of the report. Such “grounding” in the data, as advocated by proponents of grounded theory (Strauss & Corbin, 1998), has become standard practice, although the extent of it varies. Ideally, the grounding should be carried out in way that provides sufficient detail of the individuals being studied and the research context, so that readers can gain a rich, contextual understanding of the material. One example of a well grounded report is Maton, Hrabowski, and Grelf’s (1998) study of high-achieving African American young men. Themes and brief case studies are richly illustrated with excerpts from the interviews with the young men and their parents, providing a vivid and clear picture of the pathways to academic success.
Coherence of interpretive framework. In a good qualitative study, the ideas that are induced from the data are presented within a logically coherent structure and fully address the complexity of the phenomenon under study. This often consists of a set of themes or categories. However, the researchers do not simply provide a list of categories, but present an integrated picture that sheds new light on the phenomenon under investigation. For example, Stein and Wemmerus (2001) studied the life changes experienced by young adults diagnosed with schizophrenia, and their families. The researchers used a life course perspective to give coherence to the participants’ accounts, presenting the results in terms of three temporally-organized themes concerning families’ views of “a normal life”: the loss, the search, and the future.
Credibility checks. A key criterion for ensuring the rigor of qualitative research is that the investigator has undertaken procedures to check the trustworthiness, or believability, of his or her conclusions, i.e., that the interpretations made do not simply reflect the researcher’s own flight of fancy. A number of  “credibility checks” have been proposed (Elliott et al., 1999; Miles & Huberman, 1994; Stiles, 1993), the principal ones being: 
1.	Consensus: Using a team of researchers to jointly analyze the data, rather than relying on a single analyst. As recommended by Hill, Thompson, and Williams (1997), this approach can involve members of the team independently analyzing some or all of the data, and then coming together to compare notes and debate different conceptualizations, eventually reaching a consensus on the best way of representing the data.
2.	Auditing: Having another researcher (not directly involved in conducting the study) examine an “audit trail” provided by the investigator. Such an audit trail would usually start with the procedures used to generate the data, and then proceed to document each step of the analysis. The aim is to assess the soundness of the findings, for example by checking that the analytic process was logical and well documented. The role of the auditor is not necessarily to agree with the interpretations made (i.e., the issue is not consensus), but rather to judge the extent to which the conclusions are backed up by the data and the analytic procedures.
3.	Respondent validation or member checks: Asking the research participants themselves to comment on the interpretations made by the researcher. (Other terms sometimes used for this are testimonial validity and member validation.) This has been noted as a potentially problematic procedure on several grounds: for example, participants may simply agree with the researcher’s conclusions because of a power imbalance, or they may disagree because the researcher has used a level of abstraction that is unfamiliar. Thus, although member checks may not always be able to provide corroboration, they are often fruitful in generating insight and expanding the researcher’s understanding of the phenomenon. Furthermore, soliciting participants’ views of the analysis is consistent with community psychology’s value on collaborative relationships between researchers and participants.
4.	Triangulation: Establishing whether the findings are consistent with those derived from other methods or sources. The researcher may elicit multiple perspectives on the phenomenon, e.g., by interviewing different members of a family, or may use more than one source of information, e.g., field observations plus in-depth interviews.
	We are not implying that all of these credibility checks would be expected to be carried out in every study. For example, owing to limitations of time or resources, triangulation may not be feasible or respondent validation impractical. However, we would normally expect researchers to incorporate one or more of them, so that the credibility of their findings can be demonstrated.
Implications of Pluralism
	This final section addresses the implications of a pluralist stance for how community psychology research is conducted. We will examine how pluralism may operate at each of three levels: within an individual study, across different studies in a research program, or within the community psychology field as a whole. Each of these levels manifests somewhat different types of pluralism: integrating methods within a single study, using different approaches even-handedly within a research program, and mutual tolerance and appreciation of different approaches within the field as a whole.
Mixed Method Studies
	First, pluralism can occur in an integrative way, at the level of the individual study. Different research methods can be combined within that study, where, for example, investigators may collect both qualitative and quantitative data from the same participants in order to provide different perspectives on the study’s research question(s). A good illustration is Langhout’s (2003) single case study, which used qualitative interviewing, multidimensional scaling, and social network analysis to examine in depth a third-grade boy’s sense of place in his school and community.
	Of course, different methods may be combined in different ways with different emphases. Several authors (e.g., Creswell, 2002; Morgan, 1998; Steckler, McLeroy, Goodman, Bird, & McCormick, 1992; Tashakkori & Teddlie, 1998, 2002) have attempted to classify such mixed model designs (whether they are referring to single studies or linked studies in sequence is not always explicit, however). For example, Morgan’s (1998) “priority sequence model” uses a two-by-two schema to classify studies according to which method (qualitative or quantitative) is primary in time and which is primary in importance (the case where different approaches may have equal importance is not addressed, although there is no reason why this should be precluded). So, for example, a quantitative screening instrument may be used to select suitable participants for in-depth qualitative interviews (qualitative methods secondary in time, but primary in importance). As a second example, an experimental design could be used to estimate the net gain for participants who receive a peer support intervention, and then qualitative interviews could be used in understanding how that intervention works or how the participants’ worldviews change as a result (qualitative methods secondary in time, but equal in importance). Further elaboration of such mixed methods designs is beyond the scope of the present paper: readers are referred to specialist texts (e.g., Creswell, 2002; Tashakkori & Teddlie, 2002) for more details.
Pluralistic Research Programs
	At a second level, pluralism can occur when different methods are used even-handedly across a research program, by the same investigators looking at the same phenomena from several different methodological viewpoints. The intention may be either to provide triangulation on the findings of one study, or to investigate the same questions from different perspectives.
	For example, Humphreys has carried out a number of studies examining the benefits of mutual support groups for people with substance use problems. He and his colleagues have used traditional quasi-experimental designs, in order to compare rates of healthcare usage for patients in treatment programs that do and do not utilize self-help groups (Humphreys & Moos, 2001), and he has also used narrative analysis to examine the nature and function of AA members’ stories about their community (Humphreys, 2000). As a second example, our own research into the processes of social support has used both traditional quantitative observer ratings to compare the communication of different informal helpers (Pistrang & Barker, 1998) and also qualitative analysis to understand how participants construed empathic communication (Pistrang, Picciotto, & Barker, 2001).
	The integrative and even-handed approaches to pluralism take a pragmatic approach to research methodology, in other words that the researcher uses whatever methods are appropriate for a given question. This pragmatic position is in contrast to that of the methodological purists, who during the fiercely fought  “paradigm wars” contended that the incompatible epistemological positions of different approaches meant that they could never be used harmoniously (see Tashakkori & Teddlie, 1998). While different approaches do certainly draw on different philosophical positions, the view that they are inherently incompatible with each other seems no longer sustainable.
A Pluralistic Field
	Finally, at the level of the discipline as a whole pluralism can occur in a mutually tolerant way. The field can be open to evidence from both the traditionally scientific and the newer human science approaches. This is shown both by a healthy balance of publications in flagship community psychology journals, by the inclusion of qualitative studies in review articles, and by funding bodies supporting qualitative as well as quantitative investigations.
	This state, which has been largely realized in community psychology, contrasts with that in clinical psychology, where the “empirically-supported therapies” movement (Chambless & Hollon, 1998) is arguing for the primacy of only a narrow type of evidence (i.e., that emerging from randomized controlled trials). A similar position is taken in the “evidenced-based medicine” movement (Sackett, Richardson, Rosenberg, & Haynes, 1997). Randomized trials clearly yield valuable evidence for comparative effectiveness of interventions, but there are widespread concerns that setting this design as a gold standard will stifle research innovation and productivity (Elliott, 1998).
	Although community psychology has made great strides towards becoming a fully pluralistic field, some residual tension between proponents of traditional and emergent approaches remains. This tension is partly manifested in having to define one’s own approach as an antithesis to other approaches. This tendency applies equally to those traditional researchers who venerate the randomized controlled trial and regard qualitative approaches as unscientific (or condescendingly describe them as just common sense), as much as to those qualitative researchers who introduce their studies with “positivism-bashing” polemics against the traditional quantitative, “scientific” approach. Methodological pluralism values the traditional scientific approach equally as much as its alternatives -- regarding each as appropriate for different types of enquiry under different circumstances. In a mature pluralist discipline, it should be possible to state the value of one’s own approach without needing to define it in opposition to its alternatives.
	Ponterrotto and Grieger (1999) use the metaphor of cultural competency in everyday life to make the analogy with competency in understanding, appreciating and using qualitative and quantitative research methods. They argue that researchers need to attain a basic competency in at least the understanding of different modes and approaches to research. 
We are not, however, saying that every study, or every researcher, has to use multiple methods. Individual researchers will each have their own personal skills and preferences with respect to research approaches; it is an important part of the pluralist ethos that they be encouraged to pursue their own path – within, of course, the constraints of the “appropriate methods” criterion. What is, however, important is that the field itself, in aggregate, does so in an atmosphere of mutual tolerance.
Summary and Conclusions
	“Thus scientific methodology is seen for what it truly is - a way of preventing me from deceiving myself in regard to my creatively formed hunches which have developed out of the relationship between me and my material.” (Rogers, 1955, p.275)
Carl Rogers was writing in another time and another context, but his remark captures much of the essence of the pluralist spirit. It acknowledges the need for disciplined scientific enquiry – since otherwise we would all get carried away by our enthusiasms and self-deceptions – but takes an inclusive position on what “science” consists of. He implies that if an approach serves to help check the soundness of one’s ideas, it would come under his definition of scientific methodology. This is of course not to say either that non-scientific activities are unimportant, or that knowledge only derives from empirical research. Many non-scientific activities, such as social activism and critical psychology approaches, are equally important in the development of the discipline.
This paper has attempted to develop the implications of community psychology’s self-avowed methodological pluralism, both for how research might be appraised and how it might be conducted. We have done this with some qualms, as the process of setting out criteria can potentially have the undesirable consequence of leading to overly rigid applications of those criteria, ending up by limiting diversity. However, on balance, we regard the process of making methodological criteria explicit as valuable in helping the field develop a broadly agreed sense of where it is heading. We re-iterate that, like Elliott et al.’s (1999) “evolving guidelines,” on which they were based, our set is put forward in a tentative spirit, in the hope that they will stimulate productive debate.
	Although many of the themes we have touched on are shared in common with other forms of social enquiry, we consider that the methodological pluralistic epistemological approach has a particularly crucial role to play in community psychology, given its central ethos of valuing diversity and differences. Our hope is that community psychology can evolve towards being a mature pluralist field, demonstrated both by there being a substantial number of researchers competent in both qualitative and quantitative approaches to enquiry, and also by the field as a whole having by adopting an open-arms policy towards divergent types of evidence, both those traditionally labeled as “scientific” and those not.
References
Albee, G.W. (1986). Toward a just society: lessons from observations on the primary prevention of psychopathology. American Psychologist, 41, 891-898.
Barker, C., Pistrang, N., & Elliott, R. (2002). Research methods in clinical psychology: An introduction for students and practitioners (2nd ed.). Chichester: Wiley.
Barker, R.G., Wright, H.F., Schoggen, M.F., & Barker, L.S. (1978). Habitats, environments, and human behavior. San Francisco: Jossey-Bass.
Bogdan, R., & Taylor, S. (1976). The judged, not the judges: An insider’s view of mental retardation. American Psychologist, 31, 47-52.
Bolger, E. A. (1999). Grounded theory analysis of emotional pain. Psychotherapy Research, 9, 342-362.
Bond, L.A., Belenky, M.F., & Weinstock, J.S. (2000). The Listening Partners program: an initiative toward feminist community psychology in action. American Journal of Community Psychology, 28, 697-731.
Bryman, A. (2004). Social research methods (2nd ed.). Oxford: Oxford University Press.
Chambless, D.L., & Hollon, S.D. (1998). Defining empirically supported therapies. Journal of Consulting and Clinical Psychology, 66, 7-18.
Cook, T.D., & Campbell, D.T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Chicago: Rand-McNally.
Cowen, E. (1991). In pursuit of wellness. American Psychologist, 46, 404-408.
Cowen, E. (1994). The enhancement of psychological wellness: challenges and opportunities. American Journal of Community Psychology, 22, 149-179.
Creswell, J.W. (1998). Qualitative inquiry and research design: Choosing among five traditions. Thousand Oaks, CA: Sage.
Creswell, J.W. (2002). Research design: Qualitative, quantitative, and mixed method approaches (2nd ed.). Thousand Oaks, CA: Sage.
Driver-Linn, E. (2003). Where is psychology going? Structural fault lines revealed by psychologists’ use of Kuhn. American Psychologist, 58, 269-278.
Elliott, R. (1998). Editor’s introduction: a guide to the empirically-supported treatments controversy. Psychotherapy Research, 8, 115-125.
Elliott, R. (2000). Rigor in psychotherapy research: The search for appropriate methodologies. Unpublished manuscript, University of Toledo.
Elliott, R., Fischer, C.T., & Rennie, D.L. (1999). Evolving guidelines for publication of qualitative research studies in psychology and related fields. British Journal of Clinical Psychology, 38, 215-229.
Fairweather, G.W., & Davidson, W.S., II. (1986). Introduction to community experimentation: Theory, methods, and practice. New York: McGraw-Hill.
Feyerabend, P. (1975). Against method. London: Verso.
Fiske, D.W., & Shweder, R.A. (Eds.) (1986). Metatheory in social science: Pluralisms and subjectivities. Chicago: University of Chicago Press.
Fox, D., & Prilleltensky, I. (Eds.) (1997). Critical psychology: An introduction. Thousand Oaks, CA: Sage.
Fryer, D., & Fagan, R. (2003). Toward a critical community psychological perspective on unemployment and mental health. American Journal of Community Psychology, 32, 89-96.
Goodman, G. (1972). Companionship therapy: Studies in structured intimacy. San Francisco: Jossey-Bass.
Hazel, K.L., & Onaga, E. (2003). Experimental Social Innovation and Dissemination: the promise and its delivery. American Journal of Community Psychology, 32, 285-294.
Hill, C.E., Thompson, B.J., & Williams, E.N. (1997). A guide to conducting consensual qualitative research. The Counseling Psychologist, 25, 517-572.
Humphreys, K. (2000). Community narratives and personal stories in Alcoholics Anonymous. Journal of Community Psychology, 28, 495-506.
Humphreys, K. & Moos, R. H. (2001). Can encouraging substance abuse patients to participate in self-help groups reduce demand for health care? A quasi-experimental study. Alcoholism: Clinical and Experimental Research, 25, 711-716.
Jason, L., Davis, M.I., Suarez-Balcazar, Y., Keys, C.B, Taylor, R.R., Isenberg, D.H., & Durlak, J.A. (2004). Conclusion. In L. Jason, C.B. Keys, Y. Suarez-Balcazar, R.R. Taylor, & M.I. Davis, (Eds.), Participatory community research: Theories and methods in action (pp. 241-245). Washington, DC: American Psychological Association.
Kelly, J.G. (1990). Changing contexts and the field of community psychology. American Journal of Community Psychology, 18, 769-792.
Kelly, J.G. (2003). Science and community psychology: Social norms for pluralistic enquiry. American Journal of Community Psychology, 31, 213-217.
Langhout, R.D. (2003). Reconceptualizing quantitative and qualitative methods: A case study dealing with place as an exemplar. American Journal of Community Psychology, 32, 229-244.
Lincoln, Y., & Guba, E.G. (1985). Naturalistic inquiry. Beverly Hills, CA: Sage.
Maton, K.I., Hrabowski, F.A., & Grelf, G.L. (1998). Preparing the way: a qualitative study of high-achieving African American males and the role of the family. American Journal of Community Psychology, 26, 639-668.
Miles, M.B., & Huberman, A.M. (1994). Qualitative data analysis: An expanded sourcebook (2nd ed.). Thousand Oaks, CA: Sage.
Morgan, D.L. (1998). Practical strategies for combining qualitative and quantitative methods: applications to health research. Qualitative Health Research, 8, 362-376.
Orford, J. (1992). Community psychology: Theory and practice. Chichester: Wiley.
Pistrang, N., & Barker, C. (1998). Partners and fellow patients: Two sources of emotional support for women with breast cancer. American Journal of Community Psychology, 26, 439-456.
Pistrang, N., Picciotto, A., & Barker, C. (2001). The communication of empathy in couples during the transition to parenthood. Journal of Community Psychology, 29, 615-636.
Ponterrotto, J.G., & Grieger, I. (1999). Merging qualitative and quantitative perspectives in a research identity. In M. Koppala & L.A. Suzuki (Eds.), Using qualitative methods in psychology (pp. 49-62). Thousand Oaks, CA: Sage.
Prilleltensky, I. (2001). Value based praxis in community psychology: Moving toward social justice and social action. American Journal of Community Psychology, 29, 747-778.
Primavera, J., & Brodsky, A.E. (2004). Processes of community research and action [Special issue]. American Journal of Community Psychology, 33(3/4).
Rappaport, J. (1990). Research methods and the empowerment social agenda. In P. Tolan, C. Keys, F. Chertok, & L. Jason (Eds.), Researching community psychology: Issues of theory and methods (pp. 51-63). Washington, DC: American Psychological Association.
Reicher, S. (2000). Against methodolatry: Some comments on Elliott, Fischer, and Rennie. British Journal of Clinical Psychology, 39, 1-6.
Riger, S., Reyes, O., Watts, R.W., Kelley, J.G., Shinn, M., Cherniss, C., Jason, L., & Trickett, E. (2004). Faculty deconstructs participatory research. In L. Jason, C.B. Keys, Y. Suarez-Balcazar, R.R. Taylor, & M.I. Davis, (Eds.), Participatory community research: Theories and methods in action (pp.233-238). Washington, DC: American Psychological Association.
Robson, C. (2002). Real world research: A resource for social scientists and practitioner-researchers (2nd ed.). Oxford: Blackwell.
Rogers, C.R. (1955). Persons or science? A philosophical question. American Psychologist, 10, 267-278.
Roosa, M.W., & Gonzales, N.A. (2000). Minority issues in prevention [Special issue]. American Journal of Community Psychology, 28(2).
Sackett, D.L., Richardson, W.S. Richardson, Rosenberg, W., & Haynes, R.B. (1997). Evidence-based medicine: How to practice and teach EBM. New York: Churchill Livingstone.
Schumacher, E.F. (1973). Small is beautiful: A study of economics as if people mattered. London: Blond and Briggs.
Schwandt, T.A. (1997). Qualitative inquiry: A dictionary of terms. Thousand Oaks, CA: Sage.
Seligman, M.E.P., & Csikszentmihalyi, M. (2000). Positive psychology: An introduction. American Psychologist, 55, 5-14.
Shadish, W.R., Cook, T.D., & Campbell, D.T. (2001). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin.
SCRA (the Society for Community Research and Action) (n.d.). The SCRA mission. Retrieved December 1, 2003, from http://www.apa.org.org/divisions/div27/goals.html
Solomon, M., Pistrang, N., & Barker, C. (2001). The benefits of mutual support groups for parents of children with disabilities. American Journal of Community Psychology, 29, 113-132.
Steckler, A., McLeroy, K.R., Goodman, R.M., Bird, S.T., & McCormick, L. (1992). Toward integrating qualitative and quantitative methods: An introduction. Health Education Quarterly, 19, 1-8.
Stein, C.H., & Wemmerus, V.A. (2001). Personal accounts of adults with schizophrenia, their parents, and well siblings. American Journal of Community Psychology, 29, 725-746.
Stiles, W.B. (1993). Quality control in qualitative research. Clinical Psychology Review, 13, 593-618.
Strauss, A. & Corbin, J. (1998). Basics of qualitative research: Techniques and procedures for developing grounded theory (2nd ed.). Newbury Park, CA: Sage.
Stringer, E.T. (1999). Action research (2nd ed.). Thousand Oaks, CA: Sage.
Tashakkori, C., & Teddlie, C. (1998). Mixed methodology: Combining qualitative and quantitative approaches. Thousand Oaks, CA: Sage.
Tashakkori, C., & Teddlie, C. (Eds.) (2002). Handbook of mixed methods in social and behavioral research. Thousand Oaks, CA: Sage.
Tebes, J. K. & Kraemer, D. T. (1991). Qualitative and quantitative knowing in mutual support research: some lessons from the recent history of scientific psychology. American Journal of Community Psychology, 19, 739-756.
Tolan, P., Keys, C., Chertok, F., & Jason, L. (Eds.) (1990). Researching community psychology: Issues of theory and methods. Washington, DC: American Psychological Association.
Trickett, E.J. (1996). A future for community psychology: the context of diversity and the diversity of contexts. American Journal of Community Psychology, 23, 209-229.
Wandersman, A. (2003). Community science: Bridging the gap between science and practice with community-centered models. American Journal of Community Psychology, 31, 227-242.
Whittemore, R., Case, S.K., & Mandle, C.L. (2001). Validity in qualitative research. Qualitative Health Research, 11, 522-537.
Willig, C. (2001). Introducing qualitative research in psychology: Adventures in theory and method. Buckingham: Open University Press.
Yardley, L. (2000). Dilemmas in qualitative health research. Psychology and Health, 15, 215-228.

Table 1

Summary of Criteria and Values Applicable to Pluralistic Community Research

_________________________________________________________________

Criteria applicable to all research

Explication of context and purpose
Use of appropriate methods
Transparency of procedures
Ethical treatment of participants
Importance of findings

Research-relevant community psychology values and principles

Sensitivity to people’s contexts
Respect for diversity among people and settings
Addressing competencies
Promoting empowerment
Giving voice to traditionally underrepresented populations
Promoting social justice
Research using multiple methodologies

Criteria specifically applicable to quantitative research

Reliability and validity of measurement
Internal validity
External validity
Statistical conclusion validity

Criteria specifically applicable to qualitative research

Disclosure of perspective
Grounding interpretations in the data
Coherence of interpretive framework
Credibility checks (e.g., consensus; auditing; respondent validation; triangulation)

_________________________________________________________________