What is rigor in quantitative research?

Conducting a naturalistic inquiry in general is not an easy task. Qualitative studies are more complex in many ways than a traditional investigation. Quantitative research follows a structured, rigid, preset design with the methods all prescribed. In naturalistic inquiries, planning and implementation are simultaneous, and the research design can change or is emergent. Preliminary steps must be accomplished before the design is fully implemented from making initial contact and gaining entry to site, negotiating consent, building and maintaining trust, and identifying participants. The steps of a qualitative inquiry are also repeated multiple times during the process. As the design unfolds, the elements of this design are put into place, and the inquirer has minimal control and should be flexible. There is continuous reassessment and reiteration. Data collection is carried out using multiple techniques, and whatever the source maybe, it is the researcher who is the sole instrument of the study and the primary mode of collecting the information. All the while during these processes, the qualitative inquirer must be concerned with rigor.1 Appropriate activities must be conducted to ensure that rigor had been attended to in the research process rather than only adhering to set criteria for rigor after the completion of the study.1-4

Reliability and validity are 2 key aspects of all research. Researchers assert that rigor of qualitative research equates to the concepts reliability and validity and all are necessary components of quality.5,6 However, the precise definition of quality has created debates among naturalistic inquirers. Other scholars consider different criteria to describe rigor in qualitative research process.7 The 2 concepts of reliability and validity have been operationalized eloquently in quantitative texts but at the same time were deemed not pertinent to qualitative inquiries in the 1990s. Meticulous attention to the reliability and validity of research studies is particularly vital in qualitative work, where the researcher's subjectivity can so readily cloud the interpretation of the data and where research findings are often questioned or viewed with skepticism by the scientific community (Brink, 1993).

This article will discuss the issue of rigor in relation to qualitative research and further illustrate the process using a phenomenological study as an exemplar based on Lincoln and Guba's1 (1985) techniques. This approach will clarify and define some of these complex concepts. There are numerous articles about trustworthiness in the literature that are too complex, confusing, and full of jargon. Some of these published articles also discuss rigor vis-à-vis reliability and validity in a very complicated way. Rigor will be first defined followed by how “reliability and validity” should be applied to qualitative research methods during the inquiry (constructive) rather than only post hoc evaluation. Strategies to attain reliability and validity will be described including the criteria and techniques for ensuring its attainment in a study. This discussion will critically focus on the misuse or nonuse of the concept of reliability and validity in qualitative inquiries, reestablish its importance, and relate both to the concept of rigor. Reflecting on my own research experience, recommendations for the renewed use of the concept of reliability and validity in qualitative research will be presented.

RIGOR VERSUS TRUSTWORTHINESS

Rigor of qualitative research continues to be challenged even now in the 21st century—from the very idea that qualitative research alone is open to questions, so with the terms rigor and trustworthiness. It is critical to understand rigor in research. Rigor is simply defined as the quality or state of being very exact, careful, or with strict precision8 or the quality of being thorough and accurate.9 The term qualitative rigor itself is an oxymoron, considering that qualitative research is a journey of explanation and discovery that does not lend to stiff boundaries.10

Rigor and truth are always of concern for qualitative research.11 Rigor has also been used to express attributes related to the qualitative research process.12,13 Per Morse et al4 (2002), without rigor, research is worthless, becomes fiction, and loses its use. The authors further defined rigor as the strength of the research design and the appropriateness of the method to answer the questions. It is expected that qualitative studies be conducted with extreme rigor because of the potential of subjectivity that is inherent in this type of research. This is a more difficult task when dealing with narratives and people than numbers and statistics.14 Davies and Dodd13 (2002) refer rigor to the reliability and validity of research and that, inherent to the conception, the concept is a quantitative bias. Several researchers argued that reliability and validity pertain to quantitative research, which is unrelated or not pertinent to qualitative inquiry because it is aligned with the positivist view.15 It is also suggested that a new way of looking at reliability and validity will ensure rigor in qualitative inquiry.1,16 From Lincoln and Guba's crucial work in the 1980s, reliability and validity were replaced with the concept “trustworthiness.” Lincoln and Guba1 (1985) were the first to address rigor in their model of trustworthiness of qualitative research. Trustworthiness is used as the central concept in their framework to appraise the rigor of a qualitative study.

Trustworthiness is described in different ways by researchers. Trustworthiness refers to quality, authenticity, and truthfulness of findings of qualitative research. It relates to the degree of trust, or confidence, readers have in results.14 Yin17 (1994) describes trustworthiness as a criterion to judge the quality of a research design. Trustworthiness addressed methods that can ensure one has carried out the research process correctly.18 Manning19 (1997) considered trustworthiness as parallel to the empiricist concepts of internal and external validity, reliability, and objectivity. Seale20 (1999) asserted that trustworthiness of a research study is based on the concepts of reliability and validity. Guba2 (1981), Guba and Lincoln3 (1982), and Lincoln and Guba1 (1985) refer to trustworthiness as something that evolved from 4 major concerns that relate to it in which the set of criteria were based on. Trustworthiness is a goal of the study and, at the same time, something to be judged during the study and after the research is conducted. The 4 major traditional criteria are summarized into 4 questions about truth value, applicability, consistency, and neutrality. From these, they proposed 4 analogous terms within the naturalistic paradigm to replace the rationalistic terms: credibility, transferability, dependability, and confirmability.1 For each of these 4 naturalistic terms are research activities or steps that the inquirer should be engage in to be able to safeguard or satisfy each of the previously mentioned criteria and thus attain trustworthiness (Supplemental Digital Content 1, https://links.lww.com/DCCN/A18). Guba and Lincoln1 (1985) stated:

The criteria aid inquirers in monitoring themselves and in guiding activities in the field, as a way of determining whether or not various stages in the research are meeting standards for quality and rigor. Finally, the same criteria may be used to render ex-post facto judgments on the products of research, including reports, case studies, or proposed publications.

Standards and checklist were developed in the 1990s based on Lincoln and Guba's1 (1985) established criteria, which were then discarded in favor of principles.21 These standards and checklists consisted of long list of strategies used by qualitative researchers, which were thought to cause harm because of the confusion on which strategies were appropriate for certain designs or what type of naturalistic inquiry is being evaluated. Thus, researchers interpreted missing data as faults and flaws.21 Morse21 (2012) further claimed that these standards became the qualitative researchers' “worst enemies” and such an approach was not appropriate. Guba and Lincoln18 (1989) later proposed a set of guidelines for post hoc evaluation of a naturalistic inquiry to ensure trustworthiness based on the framework of naturalism and constructivism and beyond the conventional methodological ideas. The aspects of their criteria have been fundamental to development of standards used to evaluate the quality of qualitative inquiry.4

THE RIGOR DEBATES: TRUSTWORTHINESS OR RELIABILITY AND VALIDITY?

A research endeavor, whether quantitative or qualitative, is always evaluated for its worth and merits by peers, experts, reviewers, and readers. Does this mean that a study is differentiated between “good” and “bad”? What determines a “good” from a “bad” inquiry? For a quantitative study, this would mean determining the reliability and validity, and for qualitative inquiries, this would mean determining rigor and trustworthiness. According to Golafshani22 (2003), if the issues of reliability, validity, trustworthiness, and rigor are meant to differentiating a “good” from “bad” research, then testing and increasing the reliability, validity, trustworthiness, and rigor will be important to the research in any paradigm. However, do reliability and validity in quantitative research equate totally to rigor and trustworthiness in qualitative research? There are many ways to assess the “goodness” of a naturalistic inquiry. Guba and Lincoln18 (1989) asked, “‘What standards ought apply?’… goodness criteria like paradigms are rooted in certain assumptions. Thus, it is not appropriate to judge constructivist evaluations by positivistic criteria or standards or vice versa. To each its proper and appropriate set.”

Reliability and validity are analogues and are determined differently than in quantitative inquiry.21 The nature and purpose of the quantitative and qualitative traditions are also different that it is erroneous to apply the same criteria of worthiness or merit.23,24 The qualitative researcher should not focus on quantitatively defined indicators of reliability and validity, but that does not mean that rigorous standards are not appropriate for evaluating findings.11 Evaluation, like democracy, is a process that, to be at its best, depends on the application of enlightened and informed self-interest.18 Agar24 (1986), on the other hand, suggested that terms such as reliability and validity are comparative with the quantitative view and do not fit the details of qualitative research. A different language is needed to fit the qualitative view. From Leininger25 (1985), Krefting23 (1991) asserted that addressing reliability and validity in qualitative research is such a different process that quantitative labels should not be used. The incorrect application of the qualitative criteria of rigor to studies is as problematic as the application of inappropriate quantitative criteria.23 Smith26 (1989) argued that, for qualitative research, this means that the basis of truth or trustworthiness becomes a social agreement. He emphasizes that what is judged true or trustworthy is what we can agree, conditioned by time and place, and is true or trustworthy. Validity standards in qualitative research are also even more challenging because of the necessity to incorporate rigor and subjectivity, as well as creativity into the scientific process.27 Furthermore, Leininger25 (1985) claimed that it is not whether the data are reliable or valid but how the terms reliability and validity are defined. Aside from the debate whether reliability and validity criteria should be used similarly in qualitative inquiries, there is also an issue of not using the concepts at all in naturalistic studies.

Designing a naturalistic inquiry is very different from a traditional quantitative notion of design and that defining a “good” qualitative inquiry is controversial and has gone through many changes.21 First is the confusion on the use of terminologies “rigor” and “trustworthiness.” Morse28 (2015) suggested that it is time to return to the terminology of mainstream social science and to use “rigor” rather than “trustworthiness.” Debates also continue about why some qualitative researchers do not use the concept of reliability and validity in their studies referring to Lincoln and Guba's1 (1985) criteria for trustworthiness, namely, transferability, dependability, confirmability, and credibility. Morse28 (2015) further suggested replacing these criteria to reliability, validity, and generalizability. The importance and centrality of reliability and validity to qualitative inquiries have in some way been disregarded even in the current times. Researchers from the United Kingdom and Europe continue to do so but not much so in North America.4 According to Morse21 (2012), this gives the impression that these concepts are of no concern to qualitative research. Morse29 (1999) stated, “Is the terminology worth making a fuzz about?”, when Lincoln and Guba1 (1985) described trustworthiness and reliability and validity as analogs. Morse29 (1999) further articulated that:

To state that reliability and validity are not pertinent to qualitative inquiry places qualitative research in the realm of being not reliable and not valid. Science is concerned with rigor, and by definition, good rigorous research must be reliable and valid. If qualitative research is unreliable and invalid, then it must not be science. If it is not science, then why should it be funded, published, implemented, or taken seriously?

Reliability and validity should be taken into consideration by qualitative inquirers while designing a study, analyzing results, and judging the quality of the study,30 but for too long, the criteria used for evaluating rigor are applied after a research is completed—a considerably wrong tactic.4 Morse and colleagues4 (2002) argued that, for reliability and validity to be actively attained, strategies for ensuring rigor must be built into the qualitative research process per se not to be proclaimed only at the end of the inquiry. The authors suggest that focusing on strategies to establish rigor at the completion of the study (post hoc), rather than during the inquiry, exposes the investigators to the risk of missing and addressing serious threats to the reliability and validity until it is too late to correct them. They further asserted that the interface between reliability and validity is important especially for the direction of the analysis process and the development of the study itself.

Reliability

In the social sciences, the whole notion of reliability in and of itself is problematic.31 The scientific aspect of reliability assumes that repeated measures of a phenomenon (with the same results) using objective methods establish the truth of the findings.32-35 Merriam36 (1995) stated that, “The more times the findings of a study can be replicated, the more stable or reliable the phenomenon is thought to be.” In other words, it is the idea of replicability,22,34,37 repeatability,21,22,26,30,31,36,38-40 and stability of results or observation.25,39,41 The issues are that human behaviors and interactions are never static or the same. Measurements and observations can also be repeatedly wrong. Furthermore, researchers have argued that the concept reliability is misleading and has no relevance in qualitative research related to the notion of “measurement method,” as in quantitative studies.40,42 It is a fact that quantitative research is supported by the positivist or scientific paradigm that regards the world as made up of observable, measurable facts. Qualitative research, on the other hand, produces findings not arrived at by means of statistical procedures or other means of quantification. On the basis of the constructivist paradigm, it is a naturalistic inquiry that seeks to understand phenomena in context-specific settings in which the researcher does not attempt to manipulate the phenomenon of interest.23 If reliability is used as a criterion in qualitative research, it would mean that the study is “not good.” A thorough description of the entire research process that allows for intersubjectivity is what indicates good quality when using qualitative methodology. Reliability is based on consistency and care in the application of research practices, which are reflected in the visibility of research practices, analysis, and conclusions, reflected in an open account that remains mindful of the partiality and limits of the research findings.13 Reliability and similar terms are presented in Supplemental Digital Content 2 (see Supplemental Digital Content 2, https://links.lww.com/DCCN/A19).

Validity

Validity is broadly defined as the state of being well grounded or justifiable, relevant, meaningful, logical, confirming to accepted principles or the quality of being sound, just, and well founded.8 The issues surrounding the use and nature of the term validity in qualitative research are controversial and many. It is a highly debated topic both in social and educational research and is still often a subject of debate.43 The traditional criteria for validity find their roots in a positivist tradition, and to an extent, positivism has been defined by a systematic theory of validity.22 Validity is rooted from empirical conceptions as universal laws, evidence, objectivity, truth, actuality, deduction, reason, fact, and mathematical data, to name only a few. Validity in research is concerned with the accuracy and truthfulness of scientific findings.44 A valid study should demonstrate what actually exists and is accurate, and a valid instrument or measure should actually measure what it is supposed to measure.5,22,29,31,42,45

Novice researchers can become easily perplexed in attempting to understand the notion of validity in qualitative inquiry.44 There is a multiple array of terms similar to validity in the literature, which the authors equate to same such as authenticity, goodness, adequacy, trustworthiness, verisimilitude, credibility, and plausibility.1,45-51 Validity is not a single, fixed, or universal concept but rather a contingent construct, inescapably grounded in the processes and intentions of particular research methodologies.39 Some qualitative researchers have argued that the term validity is not applicable to qualitative research and have related it to terms such as quality, rigor, and trustworthiness.1,13,22,38,42,52-54 I argue that the concepts of reliability and validity are overarching constructs that can be appropriately used in both quantitative and qualitative methodologies. To validate means to investigate, to question, and to theorize, which are all activities to ensure rigor in a qualitative inquiry. For Leininger25 (1985), the term validity in a qualitative sense means gaining knowledge and understanding of the nature (ie, the meaning, attributes, and characteristics) of the phenomenon under study. A qualitative method seeks for a certain quality that is typical for a phenomenon or that makes the phenomenon different than others.

Some naturalistic inquirers agree that assuring validity is a process whereby ideals are sought through attention to specified criteria, and appropriate techniques are used to address any threats to validity of a naturalistic inquiry. However, other researchers argue that procedures and techniques are not an assurance of validity and will not necessarily produce sound data or credible conclusions.38,48,55 Thus, some argued that they should abandon the concept of validity and seek alternative criteria with which to judge their work. Criteria are the standards or rules to be upheld as ideals in qualitative research on which a judgment or decisions may be based,4,56 whereas the techniques are the methods used to diminish identified validity threats.56 Criteria, for some researchers, are used to test the quality of the research design, whereas for some, they are the goal of the study. There is also the trend to treat standards, goals, and criteria synonymously. I concur with Morse29 (1999) that introducing parallel terminology and criteria diminishes qualitative inquiry from mainstream science and scientific legitimacy. The development of alternative criteria compromises the issue of rigor. We must work to have a consensus toward criteria and terminology that are used in mainstream science and how it is attained within the qualitative inquiry during the research process rather than at the end of the study. Despite all these, researchers developed validity criteria and techniques during the years. A synthesis of validity criteria development is summarized in Supplemental Digital Content 3 (see Supplemental Digital Content 3, https://links.lww.com/DCCN/A20). The techniques for demonstrating validity are presented in Supplemental Digital Content 4 (see Supplemental Digital Content 4, https://links.lww.com/DCCN/A21).

Reliability and Validity as Means in Ensuring the Quality of Findings of a Phenomenological Study in Intensive Care Unit

Reliability and validity are 2 factors that any qualitative researcher should be concerned about while designing a study, analyzing results, and judging its quality. Just as the quantitative investigator must attend to the question of how external and internal validity, reliability, and objectivity will be provided for in the design, so must the naturalistic inquirer arrange for credibility, transferability, dependability, and confirmability.1 Lincoln and Guba1 (1985) clearly established these 4 criteria as benchmarks for quality based on the identification of 4 aspects of trustworthiness that are relevant to both quantitative and qualitative studies, which are truth value, applicability, consistency, and neutrality. Guba2 (1981) stated, “It is to these concerns that the criteria must speak.”

Rigor of a naturalistic inquiry such as phenomenology may be operationalized using the criteria of credibility, transferability, dependability, and confirmability. This phenomenological study aimed to understand and illuminate the meaning of the phenomenon of the lived experiences of patients, their family members, and the nurses during critical illness in the intensive care unit (ICU). From Lincoln and Guba1 (1985), I first asked the question, “How can I persuade my audience that the research findings of my inquiry are worth paying attention to, and worth taking account of?” My answers to these questions were based on the identified 4 criteria set forth by Lincoln and Guba1 (1985).

Credibility, the accurate and truthful depiction of a participant's lived experience, was achieved in this study through prolonged engagement and persistent observation to learn the context of the phenomenon in which it is embedded and to minimize distortions that might creep into the data. To achieve this, I spent 6 months with nurses, patients, and their families in the ICU to become oriented to the situation and also to build trust and rapport with the participants. Peer debriefing was conducted through meetings and discussions with an expert qualitative researcher to allow for questions and critique of field journals and research activities. Triangulation was achieved by cross-checking the data and interpretations within and across each category of participants by 2 qualitative researchers. Member checks were accomplished by constantly checking data and interpretations with the participants from which data were solicited.

Transferability was enhanced by using purposive sampling method and providing a thick description and a robust data with a wide possible range of information through the detailed and accurate descriptions of the patients, their family members, and the nurses' lived ICU experiences and by continuously returning to the texts. In this study, recruitment of participants and data collection continued until the data are saturated and complete and replicate. According to Morse et al4 (2002), interviewing additional participants is for the purpose of increasing the scope, adequacy, and appropriateness of the data. I immersed myself into the phenomenon to know, describe, and understand it fully, comprehensively, and thoroughly. Special care was given to the collection, identification, and analysis of all data pertinent to the study. The audiotaped data were meticulously transcribed by a professional transcriber for future scrutiny. During the analysis phase, every attempt was made to document all aspects of the analysis. Analysis in qualitative research refers to the categorization and ordering of information in such a way as to make sense of the data and to writing a final report that is true and accurate.36 Every effort was made to coordinate methodological and analytical materials. After I categorized and was able to make sense of the transcribed data, all efforts were exhausted to illuminate themes and descriptors as they emerge.

Lincoln and Guba1 (1985) use “dependability” in qualitative research, which closely corresponds to the notion of “reliability” in quantitative research. Dependability was achieved by having 2 expert qualitative nursing researchers review the transcribed material to validate the themes and descriptors identified. To be able to validate my findings related to the themes, a doctoral-prepared nursing colleague was asked to review some of the transcribed materials. Any new themes and descriptors illuminated by my colleague were acknowledged and considered. It was then compared with my own thematic analysis from the entire participant's transcribed data. If the theme identified by the colleague did not appear in my own thematic analysis, it was agreed by both analysts not to use the said theme. It was my goal that both analysts agree on the findings related to themes and meanings within the transcribed material.

Confirmability was met by maintaining a reflexive journal during the research process to keep notes and document introspections daily that would be beneficial and pertinent during the study. An audit trail also took place to examine the processes whereby data were collected and analyzed and interpretations were made. The audit trail took the form of documentation (the actual interview notes taken) and a running account of the process (my daily field journal). I maintained self-awareness of my role as the sole instrument of this study. After each interview, I retired in 1 private room to document additional perceptions and recollections from the interviews (Supplemental Digital Content 5, https://links.lww.com/DCCN/A22).

Through reflexivity and bracketing, I was always on guard of my own biases, assumptions, beliefs, and presuppositions that I might bring to the study but was also aware that complete reduction is not possible. Van Manen44 (1990) stated that “if we simply try to forget or ignore what we already know, we may find that the presuppositions persistently creep back into our reflections.” During data collection and analysis, I made my orientation and preunderstanding of critical illness and critical care explicit but held them deliberately at bay and bracketed them. Aside from Lincoln and Guba's1 (1985) 4 criteria for trustworthiness, a question arises as to the reliability of the researcher as the sole instrument of the study.

Reliability related to the researcher as the sole instrument who conducted the data collection and analysis is a limitation of any phenomenological study. The use of humans as instruments is not a new concept. Lincoln and Guba1 (1985) articulated that humans uniquely qualify as the instrument of choice for naturalistic inquiry. Some of the giants of conventional inquiry have recognized that humans can provide data very nearly as reliable as that produced by “more” objective means. These are formidable characteristics, but they are meaningless if the human instrument is not also trustworthy. However, no human instrument is expected to be perfect. Humans have flaws, and errors could be committed. When Lincoln and Guba1 (1985) asserted that qualitative methods come more easily to hand when the instrument is a human being, they mean that the human as instrument is inclined toward methods that are extensions of normal activities. They believe that the human will tend therefore toward interviewing, observing, mining available documents and records, taking account of nonverbal cues, and interpreting inadvertent unobtrusive measures. All of which are complex tasks. In addition, one would not expect an individual to function adequately as human instruments without an extensive background or training and experience. This study has reliability in that I have acquired knowledge and the required training for research at a doctoral level with the professional and expert guidance of a mentor. As Lincoln and Guba1 (1985) said, “Performance can be improved…when that learning is guided by an experienced mentor, remarkable improvements in human instrumental performance can be achieved.” Whereas reliability in quantitative research depends on instrument construction, in qualitative research, the researcher is the instrument of the study.31 A reliable research is a credible research. Credibility of a qualitative research depends on the ability and effort of the researcher.22 We have established that a study can be reliable without being valid, but a study cannot be valid without being reliable.

Establishing validity is a major challenge when a qualitative research project is based on a single, cross-sectional, unstructured interview as the basis for data collection. How do I make judgments about the validity of the data? In qualitative research, the validity of the findings is related to the careful recording and continual verification of the data that the researcher undertakes during the investigative practice. If the validity or trustworthiness can be maximized or tested, then more credible and defensible result may lead to generalizability as the structure for both doing and documenting high-quality qualitative research. Therefore, the quality of a research is related to generalizability of the result and thereby to the testing and increasing of the validity or trustworthiness of the research.

One potential threat to validity that researchers need to consider is researcher bias. Researcher bias is frequently an issue because qualitative research is open and less structured than quantitative research. This is because qualitative research tends to be exploratory. Researcher bias tends to result from selective observation and selective recording of information and from allowing one's personal views and perspectives to affect how data are interpreted and how the research is conducted. Therefore, it is very important that the researchers are aware of their own perceptions and opinions because they may taint their research findings and conclusions. I brought all past experiences and knowledge into the study but learned to set aside my own strongly held perceptions, preconceptions, and opinions. I truly listened to the participants to learn their stories, experiences, and meanings.

The key strategy used to understand researcher bias is called reflexivity. Reflexivity means that the researchers actively engage in critical self-reflection about their potential biases and predispositions that they bring to the qualitative study. Through reflexivity, researchers become more self-aware and monitor and attempt to control their biases. Phenomenological researchers can recognize that their interpretation is correct because the reflective process awakens an inner moral impulse.4,59 I did my best to be always on guard of my own biases, preconceptions, and assumptions that I might bring to this study. Bracketing was also applied.

Husserl60 (1931) has made some key conceptual elaborations, which led him to assert that an attempt to hold a previous belief about the phenomena under study in suspension to perceive it more clearly is needed in phenomenological research. This technique is called bracketing. Bracketing is another strategy used to control bias. Husserl60 (1931) explained further that phenomenological reduction is the process of defining the pure essence of a psychological phenomenon. Phenomenological reduction is a process whereby empirical subjectivity is suspended, so that pure consciousness may be defined in its essential and absolute “being.” This is accomplished by a method of bracketing empirical data away from consideration. Bracketing empirical data away from further investigation leaves pure consciousness, pure phenomena, and pure ego as the residue of phenomenological reduction. Husserl60 (1931) uses the term epoche (Greek word for “a cessation”) to refer to this suspension of judgment regarding the true nature of reality. Bracketed judgment is an epoche or suspension of inquiry, which places in brackets whatever facts belong to essential “being.”

Bracketing was conducted to separate the assumptions and biases from the essences and therefore achieve an understanding of the phenomenon as experienced by the participants of the study. The collected and analyzed data were presented to the participants, and they were asked whether the narrative is accurate and a true reflection of their experience. My interpretation and descriptions of the narratives were presented to the participants to achieve credibility. They were given the opportunity to review the transcripts and modify it if they wished to do so. As I was the one who served as the sole instrument in obtaining data for this phenomenological study, my goal was that my perceptions would reflect the participant's ICU experiences and that the participants would be able to see their lived experience through the researcher's eyes. Because qualitative research designs are flexible and emergent in nature, there will always be study limitations.

Awareness of the limitations of a research study is crucial for researchers. The purpose of this study was to understand the ICU experiences of patients, their family members, and the nurses during critical illness. One limitation of this phenomenological study as a naturalistic inquiry was the inability of the researcher to fully design and provide specific ideas needed for the study. According to Lincoln and Guba1 (1985), naturalistic studies are virtually impossible to design in any definitive way before the study is actually undertaken. The authors stated:

Designing a naturalistic study means something very different from the traditional notion of “design”—which as often as not meant the specification of a statistical design with its attendant field conditions and controls. Most of the requirements normally laid down for a design statement cannot be met by naturalists because the naturalistic inquiry is largely emergent.

Within the naturalistic paradigm, designs must be emergent rather than preordinate because (1) meaning is determined by context to such a great extent. For this particular study, the phenomenon and context were the experience of critical illness in the ICU; (2) the existence of multiple realities constrains the development of a design based on only 1 (the investigator's) construction; (3) what will be learned at a site is always dependent on the interaction between the investigator and the context, and the interaction is also not fully predictable; and (4) the nature of mutual shapings cannot be known until they are witnessed. These factors underscore the indeterminacy under which naturalistic inquirer functions. The design must therefore be “played by ear”; it must unfold, cascade, and emerge. It does not follow, however, that, because not all of the elements of the design can be prespecified in a naturalistic inquiry, none of them can. Design in the naturalistic sense means planning for certain broad contingencies without however indicating exactly what will be conducted on relation to each.1

DISCUSSION

Reliability and validity are such fundamental concepts that should be continually operationalized to meet the conditions of a qualitative inquiry. Morse et al4,29 (2002) articulated that “by refusing to acknowledge the centrality of reliability and validity in qualitative methods, qualitative methodologists have inadvertently fostered the default notion that qualitative research must therefore be unreliable and invalid, lacking in rigor, and unscientific.” Sparkes59 (2001) asserted that Morse et al4,26 (2002) is right in warning us that turning our backs on such fundamental concepts as validity could cost us dearly. This will in turn affect how we mentor novices, early career researchers, and doctoral students in their qualitative research works.

Reliability is inherently integrated and internally needed to attain validity.1,26 I concur with the use of the term rigor rather than trustworthiness in naturalistic studies. I have also discussed that I accede that strategies for ensuring rigor must be built into the qualitative research process per se rather than evaluated only after the inquiry is conducted. Threats to reliability and validity cannot be actively addressed by using standards and criteria applied at the end of the study. Ensuring rigor must be upheld by the researcher during the investigation rather than the external judges of the completed study. Whether a study is quantitative or qualitative, rigor is a desired goal that is met through the inclusion of different philosophical perspectives inherent in a qualitative inquiry and the strategies that are specific to each methodological approach including the verification techniques to be observed during the research process. It also involves the researcher's creativity, sensitivity, flexibility, and skill in using the verification strategies that determine the reliability and validity of the evolving study.

Some naturalistic inquirers agree that assuring validity is a process whereby ideals are sought through attention to specified criteria, and appropriate techniques are used to address any threats to validity of a naturalistic inquiry. However, other researchers argue that procedures and techniques are not an assurance of validity and will not necessarily produce sound data or credible conclusions.38,48,55 Thus, some argued that they should abandon the concept of validity and seek alternative criteria with which to judge their work.

Lincoln and Guba's1 (1985) standards of validity demonstrate the necessity and convenience of overarching principles to all qualitative research, yet there is a need for a reconceptualization of criteria of validity in qualitative research. The development of validity criteria in qualitative research poses theoretical issues, not simply technical problems.60 Whittemore et al58 (2001) explored the historical development of validity criteria in qualitative research and synthesized the findings that reflect a contemporary reconceptualization of the debate and dialogue that have ensued in the literature during the years. The authors further presented primary (credibility, authenticity, criticality, and integrity) and secondary (explicitness, vividness, creativity, thoroughness, congruence, and sensitivity) validity criteria to be used in the evaluative process.56 Before the work of Whittemore and colleagues,58 Creswell and Miller48 (2000) asserted that the constructivist lens and paradigm choice should guide validity evaluation and procedures from the perspective of the researcher (disconfirming evidence), the study participants (prolonged engagement in the field), and external reviewers/readers (thick, rich description). Morse et al4 in 2002 presented 6 major evaluation criteria for validity and asserted that they are congruent and are appropriate within the philosophy of the qualitative tradition. These 6 criteria are credibility, confirmability, meaning in context, recurrent patterning, saturation, and transferability. Synthesis of validity criteria is presented in Supplemental Digital Content 3 (see Supplemental Digital Content 3, https://links.lww.com/DCCN/A20).

Common validity techniques in qualitative research refer to design consideration, data generation, analytic procedures, and presentation.56 First is the design consideration. Developing a self-conscious design, the paradigm assumption, the purposeful choice of small sample of informants relevant to the study, and the use of inductive approach are some techniques to be considered. Purposive sampling enhances the transferability of the results. Interpretivist and constructivist inquiry follows an inductive approach that is flexible and emergent in design with some uncertainty and fluidity within the context of the phenomenon of interest56,58 and not based on a set of determinate rules.61 The researcher does not work with a priori theory; rather, these are expected to emerge from the inquiry. Data are analyzed inductively from specific, raw units of information to subsuming categories to define questions that can be followed up.1 Qualitative studies also follow a naturalistic and constructivist paradigm. Creswell and Miller48 (2000) suggest that the validity is affected by the researchers' perception of validity in the study and their choice of paradigm assumption. Determining fit of paradigm to focus is an essential aspect of a naturalistic inquiry.1 Paradigms rest on sets of beliefs called axioms.1 On the basis of the naturalistic axioms, the researcher should ask questions related to multiplicity or complex constructions of the phenomenon, the degree of investigator-phenomenon interaction and the indeterminacy it will introduce into the study, the degree of context dependence, whether values are likely to be crucial to the outcome, and the constraints that may be placed on the researcher by a variety of significant others.1

Validity during data generation is evaluated through the researcher's ability to articulate data collection decisions, demonstrate prolonged engagement and persistent observation, provide verbatim transcription, and achieve data saturation.56 Methods are means to collect evidence to support validity, and this refers to the data obtained by considering a context for a purpose. The human instrument operating in an indeterminate situation falls back on techniques such as interview, observation, unobtrusive measures, document and record analysis, and nonverbal cues.1 Others remarked that rejecting methods or technical procedures as assurance of truth, thus validity of a qualitative study, lies in the skills and sensitivities of the researchers and how they use themselves as a knower and an inquirer.57,62 The understanding of the phenomenon is valid if the participants are given the opportunity to speak freely according to their own knowledge structures and perceptions. Validity is therefore achieved when using the method of open-ended, unstructured interviews with strategically chosen participants.42 We also know that a thorough description of the entire research process enabling unconditional intersubjectivity is what indicates good quality when using a qualitative method. This enhances a clearer and better analysis of data.

Analytical procedures are vital in qualitative research.56 Not very much can be said about data analysis in advance of a qualitative study.1 Data analysis is not an inclusive phase that can be marked out as occurring at some singular time during the inquiry.1 It begins from the very first data collection to facilitate the emergent design and grounding of theory. Validity in a study thus is represented by truthfulness of findings after a careful analysis.56 Consequently, qualitative researchers seek to illuminate and extrapolate findings to similar situations.22,63 It is a fact that the interpretations of any given social phenomenon may reflect, in part, the biases and prejudices the interpreters bring to the task and the criteria and logic they follow in completing it.64 In any case, individuals will draw different conclusions to the debate surrounding validity and will make different judgments as a result.50 There is a wide array of analytic techniques that the qualitative researcher can choose from based on the contextual factors that will help contribute to the decision as to which technique will optimally reflect specific criteria of validity.65 Presentation of findings is accomplished by providing an audit trail and evidence that support interpretations, acknowledging the researcher's perspective and providing thick descriptions. Morse et al4 in 2002 set forth strategies for ensuring validity that include investigator responsiveness and verification through methodological coherence, theoretical sampling and sampling adequacy, an active analytic stance, and saturation. The authors further stated that “these strategies, when used appropriately, force the researcher to correct both the direction of the analysis and the development of the study as necessary, thus ensuring reliability and validity of the completed project (p17). Recently in 2015, Morse28 presented that the strategies for ensuring validity in a qualitative study are prolonged engagement, persistent observation, thick and rich description, negative case analysis, peer review or debriefing, clarifying researcher's bias, member checking, external audits, and triangulation. These strategies can be upheld with the help of an expert mentor who can in turn guide and affect the reliability and validity of early career researchers and doctoral students' qualitative research works. Techniques for demonstrating validity are summarized in Supplemental Digital Content 4 (see Supplemental Digital Content 4, https://links.lww.com/DCCN/A21).

Qualitative researchers and students alike must be proactive and take responsibility in ensuring the rigor of a research study. A lot of times, rigor is at the backseat in some researchers and doctoral students' work related to their novice abilities, lack of proper mentorship, and issues with time and funding. Students should conduct projects that are smaller in scope guided by an expert naturalistic inquirer to come up with the product with depth and, at the same time, gain the grounding experience necessary to become an excellent researcher. Attending to rigor throughout the research process will have important ramifications for qualitative inquiry.4,26

Qualitative research is not intended to be scary or beyond the grasp of novices and doctoral students. Conducting a naturalistic inquiry is an experience of exploration, discovery, description, and understanding of a phenomenon that transcends one's own research journey. Attending to the rigor of qualitative research is a vital part of the investigative process that offers critique and thus further development of the science.

References

1. Lincoln YS, Guba EG. Nat Inquiry. Thousand Oaks, CA: Sage Publications; 1991.

2. Guba EG. Criteria for assessing the trustworthiness of naturalistic inquiries. Educ Comm Tech J. 1981;29:75-91.

3. Guba EG, Lincoln YS. Epistemological and methodological bases of naturalistic inquiry. Educ Comm Tech J. 1982;30:233-252.

4. Morse JM, Barrett M, Mayan Olson, Spiers J. Verification strategies for establishing reliability and validity in qualitative research. Int J Qual Methods. 2002;1:13-22.

5. Brink HI. Validity and reliability in qualitative research. Curationis. 1993;16(2):35-38.

6. Tappen RM. Advanced Nursing Research: From Theory to Practice. Sudbury, MA: Jones & Bartlett Learning; 2011.

7. Hall HR, Rousel LA. Evidence-Based Practice: An Integrative Approach to Research, Administration and Practice. 2nd ed. Burlington, MA: Jones and Bartlett Learning; 2016.

8. Merriam-Webster Dictionary New Edition. Martinsburg, WV: Merriam-Webster; 2016.

9. Oxford English Dictionary. Oxford, UK: Oxford University Press; 2013.

10. Thomas E, Magilvy JK. Qualitative rigor or research validity in qualitative research. J Spec Ped Nurs. 2011;16(2):151-155.

11. Houser J. Nursing Research: Reading, Using, and Creating Evidence. 2nd ed. Burlington, MA: Jones and Bartlett Learning; 2013.

12. Sandelowski M, Barroso J. Reading qualitative studies. Int J Qual Methods. 2002;1:74-108.

13. Davies D, Dodd J. Qualitative research and the question of rigor. Qual Health Res. 2002;12:279-289.

14. Schmidt NA, Brown J. Evidence-Based Practice for Nurses: Appraisal and Application of Research. 3rd ed. Burlington, MA: Jones and Bartlett; 2015.

15. Rolfe G. Validity, trustworthiness and rigour: quality and the idea of qualitative research. J Adv Nurs. 2006;53:304-310.

16. Leininger MM. Evaluation criteria and critique of qualitative research studies. In: Morse JM, ed. Critical Issues in Qualitative Research Methods. 1st ed. London, UK: Sage; 1994:95-115.

17. Yin RK. Discovering the future of the case study method in evaluation research. Eval Pract. 1994;15:283-290.

18. Guba EG, Lincoln YS. Fourth Generation Evaluation. Newbury Park, CA: Sage; 1989.

19. Manning K. Authenticity in constructivist inquiry: methodological considerations without prescription. Qual Inquiry. 1997;3:93-115.

20. Seale C. Quality in qualitative research. Qual Inquiry. 1999;5(4):465-478.

21. Morse J. Qualitative Health Research: Creating a New Discipline. 1st ed. New York, NY: Routledge; 2012.

22. Golafshani N. Understanding reliability and validity in qualitative research. The Qual Rep. 2003;8:597-607.

23. Krefting L. Rigor in qualitative research: the assessment of trustworthiness. Am J Occup Ther. 1991;45:214-222.

24. Agar M. Speaking of Ethnography. Beverly Hills, CA: Sage; 1986.

25. Leininger MM. Nature, rationale and importance of qualitative research methods in nursing. In Leininger MM, ed. Qualitative Research Methods in Nursing. New York, NY: Grune & Stratton; 1985:1-28.

26. Smith J. The Nature of Social and Educational Inquiry: Empiricism Versus Interpretation. Norwood, NJ: Ablex; 1989.

27. Johnson M. Observations on positivism and pseudoscience in qualitative nursing research. J Adv Nurs. 1999;30(1):67-73.

28. Morse J. Critical analysis of strategies for determining rigor in qualitative inquiry. Qual Health Res. 2015;25:1212-1222.

29. Morse JM. Myth #93: reliability and validity are not relevant to qualitative inquiry. Qual Health Res. 1999;9:717.

30. Patton MQ. Qualitative Evaluation and Research Methods. 3rd ed. Thousand Oaks, CA: Sage Publications, Inc; 2002.

31. Merriam SB, Leahy B. Learning transfer: a review of the research in adult education and training. PAACE J Lifelong Learn. 2005;14:1-24.

32. Black JA, Champion DJ. Methods and Issues in Social Research. New York, NY: Wiley; 1976.

33. Hammersley M. Some notes on the terms “validity” and “reliability”. Brit Educ Res J. 1987;13:73-81.

34. Johnston JM, Pennypacker HS. Strategies and Tactics of Human Behavioural Research. Hillsdale, NJ: Lawrence Erlbaum Associates; 1980.

35. Lehner PN. Handbook of Ethological Methods. New York, NY: Garland, STPM Press; 1979.

36. Merriam S. What can you tell from an N of 1? Issues of validity and reliability in qualitative research. PAACE J Lifelong Learn. 1995;4:51-60.

37. Winter G. A comparative discussion of the notion of validity in qualitative and quantitative research. The Qual Rep. 2000;4:1-14.

38. Campbell DT, Fisk DW. Convergent and discriminant validation by the multitrait-multimethod matrix. Psychol Bull. 1959;56(2):81-105.

39. Kerlinger F. Foundations of Behavioural Research. New York, NY: Holt; 1964.

40. Winter G. A comparative discussion of the notion of validity in qualitative and quantitative research. The Qualitative Report. 2000;4:3-4.

41. Simco N, Warin J. Validity in image based research: an elaborated illustration of the issues. Brit Educ Res J. 1997;23:661-673.

42. Morse J. Qualitative nursing research: a free for all? In Morse J, ed. Qualitative Nursing Research: A Contemporary Dialogue. 2nd ed. Newbury Park, CA: Sage; 1991:14-22.

43. Creswell JW. Qualitative Inquiry and Research Design: Choosing Among Five Approaches. 2nd ed. Thousand Oaks, CA: Sage Publications; 2007.

44. Van Manen M. Researching Lived Experience: Human Science for an Action Sensitive Pedagogy. Albany, NY: State University of New York Press; 1990.

45. Le Compte MD, Goetz JP. Problems of reliability and validity in ethnographic research. Rev Educ Res. 1982;52:31-60.

46. Lub V. Validity in qualitative evaluation: linking purposes, paradigms, and perspectives. Int J Qual Methods. 2015;14:1-8.

47. Joppe M. The research process. The Quant Rep J. 2000;4:597-607.

48. Creswell JW, Miller DL. Determining validity in qualitative inquiry. Theory Into Pract. 2000;39:124-131.

49. Hammersley M. Reading Ethnographic Research: A Critical Guide. London, UK: Longmans; 1990.

50. Altheide DL, Johnson JM. Criteria for Assessing Interpretive Validity in Qualitative Research. Thousand Oaks, CA: Sage Publications; 1994.

51. Lather P. Fertile obsession: validity after poststructuralism. The Sociol Q. 1993;34:673-693.

52. Stenbacka C. Qualitative research requires quality concepts of its own. Manage Decision. 2001;39:551-555.

53. Maxwell JA. Understanding and validity in qualitative research. Harvard Educ Rev. 1992;62:279-299.

54. Maxwell JA. Qualitative Research Design: An Interactive Approach. Thousand Oaks, CA: Sage; 1996.

55. Merriam SB. Qualitative Research and Case Study Applications in Education. San Francisco, CA: Jossey-Bass; 1998.

56. Sandelowski M. Rigor or rigor mortis: the problem of rigor in qualitative research revisited. ANS Adv Nurs Sci. 1993;16(2):1-8.

57. Smith JK. Alternative research paradigms and the problem of criteria. In Guba EG, ed. The Paradigm Dialog. Newbury Park, CA: Sage; 1990:167-187.

58. Whittemore R, Chase SK, Mandle CL. Validity in qualitative research. Qual Health Res. 2001;11(4):522-537.

59. Schwandt TA. Farewell to Criteriology. Paper presented to the Quality in Human Inquiry Conference, University of Bath, UK: 1995.

60. Husserl E. Ideas: General introduction to pure phenomenology. (Trans. Gibson WR). New South Wales, AUS: George Allen & Unwin/McMillan; 1931.

61. Moskal B, Leydens J, Pavelich M. Validity, reliability and the assessment of engineering education. J Eng Educ. 2002;91:351-354.

62. Phillips DC. Validity in qualitative research: why the worry about warrant will not wane. Educ Urban Soc. 1987;20:9-24.

63. Mishler EG. Validation in inquiry-guided research: the role of exemplars in narrative studies. In Brizuela BM, Stewart JP, Carrillo RG, Berger JG, eds. Acts of Inquiry in Qualitative Research. Cambridge, MA: Harvard Educational Review; 2000:119-146.

64. Mishler EG. Validation in inquiry-guided research: the role of exemplars in narrative studies. Harvard Educ Rev. 1990;60:415-442.

65. Reason P. Issues of validity in new paradigm research. In Reason P, Rowan J, eds. Human Inquiry. New York, NY: Wiley; 1991:239-250.

Supplemental Digital Content