Evaluating goodness in qualitative researcher

This article explores the issue of “evaluating goodness in qualitative research”, from a perspective of novice qualitative researchers. Despite the recent upsurge in publications of qualitative studies especially in health sciences, the issue of goodness in qualitative research is still debatable. Qualitative researches in contrast to traditional research not only differ in research methodology and methods but also in data analysis. Although approaches for evaluating goodness in qualitative research are available but consensus on universality is still lacking. The development of extrinsic criteria although provide the guidelines for post positivist studies, however, it is not acceptable to interpretivist /constructivist who believe on multiple realities and knowledge as co construct. The authenticity criteria although fits well to constructivism/interpretivism, however, researchers argue that because it provide a post hoc strategy for evaluation of a study and avoid focusing during its conduct, thus causing serious threats to the credibility. Primary criteria forwarded by Whitemoore et al., although seems essential for all qualitative inquiry but because based on validity has been rejected by authors on the grounds that qualitative epistemological and ontological assumptions are entirely different to the traditional qualitative research. The criteria by Ballinger in 2006 although seems practical in application to all paradigms, however, as it also questions reflexivity which seems irrelevant in realist tradition. Further other general criteria such as seems popular because of its simplistic approach but do not address the terms of ontology, epistemology and paradigm that seem very important in qualitative research. It is important for the novice qualitative researchers to be aware of the debate on the issue of evaluating goodness of qualitative research. However, they should adopt a cautious stand while favouring or rejecting one criteria. Finally the development of a universal and uniform criteria is although important but not necessary requirement for the qualitative


Background
My issue paper is on "goodness of qualitative research".It starts with my paradigmatic location, background and the purpose of writing this paper.I will initially discuss the qualitative research, this include a discussion on the differences between qualitative and quantitative research moving onto why one should focus on evaluation of the qualitative research?how goodness can be evaluated and maintained in the qualitative research process.This will be followed by discussion on different categories of approaches and criteria used in qualitative research evaluation along with a thorough discussion on the examples of each category.I would also discuss the applicability of universal criteria in qualitative research, and conclusion and recommendations based on my discussion.

Paradigmatic Location, Background and Purpose
The purpose of writing this paper is that I would like to discuss the difficulty of evaluating goodness/quality in qualitative research, looking from the perspective of a qualitative researcher; what a young researcher feels about the issue, how tricky but still essential to have an information on this issue.A qualitative researcher once involved should have the knowledge and understanding of the issues of credibility, validity, rigor or authenticity in the qualitative research.My paradigmatic location is not firm but rather dynamic moving back and forth between the paradigms of post positivism and constructivism.On one hand I believe that reality is assumed to exist but is only imperfectly apprehendable, but I also believe that realities are apprehendable in the form of multiple, intangible mental constructions that are socially based and specific in nature.My epistemological believe moves from believe in objectivity but abandoning dualism as impossible to maintain, to findings that are literally created between investigator and those investigated and that are assumed to be interactively linked.My methodological emphasis varied form "critical multiplism" as a way of falsifying hypothesis, to that individual constructions can be elicited and refined only through interaction between and among investigator and respondents.The example of my post positivist thinking is, I believe that while performing a Randomized Controlled Trial between two drugs for reducing the incidence of Stroke in patients with Ischemic Heart Disease, definitely one drug is superior in reducing the events of Stroke or life and death as all these events are real, and there is one reality in the case of better drug and event.But I also believe that if I wanted to measure happiness or measure health in a group of people, then there will be no such reality and the meanings and understanding of the people would be historical and socially based and specific.

Qualitative Research
According to Denzin and Lincoln 3 the qualitative research is multi-method in focus, and involves an interpretive and naturalistic approach to its subject matter.The focus of qualitative research is to understand and represent the experiences and actions of the people as they encounter, engage and live through situations and where the researcher attempts to understand the phenomena from the prospective of those being studied 4 .The qualitative research involves the use of case study, personal experience, narratives, and historical and visual texts that describe the normal routine and problematic experiences and meaning that are important in individuals' lives 3 .

Qualitative and Quantitative Research
The qualitative research has now started to become popular in Health Sciences.The qualitative research is mostly narrative depending on the research question, school of inquiry, reflexivity, and data collection and interpretation.It cannot be assessed by means of p values, confidence intervals, effect sizes, minimally clinical important difference and in terms of NNTs (numbers needed to treat) which are used as measures of credibility and quality in quantitative research.Elliot 5 argues that qualitative research is concerned with understanding participant's perspectives, to understand phenomena in terms of participants experienced meanings and to develop theory from field work.According to Merriam 6 the main differences between qualitative and quantitative research using ten points of comparisons including focus of research, philosophical roots, associated phrases, goals of investigation, design characteristics, setting, sample, data collection, mode of analysis and findings.

Importance of Evaluating Good Quality Research
According to Schwandt 7 the criteria for qualitative studies should be based on standards, benchmarks and regulative ideals, that guides our judgments on the quality of inquiry processes and findings.Morse et al. 8 have expressed criteria in terms of rigor and stated that without rigor, there is a danger that research may only become as fictional journalism and not as Science.The rigor is the means by which one can demonstrate integrity, competence and the legitimacy of the research process.
In short the quality criteria are the standards that are formulated to assess the quality of the study whereas the techniques are methods that are applied to maintain rigorness in the study.However, several other authors [9][10][11] have all challenged the concept of rigor, arguing that by its nature it is an empirical analytical term and therefore does not fit into an interpretive approach.
The assessment in qualitative research is, however, difficult because the qualitative research is itself fuzzy, fluid and uncertain 12 .The qualitative study has different paradigms and schools of inquiries based on differences in ontology and epistemology making it difficult to create universal criteria.Therefore according to Morrow & Smith 13 the goodness of qualitative research should be assessed based on the paradigmatic underpinnings.However, according to Sandelowski and Barroso's 14 any research should be judged on its individual merits meaning that the utmost responsibility lies with the author who himself should provide explicit criteria for evaluation.It seems more logical as readers and qualitative researchers have their own schools of thoughts and pre suppositions that could be different from the author, thus presence of a guideline in ones research could provide a uniform assessment.
It is interesting to note that despite enrich history and contribution of qualitative research, its opponents still renders it as radical and non rigorous 11 .Interestingly this opposition also includes some of the qualitative authors 15,16 .According to McCracken 17 this issue arose as researchers moved from a detached outsider position to that of integrated insider that is the researchers change from using the instrument to become as the instrument themselves.
The literature review has showed that few of the authors 1,18-20 have adopted the approach to reject anything that might link qualitative inquiry to the positivist quantitative approach; leading to rejection of the terms validity and reliability according to these authors.However few argued that this outright criticism 21 may result in qualitative research being rejected as a science.
Mays & Pops 22 have suggested that the qualitative research can be assessed with same reference of standards that is validity and relevance other and quality criterias like quantitative research, but in a different way.But one must acknowledge that these quality criterias are not straightforward and requires subjective judgments, which itself is a source of bias in quantitative research.Despite these concerns different procedures and methods such as triangulation, respondent validation, clear detailing of methods of data collection and analysis, reflexivity and fair dealing, detailed reports and sampling techniques are all means to improve validation of the qualitative study 22 .

Approaches and Criteria in Qualitative Research
The approaches to appraising the qualitative research can be broadly categorized into parallel criteria which include trustworthiness 23 .The paradigmspecific/intrinsic criteria, examples include parallel criteria, Malterud 24 criteria, authenticity criteria and Arminio and Hultgren criteria 25 .The going beyond paradigm example includes Whittemore et al. 1 , Popay et al. 26 criteria and Giacomini& Cook 27 criteria.The last category is general considerations criteria which include Ballinger 2 criteria and Britten, et al. 28 criteria.However, only few of these criteria's are explained below.

Parallel Criteria
Lincoln and Gubahave 29 suggested a parallel criteria/criterion to that of quantitative research in terms of validity and reliability.The term credibility (vs.internal validity) refers to developing internal consistency and showing the readers the way by which rigor is maintained in the research 30 .The credibility in the qualitative research can be produced by prolong engagement, reflexivity and participant checklists.The transferability (vs.generalizability) deals with generalizability of current findings to the similar context and settings.The transferability in the qualitative research can be enhanced by providing researchers (authors) position and clear description of context, participants, selection and methods.The dependability (vs.reliability) deals that the study conducted "should be consistent across time, researchers and analysis techniques" 30 .The dependability in the qualitative study can be maintained by audit trial and discussion with peer researchers.
The final criteria isconfirmability (vs.objectivity) that addresses that the researcher should focus on the situation and beliefs of those that are being researched rather than his pre supposition and beliefs.The confirmability in the qualitative study is also maintained by audit trial.
The development of extrinsic criteria has allowed both the researchers and evaluators of journals and institutional grants board to review the post positivist studies on these guidelines.However on applying it to other paradigms can create issues where paradigms like interpretivist/constructivist depend on multiple realities and knowledge as co constructed between researcher and participants.

Paradigm specific criteria
Post positivist quality criteria: One of the examples of quality criteria in the post positivist qualitative study is Lincoln and Guba 23 parallel criteria.Another example of post positivist quality criteria is Malterud 24 criteria.The criteria demands that the research question should be relevant, explicit and focused to the aim of the study.The authors should clearly provide their background, perspectives and motives of the study; they should also provide how these issues are being dealt with throughout the research process.The selected methods should be justifiable to the aim of the study and research questions.The sampling strategy should be clear, justifiable and congruent to the research question.This can be done by providing sample characteristics and the context and setting of the study.The authors should provide an adequate theoretical frame work of the study, which should be inline with the aim of the research 24 .
The analysis section of the research should clearly provide the procedures and principles of data analysis.The derivation of codes and categories should be explained that is whether they were derived from pre conceptions in advance or driven from data.The researchers should also provide the means of validation and rigor in the study before the interpretation of findings.The findings of the study should be relevant to the aim of the study and should provide a new insight.The presentation of findings should be supported by participants quotes so that it will ensure that the findings are data driven and not from the researcher preconceptions.The study should also addresses the internal and external validity along with reflexivity.The study limitations should be discussed along with appropriate choices, the findings should be compared across different studies and finally a conclusion should be provided.The presentation of the study should be in clear contextual manner and it should be clearly written in the study whether the voices of the researcher and participants were similar or different 24 .
Although the criteria fits well to the post positivist approach focusing on rigorous methods of research sampling strategies, explicit research question, data driven findings, means of validation and rigor and internal and external validity.However, they do not addresses that how the part of measures like reflexivity which is common in costructivists, can be fitted to this post positivistic approach.

Criteria
for Trustworthiness in Constructivist/Interpretivist: The example of constructivist quality criteria is authenticity criteria 31 named as intrinsic by Lincoln 12 .It include fairness which demands that different constructions be solicited and honored.The ontological authenticity in which participants meaning are elaborated.The educative authenticity in which participants understandings and indebtedness is enhanced.The catalytic authenticity deals with extent to which actions are stimulated and tactical authenticity.Patton 32 has suggested dependability and triangulation as important constituents.While moving a step forward, Whittemore at al. 1 advocated understanding and analyzing participants meaning.Further, researcher's reflexivity provides an opportunity for the researcher to understand how his own experiences and understandings of the world affect the research process.
The fundamental problem with the trustworthiness and authenticity criteria is that they provide a post hoc strategy for evaluation of a study after the study is completed and that affects the validity and reliability of the study.Gallagher 33 , Silverman 34 and Smith 35 all have argued appropriateness of procedures such as member or dependability checks.Its ontological realism and epistemological constructivism could lead to which may result in fragmented research 36 .Finally Baker et al. 37 warns against this type of method slurring that could lead to lack of rigor.
The main problems in paradigm specific quality criteria are that they do not provide how one can determine what paradigm has been used in the particular study and whose responsibility is it to identify the paradigmatic location of the qualitative study.This means whether the author himself declares the paradigmatic location of himself in either the background or explicit in methodology section.Most of the times authors of the qualitative study do not clearly state their paradigmatic location, which leads the readers searching out the whole study report, collecting peaces of information and trying to determine the authors location.

Primary criteria and Secondary Criteria:
The primary and secondary criteria are forwarded by Whittemore et al. 1 .The primary criteria includes credibility, authenticity, criticality, and integrity whereas secondary criteria includes explicitness, vividness, creativity, thoroughness, congruence, and sensitivity are considered secondary criteria.Whittemore et al. 1 although himself re iterates the importance of primary criteria as important but also acknowledges that they are insufficient in and of themselves.The primary criteria consist of credibility, authenticity, criticality and integrity.The credibility is the means of developing internal consistency and accurate interpretation of data 38 .According to Janesick 39 and Thorne 40 the interpretation of findings must conjoint with the context of the research.The authenticity focuses at the accurate level of reflection of the participants meaning and experiences and simultaneously incorporating researchers goals 41,42 .The criticality and integrity are achieved by reflexivity, critical analysis, repetitive checks of interpretations and humble that could contribute validity in qualitative research 40,43,44 .
In secondary criteria the explicitness means auditability which means that the results should support the conclusions of the researcher 44 .The vividness is the presentation of thick and faithful descriptions 45 with artfulness and clarity.The creativity in a qualitative study is achieved by building novel methodological designs, data presenting and analysis to answer specific research questions within scientific process 40,[46][47][48] .The thoroughness refers to adequate sampling (saturation) and complete, consistent and comprehensive analysis 20,49 .The congruence is established by coherence between the research question, data collection and analysis along with the philosophical or methodological perspective 43,50 .Lastly, sensitivity refers to research that is implemented in ways that are sensitive to the nature of human, cultural, and social contexts 51 .
The argument against the primary criteria (based on validity) is that because qualitative research is based on entirely different epistemological and ontological assumptions as compared to the traditional quantitative research, therefore the term validity is inappropriate 52 .Similarly Leininger 20 contended that quantitative validity criteria applied to qualitative research are awkward, confounding, and confusing.Further, Kahn 53 espoused that the positivistic perspective on validity obscures the differing validity threats in interpretive research and ultimately leads to a "procedural charade."

General Considerations Criteria
Ballinger 2 forwarded a quality frame work that consists of four points.The first consideration is coherence between research question, aims, position of the researcher and the methodological approach.An example of a lack of coherence can be when in participatory research, researcher asked his other fellow researcher for reliability rather using participants themselves.The second point is researcher should focus on systematic and careful research process.It can be done when the researcher demonstrates how he has approached the participants and selected the desired sample.The third approach is to evaluate credibility that is convincing and relevant interpretation.According to Ballinger 2 this means that research adds significantly to the field of knowledge.Although it seems difficult to evaluate credibility in research, however, this can be achieved by presenting the findings to different mediums and readers 54,55 .Credibility could also be enhanced in the form of abstract presentations and discussion with colleagues including experts.Finally the fourth criteria is assessment of researchers role consistent with the paradigms of the research that is reflexivity which is a prerequisite for all types of research.The main problem in Ballinger 2 criteria is as in the realist tradition where researcher is not considered as an instrument and remains impartial and objective it seems illogical to question reflexivity.The credibility criteria is also in adequate, as Ballinger 2 himself pointed out that it is difficult to measure credibility and although he provides means of achieving credibility like presentation of findings to different medium; but he did not provide the exact criteria that can be looked for in respect to credibility in a qualitative study.

Applicability of Universal Criteria
The above discussion has highlighted the complexity of the issue.I firmly believes that the responsibility of quality, goodness and rigor lies in the hands of the researcher.The researchers search out the issue, select the appropriate question, develop a methodology, apply methods, involved in interpretation and analysis and comes out with the conclusions.It is them that have interest in research than any of us it is them who have spent more time than us.They have more knowledge on that aspect and therefore, it is them who should determine the standards and develop the references for the quality criteria.The reviewers having different interests, different views, even different paradigm specifications, how it be fare that they can judge the quality of the research on their standards of quality.
If one thinks as post positivist, the other true constructivist, if one is positivist, the other has preference for critical theory.All thinking and answering the same question but coming up with different perspectives, showing me different angles.How can one create a single criteria and stop them from nurturing in their respective paradigms?why one wanted to develop a universal criteria based on checklist which will stop them from growing in their respective paradigms.By trying to fit them, by trying to evaluate their thinking and then research, one would loose the novelty in these schools of inquiry and paradigms.If one believes in a single reality, or multiple reality, if some sees researcher role as objective and some as subjective, if some have pre specific hypothesis and some try to falsify it, I don't think as important.They all are correct in their believes.In short, I believe that although a universal criteria is important but not necessary and the qualitative research should progress despite the issues surrounding evaluation of goodness.
My stance on the issue of goodness of qualitative research was and still is cautious, where I believe that although universal criteria is important but not necessary.It is important that all the qualitative researches should be judged and evaluated on one single criteria, where a uniform approach can be utilized for the competing purposes of different researches.But I also believe that due to above issues that I discussed, it should not be necessary to have a uniform single criteria and people can use different criteria for the evaluation of their research.The responsibility of providing the quality criteria, however, lies to the investigators and authors of the study.But at the same time the researcher selected quality standards should be relevant and appropriate to their research and explicit in nature.

Conclusion and Recommendations
The parallel criteria, paradigm specific criteria and mix of general and paradigm specific criteria do not provide universal criteria for evaluation of qualitative research which the general criteria provide.These criteria are specific critiquing studies only that follow their criteria and components like parallel criteria could only be useful for post positivist, authenticity criteria for constructivist.The argumentation against Whittemore et al 1 criteria is validity which according to few authors 52 is inappropriate in the qualitative research.Although general criteria like Ballinger seems more feasible and simplistic as going beyond the paradigmatic approach but considers the term reflexivity which is irrelevant in realist tradition.A single universal criteria is important but not necessary and till consensus is achieved on a single criteria the best practical solution is that the authors should themselves provide the best quality criteria that closely fits to their study.______________