More discussions on criteria for assessing research... Hammersley

... and more week 1 reading. I think even I am beginning to see the theme in this early reading.

Hammersley, M. (1992). what's wrong with ethnography? London, United Kingdom: Routledge. (Chapter 4: By what criteria should ethnographic research be judged?)

Hammersley introduces the main issue with ethnography in social sciences which is, which criteria should be used to assess ethnographic studies. He suggests that there are three basic position:

1. we should use the same criteria as are used by quantitative researchers. In this way, ethnography is one of many methods and is not associated with a distinctive methodological philosophy.
2. Most believe that ethnography is within an alternative paradigm to positivism - a methodology which is more appropriate to the study of the social world than the quantitative methodology of the physical sciences. Therefore the criteria for judging ethnography will differ to those used by quantitative researchers. There is little agreement as to what "appropriate criteria" are.
3. Some researchers believe that there can be no criteria that can be used to judge qualitative research - assessing research through criteria is incompatible with the nature of the social world and how we understand it.

Hammersley starts by looking at viewpoint three:

The rejection of criteria

Smith (1984) believes that establishing 'non-arbitrary' criteria for ethnographic research is marked by inconsistency and confusion. With the qualitative tradition being 'idealist', the definition of 'validity' is one of agreement on an interpretation. Because we're unable to have certain knowledge of an independent truth the correspondence theory of reality is not appropriate. This undermines the idea that there exist criteria by which we should judge research. "To accept that social reality is mind-constructed and that there are multiple realities is to deny that there are any 'givens' upon which to found knowledge. If one accepts these assumptions, different claims about reality result not from incorrect procedures but may simply be a case of one investigator's interpretation of reality against another's."

Hammersley unpicks Smith' argument:

1. The claim that assessment of ethnographic research requires judgement, so there is always the potential for disagreement about the application of proposed criteria;
2. The argument that there are no criteria whose validity is certain, so that assessing claims leads to the possibility of disagreement and potential change;
3. The suggestion that there are no criteria in the sense of judgements that assume the reality of the phenomena studied to be independent of the researcher. Instead, the considerations in terms of which assessment is and should be made just refer to very particular, historically-located forms of social practice. The only legitimacy is that they belong to such a practice and that they are agreed on by those who engage in that practice.

Hammersley agrees with Smith's first two points - there are no criteria whose validity is a given. Applying criteria requires JUDGEMENT and the criteria are open to challenge and reformulation. Hammersley doesn't accept Smith's third argument. He disagrees with Smith's anti-realist position of rejecting that our knowledge can correspond to the phenomena it is intended to represent and that achieving such correspondence is one of the aims of inquiry: Hammersley develops his refutation of Smith's final point by affirming that we have no way of determining the certainty of our understanding of the world, but that this does not necessarily mean the rejection of the truth as correspondence.

This lack of certainty also fails to undermine the idea that some methods are more effective than others in producing knowledge of reality. It similarly doesn't counter the thought that there are criteria that we can use to judge empirical claims. All of these suppositions only follow (as per Smith's argument) if, when we discuss knowledge, it can only be known with ABSOLUTE CERTAINTY, that to be of any value, the methods used must ENSURE true findings and that the criteria must produce assessments that are BEYOND ALL POSSIBLE DOUBT.

But these assumptions are effectively irrelevant - Smith suggests you can only have realism (naïve realism) or idealism. However, there is a wider range of positions than this polar view. Many of these positions are philosophically defensible even though none can be established beyond all doubt.

Smith defines truth (within the interpretive tradition) as 'what is agreed upon' ; Hammersley states that this inhibits any rational discussion. He also believes that it is logically impossible and subject to a circular definition. He also suggests that Smith falsely suggests that choice of paradigm is down to personal preference - quantitative paradigms have criteria, but if you prefer the interpretive paradigm, you cannot have foundational criteria. Hammersley suggests that criteria for assessment should be applied but should be HEURISTIC and subject to debate.

Ethnographic criteria

Some researchers (e.g. Lofland & Lofland, 1984) suggest criteria for assessment, for the 'generic' style of ethnography. He suggests looking for several of the following features:

• a GENERIC conceptual framework, applicable to a wide range of social phenomena. Can the research be applied to other areas/relationships or is it specific to one location/social phenomenon?
• Is the framework NOVEL?
• Is the framework ELABORATED? Is it more than a skeleton frame? rather, does it draw out implications and show major variations?
• Is the framework 'EVENTFUL'? Is it illustrated by a richness of events and concrete episodes?
• The framework should be interpenetrated with the empirical materials - i.e. both theoretical framework and the empirical account enrich each other.

This generic style differs from very specific ethnographic studies which do not produce generalisations, or contain generalised description rather than detailed empirical data.

Athens (1984) produced a similar set of criteria for assessment:

• Does it generate formal theory? refining existing theory is less valuable. Formal theory relates to Lofland's 'generic' theory.
• Is it empirically grounded? Are the theoretical concepts consistent with empirical observations from which they are derived? Observations need to be included in the report so the reader can make this assessment.
• Is it scientifically credible? Is an account of the research process included?

Only this final criterion seems distinct from Lofland's.

Hammersley discusses another attempt to specify criteria for judging ethnographic research, that of Guba and Lincoln (1981 and 1985). They believe that the NATURALISTIC approach is needed for studying human social life. They believe that the four major traditional criteria (truth value, applicability, consistency and neutrality) can be used but must be formulated in a way that differs from the rationalist paradigm, as follows:

1. Truth value: concerned with credibility; do the people studied find the account produced to be true?
2. Applicability is related to transferability; although naturalists reject 'generalisability', they believe that there can be some transferability, if enough 'thick descriptive' can be produced.
3. Consistency equates to dependability; because of the emergent nature of research design in naturalistic research, replication is impossible.
4. Neutrality takes the form of 'confirmability'; is the analysis 'grounded in the data'? Are any inferences based on the data logical and of high utility?

This viewpoint adds to Lofland and to Athens. Hammersley synthesises these to bring together a new list:

1. the degree to which generic/formal theory is generated;
2. the degree of development of the theory;
3. the novelty of the claims made;
4. the consistency of the claims with empirical observation and the inclusion of representative examples of the latter in the report;
5. the credibility of the account to readers and/or to those studied;
6. the extent to which the findings are transferable to other settings;
7. the reflexivity of the account: the degree to which the effects of the findings of the researcher and of the research strategies employed are assessed and/or the amount of information about the research process that is provided to readers.

Hammersley suggests that there are questions that need to be asked about these criteria:

1. are they applied to all ethnographic research? For example, should all ethnographic research be concerned with developing formal theory? *Er, No!* We need to modify the application of criteria according to the intended product of the research.
2. Credibility shouldn't be define as whether readers or people studied judge the account to be true. The researched may not want to  acknowledge the truth. The studied do not have privileged access to the truth. The responses to the ethnographic accounts are useful data sources but agreement with these accounts should not be a criterion for assessing research. If it were, bland, agreeable results would be produced.

Applying quantitative criteria

Evaluation for quantitative methods centre on:

• internal and external validity
• reliability and validity.

The first subsumes the second.

Hammersley suggests that application of these criteria to ethnography is unsound, even to quantitative research. Campbell's development of the concepts of internal and external validity takes the 'quasi-experiment' as its research model, but it's been used for other research. The fcus is on designing research to rule out threats to validity.

INTERNAL VALIDITY: whether, in the specific experiment, manipulation of the treatmentproduced variation in the outcome.

EXTERNAL VALIDITY: whether a relationship discovered in an experiment can be generalised to other situations.

You can have internal validity without external validity.

Hammersley believes that this talk of different types of validity is misleading. The findings of a study are either valid or not - they can't be valid in one sense but not in another.

The other framework from quantitative methodology that is often applied to ethnography is the distinction between reliability and validity. There are ambiguities in the definitions.

VALIDITY: Accuracy with which a description of particular events represents the theoretical category that it is intended to represent and captures the relevant features of those events.

RELIABILITY: The degree of consistency with which instances are assigned to the same category by different observers or the same observer on different occasions. It provides evidence about validity and tells us about the usefulness of the particular strategy used.

Validity and reliability are appropriate for judging methods and products of research, but they are not sufficient. We need to consider alternatives for internal/external validity and the question of relevance.

A reformulation of the criteria

Some of what Guba and Lincoln identify as criteria are means by which the validity of qualitative research may be assessed, rather than criteria themselves - definitely 'credibility', 'consistency', 'neutrality' and possibly 'transferability'.

Hammersley concentrates on criteria of assessment and standards by which research results should be assessed. we need to first consider: what is the PURPOSE of such research? What goal is it intended to serve? Hammersley suggests that research is "to provide information and relevant to some legitimate public concern". Based on this definition, we need to judge on two criteria: TRUTH (validity) and RELEVANCE. These apply to both qualitative and quantitative research.

The importance of validity is obvious, if not unproblematic. However, relevance is important, including it being of importance to those outside the research community. Sound research requires the criteria of validity and relevance.

Validity

Hammersley suggests that he is a 'subtle' realist, using validity as a synonym for 'truth'. "An account is valid/true if it accurately represents those features of the phenomena that it is intended to describe, explain or theorise". This assumes a correspondence theory of truth. Since we cannot know the extent to which an account is true, we have to judge validity on the basis of the adequacy of evidence used to support them. We must recognise that judgements on the truth of knowledge claims rely on assumptions, many of which we are not consciously aware of, and many of which have not been subjected to rigorous testing.

We must be aware that potentially we need to keep challenging each new piece of evidence and so on ad infinitum, so where does this stop? we make this sort of judgement on a daily basis in real life, taking into account such considerations as: judgements about what is beyond reasonable doubt; likely costs of error; the scope for acquiring further evidence before a decision has to be made; the likely value of that evidence; the costs of getting this evidence; and so on...

Limits on time and resources as well as an understanding of 'beyond reasonable doubt' will set the bar for sufficiency of evidence. Hammersley suggests there are three important considerations with regard to sufficiency of evidence:

1. Issues of plausibility/credibility: are the claims made sufficiently plausible, given our existing knowledge? If so, accept them. If not, is the claim reasonably likely to be accurate (i.e. is it credible), given what we know about the circumstances of the research? researchers must ensure their findings are sufficiently plausible/credible including through anticipating the likely judgements of fellow researchers. They must provide sufficient evidence to counter any concerns over the research.
2. The evidence should relate to the level of centrality of the claim to the researcher's argument. A claim that is central to the researcher's argument will require more convincing evidence than a more marginal claim.
3. The type of claim made: we need to distinguish between: definitions; descriptions (claims about what happened at a particular time/location; explanations; theory (relationships between different types of phenomena, wherever instances of those types occur). The validity of claims for supporting theories is more complex than that of descriptions, therefore needs more evidence, via descriptions of a range of examples. We'd also need evidence to show any confounding factors could not cause the link. This requires the use of a variety of sources of information.

Hammersley suggests that the assessment of validity involves identifying the main claims made by a study, noting the types of claims these represent, then comparing the evidence provided with each claim with what is judged to be necessary, given the claim's plausibility and credibility.

Relevance

Work should not only "just" provide 'truth' but also be judged in terms of its relevance. But, relevant to what/whom? - A limted audience, generally. We need to consider the 'audience': bear in mind the researcher's own specialisations of topic but also different methodologies and theories. Hammersley believes that the primary emphasis should be on substantive relevance.

Relevance to researchers can be considered through:

1. Importance of the topic: the centrality of the topic studied to a substantive field; the ideas about importance should reflect wider societal values and circumstances.
2. Contribution ot the literature: mere confirmation of previous research is of little value. It must make a significant contribution to established knowledge. This links to Lofland's and Athens' concerns with novely and theoretical development.

we may also need ot consider relevance of the work to practitioners, as well as researchers, e.g. application of findings to practice; this i not so relevant in ethnography. Hammersley suggests that the role of research influencing practice is limited - we need to consider the same two aspects of relevance, though with diffferent implications.

1. Importance of the topic: will the research help the practitioner with a current problem? A short term and specific judgement.
2. Contribution of findings: confirmation of what is already known is of little use to the practitioner too. However, the researcher can question what the practitioner takes for granted.

Which is the most important audience for ethnographic research - other researchers, or practitioners? Assessments of practical relevance of research are continually changing. Judgements of the relevance of studies rely on prior value judgements. How can we justify those judgements? Hammersley believes there are imprtant questions about the ability of ethnography to produce relevant results. Situations studied by ethnographers rarely have intrinsic relevance; they have to be made relevant by generalisation  or theoretical inference.

Summary: Two over-arching criteria that must govern assessment of the value of social research: validity and relevance.

Who is Martin Hammersley?

Professor of Educational and Social Research at the Centre for Childhood, Development and Learning at The Open University. He has carried out research in the sociology of education and the sociology of the media. However, much of his work has been concerned with the methodological issues surrounding social enquiry. http://www.bloomsbury.com/author/martyn-hammersley#sthash.L4aMeWUO.dpuf

Any critiques of his work? I can't find any with a quick look.


OK, so what do I think? It makes sense. Sort of. I'll think some more after the first weekend session.