Nov 7, 2016

PhD blog 11/16: Research methods and pseudoresearch

For this month's PhD blog, I'd like to take a moment to talk about qualitative research methods. It's very hard to overstate the importance of research methods, because they're really the only significant difference between scientific research and, well, a blog post, say. People tend to use research as a sort of general catchall term for looking stuff up, which is a virtuous activity in itself, but the scientific definition of research is more than that: it's done according to a particular, formally defined method.

Or at least it should be. Unfortunately, I've seen quite a few bachelor's or master's theses in the humanities where the method section is more or less divorced from the rest of the work, even to the point of being completely irrelevant to it. Sometimes what happens is that a student will write out their method section, move on to the research, end up doing something completely different and never return to the method section at all. More worryingly, though, I've met lots of students to whom research methods, especially qualitative ones, are complete mysteries. Their view has tended to be that you do your research, and then you have to write some kind of method crap because the person grading your thesis said you have to. Looking at their work and listening to them talk, it seems to me that to them, research isn't scientific because it's done according to a scientific method, but rather because it looks sciency. Their method sections may, at worst, be completely incoherent because they're an attempt to look methody rather than actually convey information about the research methods they've used.


In the past, I've referred to this kind of thing as cargo cult science, in the spirit of Richard Feynman, with whom I share a birthday. That started bothering me, though. Not only does it seem kinda racist, but I also have no idea if the kind of cargo cults Feynman talked about ever really existed. I still don't know about the latter, but the central claim of the cargo cult analogy is, in fact, racist and flat out wrong.

Cargo cults aren't a group of ignorant natives who don't understand how Western material prosperity - "cargo" - came about, and try to mimic westerners' actions in order to acquire it. Cargo cults have been seen moralistically: stupid natives think riches come about by magic, while intelligent white men know they come about through hard work. Cargo cults arevthe response of people who know this is nonsense. The central point of cargo cults is that the colonizers know the secret to creating wealth, but refuse to share it with the colonized. The goal of cargo cults was to revitalize native society to overcome this disparity. As Ton Otto puts it:

For Melanesians, physical work was not so much the issue; they had been working hard on Western plantations without becoming rich. Rather, the issue was access to the sources of true - that is, effective - knowledge. With the advantage of hindsight, one could wonder which worldview would be closer to modern economic understanding.

So, you know, if you use cargo cults as an example of stupid natives not understanding how things actually work, you're not only being a racist; you've also got it completely the wrong way around. So there's yet another example of a universal law of our society: whenever a privileged white person like me wonders to themself whether something is racist or not, the answer is pretty much always not just "yes", but "yes, and even more so than you thought".


So having said that, I think a better term would be pseudoresearch. But why does it happen? How do students get as far as writing a master's thesis with only the shakiest grasp of qualitative methods? In my mind at least, the failure is primarily one of teaching. For whatever reason, far too many teachers either let students drift through with only a rudimentary understanding of what research methods even are, or even participate in the mythifying of method. This is also a subset of a wider problem in academia, where many important skills aren't really taught, but students are expected to learn them through example and osmosis. Unfortunately method, at times, is one of them.

If you want to do good qualitative research, you absolutely have to take research methods very seriously. Like I said earlier, method is the difference between essay-writing and research. But methods are more than just guidelines to make sure you're doing research right; engaging intelligently with your method literature will also usually lead you to ask better questions about your material, as well as about previous research.

In my opinion, a qualitative research project needs to be firmly grounded in a methodical starting point. This isn't actually as much work as people might think. We don't do research by randomly chancing on a pile of material and deciding to find out what it is, or I'll be very surprised if very many people do. The vast majority of qualitative research projects rise from a preconceived notion that the material you intend to study is somehow going to produce information relevant to the phenomenon you're interested in. This, in a sense, is the initial hypothesis of the project. It can be kind of trivial; in my case, I'm studying Finnish army doctrine, and I believe Finnish army doctrine is found in the army's field regulations and other official documents. This isn't a very controversial hypothesis, but it is actually a very fundamental starting point for my work. Similarly, any other research project will be based on a research question that makes sense in terms of where the information we're looking for will be found. It would be very strange to start a thesis on Monophysite theology in machine-translated vacuum cleaner operating instructions - unless you had some reason to believe you're going to find it!

Having come up with an idea of where we think what we're looking for is, the next question is how we find it. In my case, a previous study has already summarized the basic development of Finnish doctrine in terms of how, say, the field regulations changed. I have a deeper interest: I'm interested in what the various regulations and other texts tell us how the people writing them, mostly Finnish professional officers, saw warfare, technology, the battlefield, their opponents, and so on. In other words, I'm proceeding from a social constructionist viewpoint where I'm looking at what kind of world the texts create. Therefore, my method is discourse analysis, because this lets me get at the deeper assumptions and structures of the texts. I also know that military texts have a very strong factualization strategy, and are strongly tied up with nationalism; because I specifically want to challenge these things, I'll be more specific and say that what I'm doing is critical discourse analysis.

There! That's already enough for the method section of, say, a bachelor's thesis. All I've done is explain what I intend to do with my sources and cast it in the particular language of qualitative research methods. That's what a method section is; indeed, that's what research methods are. In other words, if you're doing research, you have a method, whether you admit it or not. Your research will be much better served for admitting that you do, and going to at least the trouble of putting it into words.


I mentioned a tendency for method sectons of studies to be somewhat detached from the actual work they're appended to. This can happen because they're simply pasted on afterward, but more often I think it's because people will pick a method or methods, come up with an approach, and then as they're doing the actual research work, they'll find things taking turns they hadn't expected. This, by the way, is a good thing: good qualitative research isn't about making a plan and then carrying it through no matter what. On the contrary, as you do research, you develop a relationship to the material and begin to see it in new lights and from unexpected angles, and this can lead to really good insights, even totally new approaches.

Here's where I commit something of a heresy. Don't be alarmed, I have a theology degree, I'm qualified for this. I believe this evolving relationship between a researcher and their material needs to be enabled and recorded. Believe it or not, for a lot of people this is still heresy. There's still this notion that the way you write a thesis has to be what we call "winners' history": I, a brave researcher, knew that if I ventured into this source material with method Q, I would find X, Y and Z, and ha! See how clever I am? Here they are! When what actually happened was that we were looking for A using ф, but it turned out a bit different. Often a layperson reading a thesis will come away thinking that before you even start, you have to know exactly what you're going to find and how. This is nonsense, and it's exactly the kind of mythification of research methods that got me to write this damn thing in the first place.

The actual process of forming a reflective relationship to your methods and your materials is something I'll have to get back to another time. For now, it turns out that the major activity of this fall term for me is participating in a writing workshop run by my doctoral programme, where I intend to write the first draft of my introduction chapter, which will include my research methods. This post has been an attempt to set out my reasons for doing this.


On cargo cults: Otto, Ton: What Happened to Cargo Cults? Material Religions in Melanesia and the West; Social Analysis 53.1 (Spring 2009): 82-102. In my opinion, the author dismisses the racist and colonialist dimension of the term "cargo cult" far too casually. If anyone can suggest a good source on whether Feynman's cargo cults ever actually existed, please comment below!

For factualization strategies in military language, you still can't do much better than Carol Cohn's wonderful article: Cohn, Carol (1987): Sex and Death in the Rational World of Defense Intellectuals. Signs 12:4, 687-718.

On method and reflection: Doucet, Andrea & Mauthner, Natasha (2002): Knowing responsibly. Ethics, feminist epistemologies and methodologies. In Miller, Tina, Birch, Maxine, Mauthner, Melanie & Jessop, Julie (eds.) Ethics in Qualitative Research. SAGE Publications Inc., London.

No comments: