I’ve been preparing online discussion data for alternate coders to calculate intercoder or interrater reliability (IRR) in content analysis for my thesis.
IRR can be defined as “the extent to which different coders, each coding the same content, come to the same coding decisions” (Rourke, Anderson, Garrison & Archer, 2001) Considered the primary test of objectivity in content analysis studies, a number of indices exist to report it.
The simplest and most commonly-used statistic for IRR is percentage agreement. It accommodates any number of coders, but fails to account for agreement by chance. Hence, some researchers consider this a too liberal an index (Lombard, Snyder-Duch, & Bracken 2002).
There are also statistics that account for chance agreement among raters such as Cohen’s kappa, Scott’s pi, or Krippendorff’s alpha. Some researchers like Potter and Levine-Donnerstein (1999) argue that statistics like Cohen’s kappa are overly conservative; others, like De Wever, Schellens, Valcke, & Van Keer (2005) favor calculating both percentage agreement and Krippendorff’s alpha. The rationale for choosing Krippendorff’s alpha is because only nominal level of data are taken into account in calculating Scott’s pi and Cohen’s kappa, and these calculations are only applicable for research based on two coders.
Many published studies that used content analysis don’t report any measure of reliability, but for my thesis, I plan to report at least percentage agreement, and possibly other indices. Rourke et al. (2001) suggest using SPSS’s chi square function to calculate percent agreement, but Lombard, Snyder-Duch, & Bracken, (2002) warn that “chi-square produces high values for both agreement and disagreement deviating from agreement expected by chance.” What to do? Rosemary kindly shared with me a copy of an Excel spreadsheet that she used for percentage agreement and final IRR calculations in her thesis, so I’ve started adapting this file for mine.
More later on how I’m dealing with reliability on not just coding but also segmentation. Now, it is time for the story of the black walnut!
In my backyard there stands a majestic black walnut tree. Providing dappled shade in the summer, it scatters an abundance of leaves and fruit in the autumn.
Tales of gustatory and nutritional virtues of the black walnut prompted Will to gather the green fruit, extract the kernel with a swing of a rubber mallet, and bathe the small nuts in a trug. What a great way to avoid writing a journal article!
Walnuts are used to create dyes for textiles and wool. Prior to exposure to mordant, the dye from the flesh of the fruit appears bright yellow.
After scrubbing to remove remnants of the fruit flesh, the black walnuts appear thus, ready for roasting.
Unfortunately, we found out upon emptying the water from washing the walnut that not just some plants are intolerant of juglone. Where the water went, poor worms wriggled out of the ground to their deaths. Walnuts are a part of the deadly nightshade family. Rest in peace, poor earthworms.
Leave a Reply