Kappa Calculation

There are three steps to calculate a Kappa coefficient:

Step one, rater sheets should be filled out for each rater.  In the example rater sheet below, there are three excerpts and four themes.  Enter 1 in the corresponding cell if the rater thought the theme was present in that excerpt; enter 0 if the rater thought the theme was absent.

Rater Sheet: RATER 1

Excerpt # Theme(1=Present    0=Absent)
Theme W Theme X Theme Y Theme Z
Excerpt 1 1 1 1 1
Excerpt 2 0 1 0 1
Excerpt 3 0 1 1 1
Excerpt 4 1 0 0 1
Excerpt 5 1 1 1 1

Rater Sheet: RATER 2

Excerpt # Theme(1=Present    0=Absent)
Theme W Theme X Theme Y Theme Z
Excerpt 1 1 1 0 0
Excerpt 2 0 0 1 1
Excerpt 3 1 1 1 1
Excerpt 4 1 0 0 1
Excerpt 5 1 1 1 1

Step two, you will need to fill out a matrix for each theme.  For theme W for example, Rater 1 and Rater 2 agreed 3 times that theme W was present in the excerpts, so enter 3 in box A of the matrix.  One time Rater 1 thought theme W was absent and Rater 2 thought theme W was present, so enter 1 in box B.  Rater 1 never thought a theme was present that Rater 2 thought was absent, so enter 0 in box C.  One time Rater 1 and Rater 2 agreed that theme W was absent, so enter 1 in box D.

Key

Rater 1
Present Absent
Rater 2 Present A B
Absent C D

 

Theme W

Rater 1
Present Absent
Rater 2 Present 3 1
Absent 0 1

Theme X

Rater 1
Present Absent
Rater 2 Present 3 0
Absent 1 1

Theme Y

Rater 1
Present Absent
Rater 2 Present 2 1
Absent 1 1

Theme Z

Rater 1
Present Absent
Rater 2 Present 4 0
Absent 1 0

 

Enter your theme matrix values in the calculator to determine your kappa values.

Theme A B C D Observed Expected Kappa
W 3 1 0 1 0.8 0.56 0.00
X 3 0 1 1 0.8 0.56 0.00
Y 2 1 1 1 0.6 0.52 0.00
Z 4 0 1 0 0.8 0.8 0.00

 

Click here to calculate your Kappa Coefficients