Challenges of marking

I was thinking this week about different ways of marking assignments and giving students feedback. I try pretty hard to eliminate as much subjectivity in my marking as possible and I’m a big fan of marking schemes and rubrics. However, when I talk to students about the marks and feedback I give them, I am left wondering how objective my marking really is, and whether the feedback I give enhances future learning, or just drags students down.

<side thought>Perhaps it’s worth saying that I still think that my marking and feedback are objective and rigorous, but if I believe that I cannot be truly objective about my own ideas and actions (as I realised when I wrote about Pedagogy of the Oppressed) then I need to – at least – reflect on the views and opinions of other ‘tea-dents’ in my learning community. (‘Tea-dent’ is a word I made up while thinking about Paulo Freire’s work to describe communities of learning where both students and teachers work on problems together, and in doing so learn together.)</side thought>

As I reflected on the feedback from my students on my feedback to the students(!!), I felt frustrated by the thought that if students were to mark my feedback (as I had marked their assessments) I’d only get a pass (or maybe a low merit). I’ve written about similar sentiments before, but (as always) I cannot languish in frustration and self-justification, I – as I would ask my students to do – move towards reflection and improvement. It’s too easy just to Talk the Talk, I gotta Walk The Talk! (And prove it – that is Talk the Walk!)

Time and effort

My first question was ‘had I spent enough time marking the assignments’? This question is based upon the assumption that more time spent marking the work equates to a better marking and feedback quality. While I had set aside time in my scheduler to do the marking, I’d worked out how long to spend on each assignment, by how much time I had available in my diary, rather than working out how much time I needed for the length, type and number of assignments.

However, I couldn’t conclude what a reasonable amount of time for marking an assignment was. For me personally (at my level of experience) about 100 words a minute doesn’t seem to be too far of the mark – this article suggests 30 and 20 minutes to mark 2000 and 1250 word assignments respectively.

(Incidentally the article linked above by Pete Greasley – from the book ‘Doing Essays and Assignments’ – is a really useful student-centred introduction to the marking process.)

So, I’d probably go for this equation:

Time needed to mark a group of assignments =

60 min # this time is to set up rubric, marking criteria, re-read the assignment and sort out issues such as high Turnitin scores, gradebook visibility, late submissions etc

+ (no. of assignments) x (max word length/100) min #this is the reading and marking time

+ (no. of assignments) x 5 min #this is for writing feedback

How do students see feedback?

I was googling and trying to understand how students ‘see’ the feedback they are given. There are some challenging statistics in this article (from The Conversation, a sort of academic/research public engagement opinion/comment website) including a survey that shows that over a third of students find their feedback discouraging. I can’t remember deliberately balancing my positive and negative feedback comments, and perhaps that is part of may practise I could improve. However, there is a counter-argument that comes from the ideas that there should be more positive than negative comments by a factor of 3 (the Losada ratio) or even 5 (the Gottman ratio) or even 6 in high performing business teams!. That’s a lot of positive comments that need to be ‘found’ for every negative one! If however positive comments have a bigger impact on promoting the desired learning than negative one (as proposed by the HBR article above) then it might be worth it.

The argument immediately comes up about what happens if a student’s work has more negative points than positive. I guess it’s about selecting the two or three key negative issues and asking the student to correct them, rather than just loading them up with all the negative stuff.

In terms of my practise I suspect that a lot of it depends on whether I mark assignments with a positive or negative mindset. But, it gets harder to stay enthusiastic when you’re one your 70th report!

At one point this week I tried to gather some student opinions on these reflections. The most interesting point was that positive feedback was irrelevant because the purpose of feedback was to improve. However, at this point I’m going to go with the data from The Conversation survey.

Cross correlation and consistency

One of the real challenges is trying to keep a consistent level of marking cross all the assignments. It easy if there are only two assignments to mark and you can read them side-by-side, but by the time your up to five assignments, or marking over several days, you start to forget how you scored certain features.

The challenge, as a marker, is that students seem to be able to find those scoring inconsistencies. (Perhaps that’s a sign that there are so many!)

Consistency is not just a challenge for me. This article in THE shows the extent to marking inconsistency in final degree exams, highlighting that in some cases first class papers are marked as third and vice-versa. However, in the abstract of the original cited paper the authors suggest that:

We conclude that universities should be more honest with themselves and with students, and actively help students to understand that application of assessment criteria is a complex judgement and there is rarely an incontestable interpretation of their meaning.” (taken from Bloxham et al).

That’s a powerful, bold statement.

Marker bias

The Greasley paper (linked above) highlights three possible areas of bias: ‘halo effect’ (where students who normally submit good work are given unofficial allowances and marked higher), physical attractiveness (where studies have shown that essays given to markers with a picture attached of an attractive student are marked higher) and marking sequence (that assignments marked immediately after a bad assignment are marked higher than if they had been after a good one).

Personally, I’ve moved to anonymised marking in a random sequence, so these biases should be minimised and I find it distracting to know a students identity while marking. I end up wrestling over whether I am being lenient or strict because of what I know about them.

This recent meta-analysis by Malouff and Thorsteinsson, suggests that different race/ethnic backgrounds, education-related deficiencies, physical unattractiveness and poor quality of prior performance may all have an impact of final marks by 1/3rd of a standard deviation. That said, if most University assignments have a standard deviation of 15% (say an average mark of 65% with less that 15% of the class failing and less than 15% getting above 80%), the bias reported in this study would only contribute to a 5% marking difference. This, in my opinion, is well within the ‘noise’ of the marking process (especially when considering the implications of Bloxham et al as discussed above).

One interesting aspect to Malouff and Thorsteinsson’s study is that the presence of a marking rubric contributes to lowering the bias marking observed.

As for the physical attractiveness issue the studies cited by Greasley seem to have been done in the 1970’s when the issues around gender relationships within an academic environment were far less balanced and equal as they are now.

Detailed assignments

I’m now fairly detailed in how I set my assignments and I think publishing the marking rubric helps express that detail. It is known that students often misunderstand assignment requirements, and that markers award higher marks where the concepts the students discuss in their assignment are closer to the markers own mental framework. (See comment and references cited in this paper by David Nicol called “From monologue to dialogue: improving written feedback processes in mass higher education”) But, I can’t sit back and relax with my rubric because the paper also mentions that “They note that conventional practice where a list of printed criteria (which describes my rubric) is shared with students is not enough to transfer tacit knowledge from teacher to student.” (My comment added in parenthesis.)

I spent some time digesting the ideas in the Nicol paper but that’s another blog post.

Moving forward: what actions will I adopt?

  1. Set the time in my schedule I need to mark by the assignment, not the time I spend by my schedule. Use the equation I proposed above, but, always try and be efficient and effective.
  2. Convey that the assignment of a numerical mark to an essay, or report, is a complex, subjective judgement which has to be done by a nominated person (or persons), and that other interpretations are possible. I will try to eliminate that subjectivity as much as possible, but it will always exist
  3. Keep, and strengthen, my practice of anonymous marking.
  4. Try and approach each assignment with a positive mentality, and with a fresh pair of eyes.

 

 

Categories: Tags: , , ,

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s