The problem of known unknowns
You’ve been there. Part of your class is struggling in their assessments but you don’t know what it is that they don’t understand. You ask your students what it is they are struggling with but they can’t or won’t tell you. However it doesn’t have to be this way, especially if you are using quizzes for formative or low-stakes summative assessment. Item analysis can be used to identify difficult questions and point out potential misunderstandings. Help is on its way.
Running an Item Analysis on your UTSOnline test (a.k.a quiz)
Running an item analysis is easy. Just follow these instructions. Each analysis contains summary statistics on the:
- points possible
- number of questions in the test
- number of attempts (in progress and completed)
- average score
- average time to submit
It will also contain the Difficulty and Discrimination ratings for each question. We’ll cover these below.
Identifying what students struggle with
The difficulty rating is the key factor that will help identify areas where students may be struggling. Each question in the analysis will have a difficulty score and rating. If a question has a Hard (<30%) or Medium (>30% but <80%) difficulty rating this may point to:
- students’ not knowing or understanding the required concepts and/or information
- a poorly written question
- poorly written answers
- a question that is too difficult for the class
If the latter factors are ruled out then looking at the detailed statistics available on each answer may clarify what it is that students don’t understand.
For example, the question below had a difficulty score that bordered on Hard, with only 30.99% of students choosing the correct answer (D). However most of the students who got the answer wrong chose the same answer (A). Of these, many of them went on to do poorly in the test, with scores in the third and bottom 25%. It’s unlikely that this happened by chance, though the wording of the answer could also have been a factor.
It would pay for this teacher to think about why so many students chose answer A. Does choosing this answer indicate that students have misunderstood or misapplied a concept? Have they failed to consider relevant information? Identifying possible points of confusion or misunderstanding will allow you to design activities that correct these either online or in the classroom.
Identifying questions that allow more capable students to shine
You may want to have some questions in your quiz that allow more capable students to shine. However this doesn’t just mean difficult questions. It means questions that (positively) discriminate between students who are more proficient and those who are still developing. Statistically speaking, “A question is a good discriminator when students who answer the question correctly also do well on the test” (Blackboard). So while the question in the example above almost had a Hard rating its’ Discrimination rating was Good (0.48). Including a few questions with higher Discrimination ratings will help reward those students who’ve been working hard.
How can I correct student misunderstandings identified in the analysis?
There are a number of simple methods you can use to clarify key concepts. You can point students to online resources that cover them in a different way, or give a mini-lecture that covers them from a different angle. My personal favourite would be a think-pair-share exercise using questions that required an understanding of the concepts you’ve identified as needing clarification. This could include a quick in-class poll to see if students had improved their understanding. However each teacher will have their own methods that work well for them. Please share your suggestions in the comments below!
Want to learn how to analyse your quiz results?
We’ll be running a workshop on this topic in the Academic Hub at 1 PM on the 13 July 2017. Please register to join us.
I would like thank Harry Tse of the UTS Business School for his help with this blog post.