Captured data in an LMS is only part of the story. Real human activity can’t be expressed in clicks. In particular, the word ‘engagement’ is misunderstood and misused.
Learning Designer – 2023 survey
In preparation for the upcoming Learning Design Meetup on learning analytics, we invited staff at UTS and beyond to share perspectives via a short survey. Early responses show a diverse range of knowledge, confidence and concerns with the use of learning analytics in teaching and learning, pointing to some areas where we can start challenging assumptions, broadening perspectives and building skills in learning design.
We share a snapshot of the survey findings here, which included responses from learning designers and those who teach and support others in learning design. Also contributing were staff for whom learning design is one part of their broader role, and those who have a professional interest in the topic, or are studying learning design.
What do learning analytics mean to you?
When we asked respondents what three words come to mind when they think of learning analytics, it’s clear that this topic occupies a specific, data-oriented place in our minds. ‘Data’ is most frequently mentioned, along with words that indicate thoughtfulness about the processes of working with the meaning behind learning analytics, like insights, interpretation, reporting, and the ubiquitous ‘dashboards’.
People are also considering the usefulness of learning analytics, with concepts relating to impact, action, and change often mentioned. Along with this, however, come frustrations with the confusion and ‘noise’ of large volumes of data, and issues with ethics, privacy and bias in working with student analytics.
Less prominent in these spontaneous associations are concepts relating to engagement and student experience. At the surface level, learning analytics feel quite distant from the students whose data is often analysed – although later comments on skill sets and concerns suggest an awareness of this, and an interest in ways to better connect the digital footprints with students’ actual experiences.
How are we using learning analytics?
Engagement and participation are only correlated with learning, not evidence of it.
Learning Designer – 2023 survey
Three main uses of learning analytics are noted in the survey, with the top two responses indicating a focus on understanding student engagement and participation in learning. Respondents are also using learning analytics to inform changes in course design, or to evaluate a course or subject. Commentary elsewhere in the survey adds important detail here, suggesting that concepts of ‘engagement’ and ‘participation’ are complex and require further analysis and often triangulation of other data sources to make meaningful interpretation.
The survey findings suggest slightly higher (self-rated) confidence in finding and collecting learning analytics data, and interpreting the data to inform learning design decisions. Overall confidence in the process of analysing learning analytics data is a little lower, suggesting that this is an area where learning designers are aware of a potential skills gap, or may not have access to the right tools to make analysis more accessible.
How to start when you are not confident at maths/data?
Learning Designer – 2023 survey
I would like to know what kinds of analytics matter and what are just ‘vanity’ metrics, as well as how to analyse and interpret the data meaningfully.
Learning Designer – 2023 survey
What skills do we need now?
Data is in so many places and often isn’t complete, but we need to use it to inform decisions we make. And […] people aren’t being trained/ up-skilled in how to source and interpret data.
Learning Design part of skill set/ professional interest – 2023 survey
Respondents suggested that learning designers need a range of skills to use learning analytics effectively. Most frequently mentioned were data analysis and data literacy, interpretation skills, and critical thinking – again, covering multiple points in the process of analysis, pushing beyond the raw data gathering to a need for thoughtful insights and perspectives.
Also noted, though less frequently, were skills relating to visualisation & communication, statistics and numeracy, technical/ digital literacy, and knowledge of ethics & data policies. It’s clear that learning analytics actually stretch far beyond ‘data’ for many involved, and that there is an appetite to learn more, not only about the technical skills, but what it takes to generate and communicate useful insights and use the data in appropriate ways.
Explore more at the June Learning Design Meetup
If you’re keen to delve deeper, the quarterly UTS Learning Design Meetup in June is sharing reflections and examples of learning analytics impact from three practitioners: Kirsty Kitto (Associate Professor Connected Intelligence Centre, UTS), Danny Liu (Associate Professor (Education-Focused), University of Sydney) and Donna Rooney (Senior Lecturer, Faculty of Arts & Social Sciences, UTS).
Before you come along, we would like to understand where you’re at with Learning Analytics. How have you used L.A. (if at all)? How confident are you in putting them to use? What are the skills learning designers need to know, and what questions or concerns do you have? Share your responses and take a peek at the results: https://www.menti.com/al947qazo5y7. To attend the Meetup, contact Mais Fatayer and join the conversation in the Learning Design Meetup Teams channel.