Tag Archives: Assessment

My short and long “must reads” around ChatGPT and LLM⤴

from

Undoubtedly LLM (large language models), and in particular ChatGPT, is the hot topic in education right now. David Hopkins has helpfully started and shared a flipgrid where he is sharing articles around generative AI, and I know many others are doing the same. Amongst the hype there are thankfully a growing body of people who are writing informed critiques. In this post I just want to quickly highlight a couple of publications that I think are a must read.

Firstly the UNESCO Quick Start Guide to ChatGPT and Artificial Intelligence. This is provides a really good overview of issues including a useful flow chart to help decisions around using ChatGPT, applications for education and some of the current issues. I suspect this will become a “go to” resource. It’s something that all educators should read.

And once they’ve done that then I have to recommend 2 longer pieces by Helen Beetham. Firstly, “on language, language models and writing“. In this essay, Helen really gets to grips with a key issue that is missing in many of the articles about LLM and ChatGPT, that is what is the purpose of writing? Why do we do it? It’s not just about structuring of text, personal reading. I think most people (well at least you, dear reader) does now understand that these language models work on prediction, and have no sense of context. So although the text may read well, it will often lack purpose and understanding. As Helen points out ” Writing by human writers is not only about the world, it is of the world and accountable in it.”

She goes on to explore some of the potential benefits of using systems such as ChatGPT. Can they be seen as writing partners? We supply the prompts, they supply the text . . ? I was struck by this.

“The illusion that these are more than tools or interfaces – that they are our partners in language, our interlocutors. We already spend large parts of our lives engaged in vivid graphical and sensory illusions. We should count the costs and benefits before rushing into a life of dialogue with illusory others”

And this

“Students see writing as a diverse, messy, inexact, variously motivated practice they are developing for themselves. Then perhaps they can aspire to be a writer among writers, and not a human version of ChatGPT.

I thank Helen for being the writer she is to have come up with that last turn of phrase. And then she goes on to point out:

But tools are not neutral. Just as language is not ‘simply’ the words we use to express our meanings to other people, tools are not ‘simply’ the means we use for exercising our personal intentions in the world. Tools carry the history of how they were designed and made. They shape practices and contexts and possible futures. . . With so many other tools we can use creatively, we must surely weigh the risks against the creative possibilities.”

In terms of education Helen also raises some really valid points for strategic leadership in universities. It does seem an awful lot of responsibility is being heaped on students, maybe we need to be asking these questions

“While students are held stringently to account for their use of LLMs, how will universities account to students for their own use of these systems? Can they hold out against black-box capabilities being embedded into the platforms they have come to depend on? Who is assessing the risks, and how are those risk assessments and mitigations being shared with the people most affected? These are questions that universities should be attending to with at least as much energy as they are policing students’ use of apps.”

There is also an accompanying piece students assignments in a time of language modelling. Again this is a really thoughtful (and pragmatic) piece about why, how and when to use writing tasks in assessments.

I would thoroughly recommend reading both essays, and engaging with Helen’s writing over on substack.


Learning Despite Assessment⤴

from

Gold star
“Gold star” flickr photo by NomadWarMachine shared under a Creative Commons (BY-NC-SA) license

Recently I’ve heard two different people say the following about assessment:

  1. Assessment is the driver for all learning (in HE)
  2. Active learning and assessment are the same thing

I think that both of these statements are obviously wrong, or they ought to be.

I can, sadly, understand how assessment becomes the motivating factor for many students because the stresses of being a student and the pressure to always get the best grade override any joy that there might be in learning for the sake of learning, so I can appreciate (1) being said, at least as an observation about the current state of affairs in HE.

The last semester that I taught undergraduate philosophy was particularly fraught, with many tutorials being cancelled due to union strikes and others snowed off, but one event stands out, even now. A particularly bright student went off at a tangent, and I suggested some reading that might interest him, probably some Wittgenstein. After the tutorial, a group complained to the course convenor that I was wasting their time by talking about materials that would not be needed for the final exam. This wasn’t the only reason that I handed in my notice and walked away from that post, but it was a contributing factor.

However, (2) just strikes me as bizarre. Putting to one side my issues with the term ‘active learning’ (I don’t believe that learning can be passive – for learning to be happening, there must be something active ‘in the learner’s head’), then I think that we could reasonably assert that a student who is working on an assessment is actively learning, but that does not entail that all (active) learning is or involves assessment. When I reflect on the times that I have learnt the most, they have usually had nothing to do with assessment. The serious fun that happens during experiences such as #CLMooc, #DS106 and the rhizos have led to some of the most meaningful learning that I’ve ever experienced – and assessment most definitely had no part in those experiences.

The bright student in my story above might have ended up using the ‘extra’ learning he had done as part of a future assessment, or he might not – learning is not contingent on past or future assessment. All of this shows that active learning and assessment are not the same thing, and that there can be learning without assessment. And I think I am going to go further and assert that there can also be a type of learning that happens despite assessment. If students are over assessed, as they sometimes can be, then any extra curricula activity – and the meaningful learning that happens as a result of serendipity – will have to be fitted in somehow. And if there is pressure, from whichever direction, to teach to the test, then the keen student who strays beyond the required materials will be learning despite assessment.

Assessment has its place, and accreditation is important. But assessment is not the whole picture, and we should firmly resist anyone who suggests that it is.

Pedagogy, place and pragmatics⤴

from

Following on from the report that has just been published on Approaches to Curriculum and Learning Design in the UK HE sector, Helen Beetham and I are exploring some of the key issues that were highlighted through the survey and the interviews we conducted. Central to this are issues around time, space and place. Earlier this week we were able to start to share some of our initial thinking during a workshop at the Jisc Student Experience Experts Meeting.

In the interviews I conducted as part of the project, there was a general consensus that after the first lockdown most organisations were quite keen, even quite ambitious about their future plans for new approaches to learning and teaching. There was a sense of an appetite to embrace some the changes to practice that being forced off campus had brought about. Assessment was a huge part of that.

Rapid changes to assessments had to be introduced, along with rapid changes to assessment regulations. Student care was high on the agenda – a visible sign of that was the no detriment practices that many adopted. Again in the interviews, it was clear that lots of the changes from f2f exams to online submissions of various types including open book, authentic assessments have now been adopted.

In terms of wider curriculum change, it was also clear from the survey responses and interviews that the appetite for changes to other aspects of curriculum design and delivery had been divisively impacted by the UK Governments’ insistence that everyone needed to be back on campus, at lectures and doing “proper” in person exams. Never mind the lessons that had been learnt from students about the benefits of more flexible, accessible and inclusive approaches. Strategic statements were subtly altered to reflect as a pragmatic response to that political driver.

However, back in the real world, we can’t ignore that our understandings and use of the spaces, places (both physical and digital) and times for learning and teaching have been altered by the pandemic experience. Students have been off campus, on campus, off campus, on and off campus for a bit . . . and now on campus. Typical 1st and 2nd year students have had their final years of school turned upside down in the same way.

I think how “be” a student has changed, and that might be one of the reasons there have been so many issues around engagement. Where (and when) you actually need to be isn’t as clear cut as it was in the “before times”.

Going back to assessment, some of the comments student interns on the Irish EDTL project made during one of their webinars really struck me. Including the student who very eloquently shared how being able to take assessments off campus, in a space that was comfortable for them, massively reduced their stress levels; another who felt that the design of some of the online MCQs exams they had taken were “mean” as they didn’t allow you to go back to a question to answer it. That experience was making them want almost long for pen and paper exams. In the panel discussion at the experts meeting, Deborah Longworth from the University of Birmingham shared how some changes to assessment are now having impact on the mental health of students. She described how some students can think that a 72 hour open book exam means that they need to be working on it for 72 hours. Does this mean taking time to develop more scaffolding around time expectations, or is it an “in” to go back to fixed, in person exam that everyone understands the conventions of ?

Whilst terms such hybrid and hyflex are commonly used and, are they really fully understood by both students and staff? Do we really have effective examples of how these approaches work in practice. This is one area Helen and I want to explore from a pedagogical lens.

We are starting with time, and thinking in terms of synchronous and asynchronous. Then considering what types of activities/interactions that work best in these contexts, and then starting to map the spaces and places that students and staff need to be in as these activities are instantiated. In terms of broadening our approaches to learning design, do we need to be more explicit about time, space and place expectations in?

As the cost of living crisis starts to really kick in, what additional changes do we need/ are we making to make to our physical estate to support our students (and staff). Warm areas, areas with kettles? What choices might commuting students have to make about how many times a week they can be on campus?

As we discussed these issues in the meeting, a dose of pragmatism was injected into the conversation. Whilst it is often said that pedagogy should always come before technology, in reality it’s pragmatism, and the contextual constraints that everyone has to work with that really make have “the power”. Pragmatics always win over everything else.

I know I have run many learning design workshops where some really innovative approaches have been planned, only to find out that 2 weeks before the start of term, the plans have been changed because of timetabling issues or more commonly not enough staff resource or time.

As the sector moves forward is it just easier to cope with increases in student numbers, and the staff/studio ratio to just timetable in lectures? Is it just pragmatically more effective not to change workload models and notions of contact time to reflect the shifts in preparation/contact time and presence needed, and stick with the conventions we are all familiar and comfortable with?

Hopefully not, and that’s what we are working on now, to develop resources that can help provide guidance and exemplars of how the sector can, and is, evolving to allow us to think about pedagogy and place and hopefully start to change some of the pragmatics and constraints approaches to learning design, and in turn the student experience, exist in. I know Peter Bryant’s recent post on the “snapback” discusses many of these issues in more depth so is worth a look if you haven’t seen it yet.

So if you have any thoughts on this, or would like to share any examples, please do get in touch, or leave a comment. We want to provide spaces to have these conversations and hopefully provide some resource to help others have them in their contexts.

PowerPoint Presenter Coach to help become clear, confident and expressive when communicating & presenting⤴

from @ Digital Learning & Teaching in Falkirk

If you are looking for a way to support learners (or indeed, yourself!) to be clear, expressive and confident when communicating & presenting to others using digital tools​, then Microsoft PowerPoint Presenter Coach is a handy feature of Microsoft PowerPoint. It’s built in to PowerPoint whether on the web, on desktop/laptop or mobile device, and …

Inspiring Digital Enterprise Award – iDEA⤴

from @ Digital Learning & Teaching in Falkirk

The Inspiring Digital Enterprise Award, known as iDEA, is an award programme which provides access to a range of free on-demand online courses. In completing courses from the wide range available they not only aim to support developing digital, enterprise and employability skills, the completion of the assessments built into each course provide the means …

Beyond Multiple Choice #BMC2020⤴

from @ ...........Experimental Blog

Logo Beyond Multiple Choice

I am doing a bit of tidying up and catching up on some posts that should have appeared earlier in the year.  What a strange yet busy time it has been.  In early November I was asked to contribute to this global conference on assessment. You can find my modest contribution from a College perspective in day three of the proceedings. 

Beyond Multiple Choice #BMC2020   ( worth clicking on timestamp below videos to see running schedule) 

It was a really interesting conference given the world's reaction to CoVid - suddenly on-line testing came into its own - in ways that may surprise you - if you watch one session have a look at final session of the day about how the national testing system in Scotland is operating and the political pressures around this.  It's all about putting the learner at the centre. 

My slides below - if you have an interest in following this global community you can join the linkedIn Group. 

Teaching in Higher Ed podcast: Time, space and place⤴

from

A couple of weeks ago I was delighted to spent a really lovely hour or so chatting with Bonni Stachowiak as part of her amazing Teaching in Higher Ed podcast.

We covered a myriad of “stuff” around some of big questions around time and space and how we are all “being” at university just now. I really enjoyed the conversation – I hope you do too.

Time, Space, and Place

Reflections on the SQA Technical Consultation⤴

from @ stuckwithphysics.co.uk

On Friday 14th August, the SQA announced its Arrangements for National 5, Higher and Advanced Higher courses in the 2020-21 session. This included links to the SQA's Technical Consultation on proposals for modifications to the assessment arrangements for the 2021 exam diet. The announcement also included links to a survey which closed for responses on Monday 24th August. The announcement stated that confirmed modifications would be published in the week beginning 31st August.

I responded to the survey both as a teacher and a parent, but didn't make any copies of my responses, so what follows is from memory.

I was not at all happy with the proposals for the sciences, which amounted to nothing other than a return to the shorter format examination papers of a few years ago. Having lost about a month of teaching time, simply taking 30 minutes out of the exam does nothing at all to help students or teachers. The notion that this reduces work for staff as they will be able to write shorter prelims is laughable. If anything this will create work as prelims will need to be rewritten to match the shortened format.
The failure to recognise that Covid-19 procedures in schools are making practical work exceptionally difficult to conduct, and the lack of a decision to remove the assignment components of science courses does nothing to reduce pressure on students or teachers. Such a decision might have freed up some time to allow courses to be better covered in the reduced time available.

Other options that might have been considered include -
+ reformatting exams to include a mix of mandatory and optional questions, allowing schools to decide which content to leave out in order to compensate for the time lost under lockdown
+ delaying the exam diet until June to allow greater time to complete courses
+ specifying content in courses to be removed to allow the remaining content to be covered effectively

In other subjects proposed changes include removal of coursework components or sections of courses (reduced folio pieces and the removal of the speaking component in English), with no reduction in the examination duration.

Another major concern is about the worth of the consultation exercise at all. With SQA having made it very difficult to understand the related documentation - it is loaded with edu-jargon and emphatically not written for the lay reader - they made it very difficult for students or their parents to respond to the survey.

Whilst many teachers will have responded to the consultation, the limited time available between the release of the proposals on 14th August and the closure of the survey on 24th August will have limited the number and extent of responses.

The plan for SQA to confirm their finalised arrangements during the week beginning 31st August gives little cause to believe that much attention can or will be paid to the survey responses.

I sincerely hope that SQA will do the right thing, listening to teachers, students and parents, and come up with a fair and workable set of modifications.

Thoughts on Assessment (given all that’s been going on)⤴

from @ stuckwithphysics.co.uk

The recent issues surrounding assessment and certification both in Scotland and around the UK, have prompted a great deal of discussion.
With a recent increase in traffic to this blog, mainly to my previous posts on assessment, following a piece I had published in the Times Educational Supplement, I thought it'd be worth putting together an alternative model for assessment and certification.

I should stress that these ideas are not solely my own, they come from discussions with many of the great teachers I count myself lucky to know through Twitter, TeachMeet and Pedagoo, from visits to Canada and the USA, where I heard about systems very different to those here and from an overwhelming sense that what we are currently doing to assess and certificate the learners in our schools isn't good enough.

I'm not suggesting that we get rid of exams. They clearly have their place, but it shouldn't be the only method by which learners in our schools can demonstrate their skills and knowledge. Nor am I suggesting that everything should be internally assessed, this too has its flaws, not the least of which is workload for already time-poor teachers.

My suggested system has four main components -

1. All units at all levels should be assessed online via an eAssessment platform provided by SQA (this already exists in the form of SQA Solar). Unit assessments should be Pass/Fail, with each unit gaining SCQF points at the appropriate level. Students should attempt units when they are ready, and multiple attempts available to students to show they have achieved minimum level of competence (this could be time locked to prevent immediate retest, like the DVLA Driving theory test). This arrangement would need reliable ICT in schools, but if all schools are currently able to undertake the SNSA, then there is already existing capacity which can be built upon.

2. Coursework components should be elective and gain students additional SCQF points. This would avoid the significant burden of multiple assignments for students following more than one in science or social subjects course, allowing a single exemplification of generic skills within a subject area.

3. All terminal exams at all levels (including N4 if there is sufficient demand) should be elective, allowing students to gain additional SCQF points. This would allow flexible routes for students to bypass exams if not required for their chosen path. Students who require Higher passes, i.e. for university entry, are still able to meet these requirements.

4. All learners accumulate 'learner credits' via a unique online profile, which could be integrated into, or linked to their Glow account. This would allow all of a learner's achievements, not just SQA, but Prince's Trust Achieve, John Muir Award, Duke of Edinburgh Award, Saltire Award and any of a range of other awarding bodies included in the SCQF framework, to be recorded. Each achievement could be electronically 'tagged' with metadata to detail the knowledge, skills and experiences underlying the award (using Mozilla Open Badges or similar). These could be cross referenced with searchable index of skills & awards which could be used by employers, FE colleges and Universities to assist in candidate selection.

I don't pretend that my proposed model is perfect, I know it would take a great deal of investment, both in financial terms and in terms of time, to develop and bring about such a set of changes.

What I do know is that, given all that's been going on, we are long overdue for a serious discussion about how we assess and certificate our learners. Such a discussion cannot be left to those in the walled gardens of the SQA, Education Scotland and the Scottish Government - they're most of the reason that things are so greatly in need of reform.