Today, I participated in a Virtually Connecting session featuring some great folks, including Jess Mitchell, one of the keynote speakers from this year’s Open Education conference in Niagara Falls. I was so struck by her answer to a question by Hal Plotkin about what assessment could look like from an “open” perspective that I transcribed her comments so I could mull them over more closely. Here’s what she said:
“Assessment is one of the lines where we determine the Haves and Have Nots, where we can change the trajectory of somebody’s life because of the way we are assessing. I want us to have a pit in our stomach about the possibility that our assessments are getting it wrong. That’s what I’m talking about. The Line. Our tolerance for false positives– less than our tolerance for false negatives…
I want us to have those conversations and I want us to talk about what does it actually mean to assess? What are learning outcomes? And how do we do this in a way that, yes, it can be measured….
I worry that people have an over fondness for data and they treat it as though it’s a pure mountain stream. But what I often say is mountain streams have a lot of crap in them; don’t drink from the mountain stream! And data can have a lot of crap in it, too: how it’s collected, who is interpreting it, what they’re doing with the interpretation of it, what we’ve come to rely on as a “bang for your buck most people.” All of these are very dangerous. We see this manifesting in areas like artificial intelligence and machine learning. We see that when you are part of a marginalized group, you’re not going to be the bang for the buck, you’re never going to be the big numbers.
It points to our failures in assessment, because our failures in assessment are us absolving ourselves of what is very hard. We don’t measure the hard things. So we measure the easy things, and what you measure is what you value, we know that, so what are we measuring? We’re measuring admissions; we’re measuring retention; we’re measuring graduation; we’re measuring in a micro form the regurgitation of information…
Again Jesse and Sean get at some of this in their book on critical digital pedagogy. How are we measuring love of learning? Somebody comes along and says, “I love these two different topics and I want to figure out a way to combine them.” Like, “I love music and I love chemistry.” Where can these two meet? We have been so silly, thinking of the world in these boxes. And the absence of boxes doesn’t mean nothingness, and it doesn’t mean that we have no idea what we’re going to do. The absence of the boxes means we’re free and we can learn so much more and do so much more.
I look at something like in healthcare. There’s a scorpion– I think it’s from Israel, somewhere in the Middle East– its venom can illuminate cancer cells so that when you’re resecting tumors, you can know whether or not you got all the cells. Who on earth went out there into the desert and said, “You know what I’m going to do? I’m gonna to try to get this venom out of this scorpion, and I’m gonna sprinkle it on some cancer cells!” It is magic! But it’s science, too.
I think that our fondness for data and are our lack of criticality about data: there’s a fondness for science but a lack of understanding of what science is. Science is trying to prove the no, trying to prove it doesn’t work. So what I love about science is it asks the other question, not the “this is my assumption and I’m going to try to prove that,” but how do I disprove my assumption.”
Watch it here: