Brenda Boyd was a student who was returning to studies, poor and determined, and her graduation was depending on her completing a single online course at an institution which she had never gone to. The course was badly designed, the instructor could not be reached, and then one day she got an email saying that since she had sent a message in the past three weeks she should resend it because the instructor was experiencing computer problems.
It was then that she realized that there had to be a better method and that someone should work on discovering it.
Boyd then remained with Quality Matters as a senior leader for over fourteen years, overseeing the rubric suite which became the gold standard for online course quality, expanding professional development so that it could reach more than 12,000 teachers each year, and managing the national standards for quality online learning. She has since established Practice Learning, a consulting firm whose aim is to assist institutions in creating and maintaining high-quality learning experiences for their students.
Her work happens in the background: the standard that makes a course navigable, the review that surfaces what faculty never knew was missing, the professional development pathway that quietly transforms how an instructor thinks about student success. It is, as she puts it, making a difference in the lives of people who never know her name.
Listen to the podcast on your favorite directory:
What Quality Actually Is
Boyd has a gift for grounding abstract concepts in concrete images. When she talks about recognizing quality, she compares it to thrift stores.
As a young woman without much money, she learned to identify quality fabrics, the difference between wool and polyester, between something built to last and something that would fall apart. That skill let her build a professional wardrobe on a tight budget by finding things worth keeping among things that were not.
“When you can recognize quality,” she says, “you can choose it.”
The same principle applies to students navigating higher education. Without quality signals, a returning adult learner has almost no way to evaluate whether a program will serve them well. Accreditation provides baseline assurance, but it does not give students the granular signal they need to choose the right program for their circumstances, their goals, and their life. That is part of what quality frameworks like Quality Matters, OLC, and OSCQR are designed to provide.
The problem is that these frameworks are often introduced to faculty in ways that make them feel like compliance exercises. Boyd’s counterargument is one of the most practically useful ideas in the episode: most faculty are already doing most of the things in the standards. The work of a quality consultant is not to tell them what they are doing wrong. It is to help them see that what they are already doing has a name, a framework, a community of practice, and a set of colleagues who have figured out how to do it even better.
“Standards read like a stick when they should read like a carrot,” she says. “And the carrot is: this is good for your students. And you are probably already doing most of it.”
The Profession That Is Still Growing Up
One of the most honest exchanges in the episode comes when Gorman asks Boyd where she thinks the online learning profession sits on its maturity curve. Her answer is careful and candid.
We are still growing, she says. The technology keeps changing. The expectations on practitioners keep expanding. And the field has not yet settled on the clear skill taxonomies and career progression thresholds that define more established professions. The legal profession knows what you need to know, what you need to be able to do, and what the thresholds are at each career stage. Learning design, especially in online spaces, has not quite gotten there.
Boyd sees this as both a challenge and a reflection of how dynamic the work actually is. Every course is now digital to some degree, a significant shift from the pre-pandemic landscape. Students are content creators. Assessment is being rethought from the ground up, partly because AI has made the five-paragraph essay a less meaningful signal of learning than it ever was.
“Why do I have to write a paper for every single class except photography?” she asks. “Why can’t I do a photography project for my physics course?”
The question is not rhetorical. It is a design challenge. And it is one that the field has the tools to begin answering, if it can agree on what quality looks like before deciding which new forms of expression are worth building toward.
Standards That Withstand Technology Change
Boyd has watched quality rubrics evolve through multiple technology cycles, and she has a clear view of what makes a standard durable versus what makes it brittle.
At the time when AI made its appearance shortly after Quality Matters had issued its seventh edition, the term was not included in the rubric. However, the principles relating to the use of technology were already in place: is the technology up to date, does it carry out the function that the course requires of it, and does it serve the aims? These questions are just as applicable to AI as they are to all the other tools that have come before it.
“Having a standard around technology,” she says, “is really just writing something that means it’s current and it helps meet those objectives, regardless of what it is.”
Standards become brittle when you write them too close to specific tools rather than the principles beneath them. The tool will change. The principle of using the right tool for the right pedagogical purpose will not. The standard that is written at the level of the principle survives the technology stress test. The one written at the level of the tool has a much shorter shelf life.
Room to Fail
The most personal moment in the episode belongs not to online learning but to a kitchen counter covered in failed cupcake flowers.
Boyd has been learning to pipe floral designs onto cupcakes: roses, peonies, hydrangeas. Some of them do not make it. They get pushed off the cupcake before they are finished. They collapse before they set. They simply do not work.
She draws the same lesson she draws from course design. If you do not have room to fail, you do not have room to learn. The student who faces a course with a midterm and a final and nothing in between has no opportunity to practice, no low-stakes moment to discover what they do not yet know, no chance to fail where failing is safe.
“That’s what education is really all about,” she says. “We learn a lot from our mistakes.”
She extends this to the profession itself. Continuous improvement is not a project with an end date. It is a culture, a disposition, a willingness to deconstruct what did not work and iterate without treating the gap as a failure. The institutions that have built that culture, she has found, are the ones that are genuinely ready to make change. The ones that have not built it are usually the ones that want quality to be an initiative someone else manages.
What AI Has Not Changed
At the end of the conversation, Boyd offers the prediction that feels most reliable: community will remain essential. Connection between people, shared learning across different perspectives, the kind of conversation that happens in rooms where unlike-minded people choose to stay, none of that has been replaced or diminished by AI. It has, if anything, become more valuable as everything else accelerates.
It is, perhaps, the most grounded thing anyone can say about where education is headed. The tools will keep changing. The standards will keep evolving. The rubrics will need new annotations. But the reason any of it matters is the same reason it always has: somewhere, a student is trying to get somewhere, and the quality of the learning experience she encounters will either help her get there or put obstacles in her way.
Brenda Boyd has spent a career making sure more of them get there. Most of them will never know her name.
Listen to the full conversation with Brenda Boyd on The LX Ops Show or wherever you listen to podcasts.
Podcast Transcript– Jason Gorman with Brenda Boyd
Jason Gorman (00:00)
Welcome to the Learning Experience Ops show where we talk about making learning better that’s future ready. I am Jason Gorman. I’m the founder and CEO of Jackrabbit Learning Experience. Right now is a moment of shiny things. It’s a moment of tremendous change. It’s a moment of getting caught up in the moment. And in moments like this, we often get distracted from what is actually deeper and was more important. Things like quality.
The quality of what we do, how we define the quality of what we do. So we get all of these new AI powers to do new things, and sometimes we lose track of why we’re doing them at all. And that is exactly why I’m excited to be talking with my guest today. I’m really happy to be joined by Brenda Boyd. She is the founder of Practice Learning, a consulting firm that is dedicated to the support and growth of institutions.
That care about quality learning experiences for their students. Brenda was previously a senior leader with Quality Matters for more than 14 years, where she stewarded the Gold Standard Quality Matters Rubric Suite, project managed the national standards for quality online learning, led program services, and scaled professional development to serve more than 12,000 educators annually.
Prior to QM, Brenda served higher education for 11 years as a director of academic technology and as an instructional designer. Welcome to the podcast, Brenda.
Brenda Boyd (01:34)
Thank you so much for having me, Jason. It’s wonderful to be with you today and talking about quality and all of its many tentacles.
Jason Gorman (01:43)
All of its many tentacles. Well, let’s just start off with a little elevator pitch about what you do. What’s the 30-second pitch about what your work is?
Brenda Boyd (01:52)
So what I do is I help institutions improve the quality of their learning experiences. They that can look like fractional quality assurance officer. It can look like helping them with their standard sets to create replicable standard sets that are that can be reviewed across different modalities, different courses, different disciplines.
And also professional development that’s near and dear to my heart as a instructional designer and and I’ve seen the transformational power that professional development academies can have when you have a an intentional pathway for faculty and and educators to take to to really kind of transform their practice.
Jason Gorman (02:42)
And in that work, what are the things you love doing the most?
Brenda Boyd (02:46)
You know, I there’s a lot of things I love doing the most. So it’s hard
to narrow
Jason Gorman (02:50)
Okay.
Brenda Boyd (02:50)
that one down. But I think I really love working with an institution that’s committed to quality and wants to make a big change and that is ready to make that change. And I think that’s key because change is difficult. We think we want to do it, but it it’s actually hard for us as humans to change. but
we can also get ready for that and having a growth mindset is is a big part of that. I think too the other part that I think is is really exciting thing I love is that when we make a difference for students and they come back and say, my gosh, this course is so great. I wish all my courses were like this one. Because that really you know, the the work I’ve done in the past, it I used to say I
I make the difference in the lives of people who never know my name because they would see, you know, they were they’re going to to have an you know, their faculty member has implemented a standard that has made it easier for them to get their education. And that’s what I really love is that having a a an impact on people who who don’t even know who I am.
Jason Gorman (03:57)
That might
kind of be a lead into the next question. I mean, was there a specific moment like that that inspired you to do this work?
Brenda Boyd (04:06)
Yes, there was. So I was one of those returning students that is a statistical anomaly, right? Like I started my degree and I dropped out of college and I went back later. And part of the requirements for me to complete my degree involved taking a higher level course in religion because I went to University of Dayton. And I took that I took a course through another institution that offered it online.
And it was in the early aughts. Online education was really just at the beginning and I did not have a good experience in this online course and I knew there had to be a better way. And so I I was upset because my graduation depended on me completing this course successfully. And the day that I got the email from the instructor saying, If I if you sent me an email in the last three weeks, please send it again because I’ve had computer problems. It’s like
Jason Gorman (05:02)
gosh.
Brenda Boyd (05:03)
well, I think
I might have sent you an email a couple weeks ago. So and and I think that there’s a lot of it there’s an importance in course design, but there’s also a lot of importance in instructor presence and how that course is actually delivered as well. So it’s not a an either or, it’s a yes and with with quality in in a learning experience for students.
Jason Gorman (05:28)
I have to say you you’re in good company here because I was a one-time college dropout that found my way back to. so glad, glad we’re doing this today, a couple
of years later.
Brenda Boyd (05:39)
Yeah.
Jason Gorman (05:41)
what as I mentioned in my intro, we’re in this crazy moment of change.
When you think about the future, is there anything in particular that you are anxious about?
Brenda Boyd (05:51)
Well, I think I think AI is anxiety producing for all of us. because we see the potential but we also see the downsides to it. I’m anxious because I I want it to be better, I want it to get better, I want it to be more powerful and easier to use. but it but I also see the environmental impact of it and how it’s impacting communities and and I believe that, you know, we can if the
if the tech companies can do this in an ethically responsible way to, you know, create data centers that are going to be, you know, involve the community and and respect the land, I think that’ll be really important for the future of artificial intelligence and and for how society views it and uses it and whether or not they embrace it. So that’s definitely anxiety producing.
Jason Gorman (06:44)
back to AI. but before we do, I’m curious if there’s something, some kind of skill that you’re trying to pick up now. It could have something to do with your work. It could be completely unrelated. What are you learning right now?
Brenda Boyd (06:58)
So I am learning both AI. I’m taking some courses through New State University, which is all AI all the time. It’s the certificate in business in AI. So I can understand how it works. And the other thing that I’m learning is how to create floral cupcakes designs. So this is a a little a creative outlet for me and I’m learning how to
pike flowers on on cupcakes and turn them into floral bouquets that you can eat. So that’s another
Jason Gorman (07:29)
That’s amazing.
Brenda Boyd (07:30)
part of my fun time that I that I have on the on the side. So
Jason Gorman (07:36)
That’s great. Are these things, are
Brenda Boyd (07:37)
And in any article.
Jason Gorman (07:39)
these really intricate? I mean, are it sounds like what you’re describing is not like what I had initially had in my head, which is I can’t do these things at all. I’m quite, I get frustrated very easily with anything baking related, to be honest with you, but making like a flower, I I I’ve never been able to do it. Anyway, so you’re doing things that are, they sound, this sounds complicated, what you’re what you’re learning.
Brenda Boyd (08:04)
They
they look like flout like a rose or a peony or a hybrania and
Jason Gorman (08:10)
peony. Holy cow.
Brenda Boyd (08:12)
you know, I could send you a picture if you want to cut it in and I can put it on the camera and you can
see like what’s what they look like.
Jason Gorman (08:17)
Yes. my
god, what? All right.
Brenda Boyd (08:21)
So
Jason Gorman (08:22)
If you’re listening to this, go to YouTube and watch
Brenda Boyd (08:25)
Okay.
Jason Gorman (08:27)
just go look at the flowers on the cupcakes. That’s crazy. So all right, I think I think we can safely say here from the top, Brenda, that you are a a patient person. Cause I
Brenda Boyd (08:39)
Yeah.
Jason Gorman (08:40)
I don’t think
I c I cannot I cannot I do not I do not possess the patience to do that. That I that is incredible. That’s amazing.
Brenda Boyd (08:45)
Thank you. Yeah, well
that’s what a fine art degree will help you with as well. So
Jason Gorman (08:53)
L
love love that. well, so in addition to becoming like a a master pastry chef, you’ve also just launched practice learning.
Brenda Boyd (09:03)
Yes.
Jason Gorman (09:03)
And for people who don’t know you, could you talk about the kinds of problems that you solve in the work that you do?
Brenda Boyd (09:11)
Yeah, so right now I have a couple I have a few clients and I’m doing a lot of support for them with quality assurance. So that can look like level setting, like doing a benchmark study for an organization to see how their courses are doing. And that is ironically not on my website, but I’m I’m aspiring to do things like quality assurance,
fractional quality assurance leadership for institutions and organizations because this is always an initiative that is it sounds really good, but then it’s put upon somebody who doesn’t have time and they already have a full time job. And so I can come into an institution and or an organization and support them in their quality assurance their i in in making that change and in initiative.
having meetings with faculty, helping them understand how the quality initiative can support what they’re already doing and helping them grow. Because this work is there is so much of it out there to do. There are so many institutions that want to make change, but they just don’t have enough hands on deck. So that’s fractional quality assurance is part of practice learning in addition to supporting the quality standards development and then again the professional development.
But what I’m but the work I’m actually doing is more around like let’s take a look at some courses through these different lenses. And I’m using like I’m using Quality Matters standards for some of the course reviews. I’m using some of the course reviews, I’m using OLC, and I’m looking at Oscar standards too. So and I haven’t started working with anyone in K-12, but I have project managed, as you said in the intro, the National Standards for Quality Online Learning, which is K-12 focused.
But those standards are Creative Commons and they’re out there and they are there’s a lot of really great stuff in them. They were also community created, just like all of these rubrics. and that’s one of the things I think is most valuable about them. But so practicelearning.co is my website and I’m you know, I have my advance card is getting filled up already, so that’s
that’s great to hear.
Jason Gorman (11:18)
Congratulations.
Brenda Boyd (11:19)
So yeah, so it’s exciting to
to begin this new adventure and you know I really realized that the the work is still there, that you know the the commitment and the need for quality, it doesn’t really go away. I was talking to somebody earlier today and I was thinking about when I was a young woman and I was cash strapped as working as a server or whatever and I would go to the thrift store.
And you know, you can find some amazing things at the thrift store if you know what you’re looking for. And one of the things that I was thinking about was I learned what quality fabrics were, right? Like the
Jason Gorman (11:58)
Yeah.
Brenda Boyd (11:59)
difference between wool and what’s cotton and what’s polyester and what’s built to last and what’s fast fashion. And then I could like build a a a classic wardrobe on a on a budget that would allow me to present myself more professionally.
without having to spend a lot of money on it because of the quality. And when you can recognize quality, then you can choose it. And I think that that’s where students have like us, right? Like when we returned to school, how do we know where to go? Do we go, you know, like I went back to where I started from, but I had also explored other options. And I didn’t really know like what’s the best option for me. And I think that having quality signals
really helps students that they can see like, this program has been reviewed by somebody other than the accreditor, right? Like the accreditation is super important. We want accreditation to be there in place so that there’s oversight for institutions. But I think that there’s also this need for students to or returning, you know, just people to be able to pick a program that
fits their lifestyle and what they’re trying to achieve and help them learn earn living wage jobs and all that fun stuff because the signals are in the brand, right? So we look at US News and World Report or well yeah I heard that’s a good school, but there’s no like there’s no real like qualifier to help them pick out like the best program.
Jason Gorman (13:34)
Yeah. And when does an institution know that it needs an independent quality assurance person? What is the what is like the common moment where they say, yeah, now’s now’s the time to do that?
Brenda Boyd (13:48)
Well, it it it’s going to depend on that institution really. I think that they sometimes will say, like, well, during the pandemic they were all like, we need a plan because
it
Jason Gorman (13:59)
Yeah.
Brenda Boyd (14:00)
was a mad dash, right? That was emergency remote instruction. That was not online learning. Like that was not thoughtful planned out. It was like, gosh, we have to do something. And I think that gave online learning a bad rap because, you know, Zoom University was not
really what what true quality online education can be. And I think that that stigma still hangs over it. But I think that sometimes schools want to do things like improve their drop, fail, withdrawal rate, their DFW rate. They may want to, you know, they may see a program as having trouble or they’re getting complaints about a particular program. Some those things can be drivers.
But I think too it’s a kind of an institutional culture that they have to make a decision of what kind of institution do we want to be and how do we want to project that up to our accreditor. A lot of times institutions need to have to show that we have a good program and they need to show their accreditor that. And so adopting some kind of a review process can support their accreditation as well.
And but a lot of times a l a school will just say we just want to have our students to have the best experience. And that is if they choose to come here, we want them to feel welcome. They we want them to feel like they belong, and we want them to be successful, right? And we want to
Jason Gorman (15:29)
Yeah.
Brenda Boyd (15:29)
do everything we can. And that’s the thing I love most too, I think, going back to your earlier question, because when faculty care really about their students, they want them to be successful.
and they want to share what they know and you
Jason Gorman (15:41)
Mm-hmm.
Brenda Boyd (15:42)
know it’s I think sometimes it’s it can be really confusing in online because you know since we were five years old we walked into the same classroom we sat down at little desks and teachers in the front of the room and online there’s none of that structure you have to build it right like the elements
Jason Gorman (15:59)
Yeah.
Brenda Boyd (15:59)
can often provide some structure but you have to fill in where are the boundaries, who is the instructor and
w and where do the students sit, right, in in metaphorical terms. Excuse me. So I think that the that the frameworks that are out there provide that structure and institutions have to choose like w which way they want to go with
Jason Gorman (16:26)
And I’m kind of curious because you mentioned before that you you left school at some point, you came back. When you came back, you were an online you were taking online courses, but you were also, I think at the time, building online courses too at this. So I’m kind of curious actually to hear about this about your experience at that time. I mean, what was what was like missing or bad that kept happening?
that you just said, all right, this is I gotta I gotta step in and do something here.
Brenda Boyd (16:57)
Yeah, so interestingly when I s I started my career in online education through web design, I had gone to
Jason Gorman (17:04)
Hm.
Brenda Boyd (17:05)
I had gone back to a technical trade school and I learned how to do how to work with Photoshop and Dreamweaver and use different online tools and this was in the late nineties, right before right when the internet was like blowing up. And I thought, well there’s
the you know, and I wanted to do meaningful work and I I met this lady who worked at Sinclair and she’s like, I really need help. can you come and and like help me build these courses? And so I was building them in Dreamweaver and I’m I’m building a course in Dreamweaver which is an HTML editor for the young people listening, which enables you to it’s a WYSIWYG editor, you can just build the course right there. And I learned I also learned HTML so
I was building these courses and then I was we what we did was we broke the navigation in Web CT and we put it made it so that when the student logged in they saw our website instead of the you know the normal the normal interface that would be available to a student when they logged into Web C T. So we made it so there was a landing page and we had our own internal navigation and we connected it to the gradebook and all that fun stuff. So
So we still have we still use the tools in there, but what happened was I started reading the content and I was being asked to like upload this image and I’m like, that’s copyrighted, I can’t do that. And so, you know, like there were just a lot of points where I was like, Okay, well I I was turning into an instructional designer and I didn’t know it at the time. And then I s but I was building these courses, I was taking courses because I knew I needed to finish my degree if I wanted to
To get a full-time job at the college. And so I I was fortunate enough to go to my very first conference and present about our process. And it was at the Instructional Technology Council, and it was in San Diego. It was very exciting. And I got to go to that, and the I went to a pre-conference workshop on Quality Matters, which was a grant funded project at the time. And I learned about it.
And I was like, I had this Eureka moment where I was like, This is it, this is what we need is a standard set so that we can all be working toward the same things because no one really knew what to do and they had put that in writing at that point with the in the early aux and I was like, This is wonderful and so I got involved with the grant and I stayed involved as QM grew into this other you know into an
nonpro to be part of a nonprofit and then I came on board with them eventually. But it was a but it was a real aha moment for me because I because I had this experience in this course where I was just like, there’s nobody in here. There’s no one teaching it and there there weren’t opportunities to interact with the instructor. And I was really a little bit confused, honestly. And so were most of the other students and it was
Post three times to the discussion, take this quiz and then take this midterm and take this final. And I was just so scared that I wasn’t going to pass the midterm or the final because so much of the grade weight was on them. And so I was like, you know, there has to be a better way. And so even when I review courses now, I’m looking at, you know, are there opportunities for students to like practice their learning? Because if you don’t have that opportunity to practice your learning, you don’t have opportunities to fail.
And that’s what education is really all about because I could tell you even sit in my kitchen piping cup piping those cupcakes, there were flowers that got pushed off of them of the cupcake because
Jason Gorman (20:49)
I have no doubt.
Brenda Boyd (20:49)
they failed. And so you have to have room to make mistakes because that’s how we learn, right? We learn a lot from our mistakes. And I think that that’s you know, and then the other side of that coin is often well, they don’t do it if there’s no points attached, but
you know, there’s ways to to create opportunities for practice that don’t have to involve grades or points and just enable students to to try something. Try this out and see how it goes. And then, you know, you can say it’s either you did it or you didn’t. So I think as a as I as I grew in my profession as a student, I I really took that student point of view to heart because I was that student and it helped me
develop this empathy for them in in a way that I I don’t know that I would otherwise have because I have been in their shoes and you’ve got a lot riding on it. You’re spending money on a course, you’re trying to get to your goals and it and and it and and let’s make it easier. Let’s not make it harder in terms of like how do I how do I get through this in a way that’s going to support my learning and growth.
Jason Gorman (21:59)
I’m gonna go off script for my next question because I I I have something that I’m really dying to ask you. Because this is something that comes up again and again in my conversations here and throughout my career. I think if we’re talking about K-12 education, that’s
Brenda Boyd (22:13)
Mm-hmm.
Jason Gorman (22:14)
clearly an established profession, right? We understand what good. I wouldn’t say there’s you there’s not universal agreement on what good teaching is, but we understand the teaching profession. When it comes to creating online courses, I don’t think
I don’t know that that’s true. And this is my question to you. I believe personally that we’re still in a very immature phase of what we are, right? This the the online learning professional, not to say that there are not very skilled people in this work. There clearly are, but as a as a an area of expertise and work, I have always felt.
That we are still growing professionally, but we are growing. But you’re the one that actually has the answer answer to this question. So I’m really curious, what do you think? Where do you think we are just on our general trajectory as learning professionals and online spaces? Are we immature, growing immature, fully fully mature? If we’re not there yet, what would it look like to get there?
Brenda Boyd (23:22)
Yeah, I I would tend to agree with you that we’re still growing as a profession because everything is changing, right? So we still see the same cycles of theories and so forth coming around again and again, but we also are seeing new technologies coming in all the time. And that is, I think, the learning designer’s superpower, and it’s also the
the ball or the you know the ball and the chain around their leg because it’s
Jason Gorman (23:53)
Mm.
Brenda Boyd (23:54)
it’s such a it’s it’s a challenge right to to s to stay current and it’s an expectation to stay current. So there’s two cut sides to that coin. But I I think that we are growing and what has happened since the pandemic is that all courses are digital now. There’s not any course
really that’s out there like there was pre-pandemic that was just come and sit in the classroom and we will do things in the classroom. I don’t think that exists anymore because I think institutions have at least a syllabus online, like faculty have to at least put their syllabus online. They have to and they’re and they have s used these tools now. They because they were they had to and now they see the value in how those tools can support their students learning. But
I think that there’s it’s going to continue to evolve and we’ve come such a long way, right, since the late nineties, early aughts when online came on board, right? That there were there were people teaching online in the nineties, but they were mostly doing it via email. And so now we have these now we have these learning management system platforms and we’ve had them for a long time. And it’s time for them to evolve too, right? Into
spaces that I think enable more interaction because we have students that are content creators and that’s what we ultimately want. We want them to have the ability to come in and create their own to create their content, show their learning. And I think the one of the things that has been turning in my brain for the last year is around assessment. How do we enable students to be assessed without
you know, the five page the five paragraph essay, right? Like the things that AI can do, like we don’t really need students to do that anymore. We need them to do other things to to show what matters to them and and and I’m hearing too that they’re rejecting it too. Like some of them are
Jason Gorman (26:00)
Mm-hmm.
Brenda Boyd (26:00)
like using it to write their paper, but but even as a student myself, I thought
There have to be other ways to assess this. Like, why do I have to write a paper for every single class except for a photography class? And I can communicate visually in my photography class. Why can’t I do a photography project for my physics course? Right? Like there there could be other ways to do that expression. And we all are learning on YouTube, right? Like there’s we’re all learning how to do like I’m learning how to put cupcakes on YouTube sometimes. And so I think that there’s there’s
so much knowledge out there and I think the challenge is gonna be to separate fact from fiction moving forward too. That’s gonna be a really big important thing for learning designers as a as a challenge and an opportunity.
Jason Gorman (26:51)
So is it so is it that you think that maybe we’re not quite there yet because the world keeps changing around us so quickly, specifically the technology? do you think I I don’t know, you didn’t say this, so I’ll I’ll just ask, is there a is the way to bridge the gap to have standards that clearly withstand technology change?
Or
Brenda Boyd (27:14)
Well
Jason Gorman (27:14)
or is that part of it? Is that all of it? Or is that none of it? Mm-hmm.
Brenda Boyd (27:17)
I think that’s big piece of it too. You know, when
I was at QM and AI busted on the scene right after we released the seventh edition, right? And so we didn’t have a we didn’t have the words artificial intelligence or AI in the rubric anywhere. And and so I had I talked I did a couple talks about this around how it’s in there. It’s called technology, right? Like it’s the standards are about like are the technologies current.
Are you using, you know, and and then you have policies, right? So policy could be about AI use, for example. So there those examples were not in the annotations in the QM rubric at that point. But right before I I separated from QM, I I did add them in. That was a one of the things that I did right before I left. But I think that the that when you write a standard, it does have to last that
that technology stress test of across how across how technologies are going to change. In those examples I’ve watched them evolve over time. Like tools and technologies that you use in your course, what kind of are you using the right tool for the job? Like you don’t want to have use
one type of tool when another tool would be a better connect, you know, a better way. Like if you’re trying to meet an objective and you want students to discuss something, a a quiz probably isn’t going to be the way to do that. You’re probably going to want to have some kind of forum for them to talk to each other or have a discussion. So I that and that’s just like a really basic example because there are so many amazing tools out there that I don’t think we leverage enough either, like simulation tools. And the barriers have been really high. And
because the learning designers have to learn them, the technologists have to create the learning in them. And but I have also seen when they are used well, they are transformative in student learning. So that you’re not able, you know, you can’t like mix these chemicals at your home, but you can use this simulation to do it and see how what you’ve mixed will come out in a in like a chemistry course. But I think that
you know, the th that’s not ubiquitous. You know, like we could have more technology use. There’s I think that the bandwidth had been a real barrier for a lot of things too. And right when Zoom came on, right before the pandemic, people were able to finally stri you know, have video in real time that wasn’t glitching and
Jason Gorman (29:53)
Mm-hmm.
Brenda Boyd (29:53)
and was much more stable. So I I
You know, I I think that having a standard around the technology is important because the technology’s gonna continue to change. It’s gonna evolve, right? Like there’s we didn’t have cell phones when this all started, right? So there were and creating a standard that enables d a a learning design to be on a mobile phone, to be on a laptop, like we designed always for laptops and then
Later on it was like, well, students are taking courses on their phone, and especially during the pandemic, like so many students only had a phone. They didn’t have laptop loan programs or anything like that at schools yet. But a lot of them have that now. So but these are all like considerations like necessity is the mother of invention, right? So that’s what has happened to a lot of schools with how they’re approaching using technologies, but
But having some s a a standard for technology is is just writing something that means that it’s it’s current and it does the it helps meet those objectives, I think is is really the bottom line with with any technology, regardless of what it is. new technologies are always gonna come along and I think AI has just been so different.
than anything else we’ve seen before, that it’s it’s been a change and that we’ve all had to wrap our mind around at the same time because there was no advanced copy issued to certain people and not to others. So
Jason Gorman (31:31)
Well, I I would imagine that when you go into an organization that you might find lots of different states that they could be in, right? You could find
Brenda Boyd (31:41)
Yeah.
Jason Gorman (31:41)
a team that is really well aligned on what they’re doing, that they are pretty consistently meeting standards, they have a common way of working, they understand what success is, they understand how to get there, and then other teams that might not have that. And I guess I I am
I think about the individuals across what we do, and whether or not I think QM is one of a few examples of clear standards of good work. I think outside of that, I’m not sure that there’s a whole lot for the average practitioner.
To hold on to to say, well, if I’m if I’m doing things in this way, or if I’m hitting these kinds of if if if the quality of my work is hitting these targets, then the work is good. Or these are the skill sets that I need in order to do the work. I think that we have not really settled on those things. And I think unlike something like the legal profession.
Where it’s really specific what you need to know and what you need to be able to do and what the thresholds are for certain levels of progression in your career. I don’t think we’ve quite gotten there with with our work. I I’m I’m curious to talk more about the standards themselves. You have said to me in a previous conversation that poorly written standards read like a stick and not a carrot.
Brenda Boyd (33:18)
Yeah.
Jason Gorman (33:18)
And I’m kind
of curious if we’re going to continue to level people up, level organizations up, how do we write a good quality standard?
Brenda Boyd (33:27)
Yeah, that’s a great question. Because sometimes a standard is sounds like a policy instead of
Jason Gorman (33:34)
Mm.
Brenda Boyd (33:35)
a a standard. And and I think the difference is like there’s a standard, right? There’s a standard of quality. And then the policy is like how you do it, right? And the the policy, I mean the the the standard has to be the same, right? Across a whole
set of things. You can’t have like one standard for one discipline, a different standard for another, unless it’s like discipline related. And so the Quality Matters rubric was created to be discipline agnostic, right? So it didn’t matter what institutional what what discipline you were in, the standards would still apply. And what I think scares people about about standards in general is
that it’s like a compliance exercise. Like you have to do this. And what I think gets lost in that is that they’re probably already doing most of the things that are in the standards. And that it it it’s a welcoming thing. Like the like so the course grading policy, right? That’s a standard it’s found in pretty much every rubric. Like you have a clear
Policy, it you have a clear grading policy, you’ve stated up front, it’s consistent throughout the course. That’s a QM standard. OLC has one about the grading policy, Oscar has one about the grading policy. So there’s so having a grading policy doesn’t mean that if you’re doing ungrading, for example, so you’re doing ungrading, you’re not applied. I don’t want to assign grades, I don’t want to say that this is an A, this is a B, this is a
That’s okay because that’s your policy, right? So it’s still you can still say that this is how grading will be handled in the course and that’s still a grading policy. Just tell the students, right? So that was something that I think came up a few years ago when the ungrading movement was happening and there were a lot of people who were saying like, Well, grades are stifling students from learning. That’s it’s a barrier and I can see that, right? Like I
took many a photography course. I had no idea what my grade was because I was so because my attention was to be focused on my work, right? And that I was developing this practice and I was doing I was consistently creating work and did and in and improving my eye, right? And discerning in that editing eye and discerning what was a successfully communicating an idea and what was not. So I definitely understood
the ungrading movement and that it was not something that was specifically like is a pushback against grading rubrics and then all rubrics and it’s like, but you have to have some standard or people aren’t going to understand where you’re going, right? So so having just outcomes like these are the things that we expect you’ll be able to do when you finish this course, having that written in a way that can be measured
It makes it easier for the faculty member. Rubrics, grading rubrics make things easier for faculty because two a person, every faculty member I know loves to teach, hates to grade, and they don’t want to figure out how to, you know, how do I do that assessment f of learning that it’s it’s very hard to do that, but if you have clarity up front about how it’s going to work, that’s that’s a good example because
Jason Gorman (37:07)
Mm-hmm.
Brenda Boyd (37:07)
it’s not
a it’s not a stick like you’re gonna have to have you know you have to turn grades in at the end of the term. It’s not something that you can do that.
Jason Gorman (37:13)
Sure.
Brenda Boyd (37:15)
And and I, you know, I did for a time work at an institution that had narrative grading. And I was in rooms where nar the pros and cons of narrative were discussed because students who had narrative grades had a hard time getting into graduate school because
Jason Gorman (37:33)
Mm.
Brenda Boyd (37:35)
The graduate school didn’t know how to what to do with a narrative evaluation. They could read it, but it didn’t translate into their system. So I think that there’s there are pros and cons to everything that you do, every choice you make and and and so forth. And I think that the incentive there has been for me and for many people all about like helping students be successful, helping people s succeed and reach their goals.
And that may look different in different contexts. But, you know, a standard is just saying just tell what you want it, you know, what you’re doing. And so that’s and and be consistent with it. Don’t change it in, you know, week six of the term and and change it to something completely different because that’s gonna confuse everyone. And so I think having a you know, a a carrot, you know, turning a standard into a carrot is like
because this is a good thing. And that vegetable narrative metaphor could go in a lot of different directions, but it’s a it’s a positive for for the faculty member and for the students. So how you know make it transparent how you’re going to evaluate their work. How are you going to show them how, you know, this their work is going to be evaluated because that’s the job that those grading rubrics do. They s and then I I use them all the time in grad school. I’m like
Okay, what’s the exemplary column? That’s what I have to do, right? Like it just made it just told me how to how to earn a high grade in the course. And it makes it transparent on both ends. Everybody’s understanding the same thing. And that’s a win for for students and for faculty. But that doesn’t mean that it’s that everybody should use them or has to use them. They do them the way that they need to do it.
Jason Gorman (39:23)
In in the conversation we had before, you talked about the National Accreditation Association’s AI hub. And you described it as a tool that pulls data from multiple sources. So institutions only have to enter their quality information once. And I’m curious what that signals to you about kind of where we are in this moment with how quality is being
stored, referenced, leveraged, used, and what it might look like for the future.
Brenda Boyd (39:56)
Yeah, that’s that’s a great question. You know, the National Accreditation Commission is doing an amazing job. It’s a programmatic accreditor for workforce education. And they are working with three of the institution accreditors with WASC and Middle States and Northwest Commission on Colleges and Universities, and those accreditors are are helping their institutions by having this AI hub that’s gonna enable students
to understand to figure out like what short term program can they take that’s going to qualify for workforce Pell that they can take from their local school? Like can I go to this school and take this program and and if I do that, what will will I be able to get a a job? Like does it have a good placement? Are there jobs in my area for it? And can I make a good living? But it also on the institution side helps them not have to like submit this
some stuff to the programmatic accreditor, some stuff over here to the institution accreditor, and and then also do the work of figuring out what does the Bureau of Labor Statistics say about this particular job and its earnings. And so NAC is doing this work with the AI hub that’s going to pull all of this data in from iPads and from the Bureau of Labor Statistics and then even contextual regional data.
And so it’s it’s really exciting to me because I think it’s going to help people get workforce Pell funding and it’s gonna help institutions stay in compliance with workforce Pell. I learned that if they’re not in compliance with Workforce Pell, which includes like seventy percent completion, seventy percent placement, if they’re not doing those things then they will lose workforce Pell for two years. So it’s super important for s for schools that are
you know, to to do this work and workforce Pell will be coming online I think in a about either in the fall or in the spring. So it’s it’s coming up pretty soon. And so I’m excited for the work with National Accreditation Commission because this is so akin to the work that I did at QM where we were helping schools prepare for the accreditation.
And it’s going to I think there’s a lot of schools out there that don’t realize that they have workforce programs already, you know, like they may have a certificate program, but they could be building in digital credentialing into programs and when they do that with a digital credentialing that they’re going to qualify for workforce Pell monies. And this and this AI hub is gonna help them pull that together. And there’s a great demo out on YouTube about it that
people can watch, it’s like 17 minutes long, but it’s I think that it it’s the thing that I think is really exciting about it is that it’s kind of creating a new model and it’s gonna help schools not have to be so redundant because now we have technologies that can make these systems talk to each other and and to you know put it all in dashboards for them.
Jason Gorman (42:59)
That’s incredible. And the thing that you just talked about around workforce Pell and the thresholds to get funding that sounds like it’s forcing a measure of ROI for students.
Brenda Boyd (43:13)
Yes.
Jason Gorman (43:15)
this is something that education we struggle with is a very tough question for us because how do you measure something where often the the ROI for the student is a a decade or more later?
Brenda Boyd (43:28)
Yeah.
Jason Gorman (43:30)
But
how in your work, how do you think about measuring ROI?
Brenda Boyd (43:34)
Well, you know, it’s I think measuring ROI is really contextual, right? So for like for Workforce Pell, they put these these put these guardrails on seventy percent completion, seventy percent placement. So that’s you know, that’s the ROI right there. So that means seventy percent of the students are finishing the program and then seventy percent of them are getting jobs. So in that field. And you know, and that those are high earning
jobs, they’re jobs that have that there’s high need for. So there’s a lot there’s, you know, parts to that. But I think too, like, even in my work with with with organizations with practice learning, like the what is the return on investment? Well it’s it’s usually going to be an improvement, right? Like an improved experience. Because quality isn’t like a thing that you’re just gonna get it, right? Like you’re gonna get to quality and then
Here we are, it’s a continuous improvement journey that’s gonna go on for years. And that’s I think one of the hardest thing is is having that is is living in a continuous improvement culture where you’re figur you’re trying things out and you’re seeing if they work or not. And then if they don’t work, then you’re like, okay, what didn’t work? And then and then really reflecting and
and doing those kind of deconstructing and figuring out what didn’t work and how can we fix that, improve it, and improve our whole system overall and move forward and make it better. So ROI is is going to be contextual for every everyone that I work with for and for every school, right, too, because they may want to lower their job failure withdrawal, their DFW rate. They may want to lower that. They may want to increase grades in a particular and it and they may want to
in sh you know, increase their completion rates and ple include in achieving the dream does lot of this work too, which is improving outcomes for for minoritized students to how so that their pathways are open and that they’re able to be successful in programs. and I think that the so depending on the goals
So every school has goals. They all have a mission. They all have strategic plans. They all have ideas of where they wanna be, what they wanna be known for, how they wanna and so anything that can help and I think quality is one of those it’s kind of like it this intangible. and I’m gonna be launching a podcast to talk about that too. So I hope people will turn into my podcast
when I launch it here in about another month or six weeks, it’s gonna be called the quality difference and we’re gonna be talking about all this fun stuff. Like what is what does
Jason Gorman (46:26)
Excellent.
Brenda Boyd (46:27)
quality really look like? And so if you want to take a deeper dive, I hope you will check that out. But but that return on investment is something I want to talk to people about too because it’s hard to measure, but you have to have your data up front. You have to know where you’re going. What is it that we’re trying to move?
Are we trying to move the needle on student grades? Our completion rates, or what is it that we’re trying to do? But having good data is the basis for all of this. It’s the foundation. And if you don’t know where you’re going, anywhere will do, right? So I think
Jason Gorman (47:01)
Mm-hmm.
Brenda Boyd (47:01)
having an idea of of what is it that we’re trying to achieve and starting off with where you’re at now. And that’s
And and that can be a matter of pulling information from a lot of different departments, a lot of different data points that are there. Learning management systems are chock full of data. There’s student information systems, there’s the institutional research departments and institutions, there’s business intelligence and organizations. So everyone has data. So figuring out what is it that’s most important to us and how do we measure things to move how what is what do we want to move the needle on and how are we going to measure that?
That can be a real return on investment.
Jason Gorman (47:41)
That is so reassuring to hear that there are all of these existing standards, standard ways of thinking about return on investment, both for the student who’s going, who’s you know, investing their time and money to get a degree or certification in something, but also for the institution as it thinks about how to spend its money on how
Brenda Boyd (48:04)
Mm-hmm.
Jason Gorman (48:04)
to improve what it’s doing. I haven’t
I’ve been familiar with some of those things, but not all of them. And it’s it’s great to hear that those things exist. things definitely will keep changing in our field. And I’m curious, we just talked about this this AI hub, and that sounds like a pretty exciting change that is is going to change how we think about quality in the future. But do you have any
Predictions about where things will go in the next few years.
Brenda Boyd (48:38)
Yeah, I think that we’re going to continue to see more of this embracing of building out like now we have this technology, right? Like we don’t have to like look to these big tech giants. We can still we can build our own like knowledge bases and we can build our own language models because that underlying structures are yes, they are from big data, but they’re also
There are there are ways of working. And I think as we begin to discover ways that we can create things that we’re going to we’re going to be able, you know, like people today can build things out that it used to take a whole development team and you can develop something in using AI. So I I’ve created calculators in AI. I’ve you know, I’ve seen some really interesting proofs of concepts in AI that were built out with AI, but
So I I I’m I’m a little nervous about making any big predictions because they they don’t often bear out because you know the adoption curve is like we’re still like at the very beginning of AI. Like the the we’re
Jason Gorman (49:43)
Amazing.
Brenda Boyd (49:44)
the early adopters. Like there’s we’re definitely I don’t think it’s mainstreamed at all yet. And I I don’t think that we’re even in the trough of disillusionment yet. I think that we’re
You know, or maybe we have gone through that with the first rounds of where there were hallucinations and we’re like, This isn’t gonna work and then it’s like, Okay, now it does. But but I think we’re you know, the we expect that these models are gonna improve and that’s the promise and that’s the hope, but and I hope that they get better too, but there’s no guarantee that they will. So, you know, we are
So I think that the best prediction is to be pragmatic and to continue to you know think about how the you can use these tools and d and I think that it what’s gonna matter, what’s my big prediction is I think that people are gonna have to develop more literacies in terms of of how to use tool how to use tools like AI in a response ethically responsible way.
as well as I think that people are also going to need in to develop discernment more more so because that’s going to be so important for you know determining like there’s games out there right now like is it AI or is it a human did a human write this or did AI write this? And so
Jason Gorman (51:03)
Mm-hmm.
Brenda Boyd (51:04)
you know that level of discernment is something that we’re all going to have to to rise up to and
And I I and I think I can safely predict that community is gonna continue to be really important for us as humans to connect with one another and to be and to be together with other people who are learning or like minded or who are not like minded and we want to learn from them. So I think that there’s I I think that connection is still gonna be really important and as our future unfolds.
Jason Gorman (51:39)
Brenda, I have to admit, that’s the reason I do this podcast. I do this podcast. It’s the reason why I want to have these conversations. So, all right, check, check for today. We can check that off our list. Just
Brenda Boyd (51:49)
Mm-hmm.
Jason Gorman (51:49)
a couple more questions. this is a little bit of one from the side door. So we kind of take inspiration from a lot of different places in our lives for our work. And I’m just kind of curious: like, what’s something that you have borrowed maybe from an unusual
source or place in your life that you do use in your work.
Brenda Boyd (52:11)
Yeah. I I think, you know, I was a server for a long time and I really enjoyed that job because I was feeding people and having helping them have a good time. And and I think that that really helped me with you know, the relationship aspect of work and and how
Everyone is different. So early in my career when I was working with faculty members, one-on-one, turning their face-to-face courses into online courses, it really drew on my experience as working in the hospitality industry because I everybody was different and they all had their different approaches. And I think even like when you’re working with the general public and retail or restaurant or whatever it is.
you learn to adapt to that person and their needs as you’re helping them. And I think I think that that’s kind of an unusual connection that most people wouldn’t make, but I also think that most people who have have been a server would say everyone should be a server and have that experience because it’s a it’s a job that doesn’t
it it’s a hard job but it’s also a rewarding job and and I think that it it can be you know really fulfilling for for people but it’s also so when you when you go out be sure to tip
Jason Gorman (53:39)
Yes, definitely. Just not your SME. Do not never tip your SME. You want to do that. But I I I
Brenda Boyd (53:43)
Yeah.
Jason Gorman (53:44)
take your I take your point though. The just that like needing to be a little bit quick on your feet, coming into a new interaction, being very open to the other person, getting a sense of where they are, calibrating to that.
Brenda Boyd (53:59)
Exactly.
Jason Gorman (54:00)
I it’s something that we can’t do our job. anybody who does anything
Creating online learning cannot cannot not do that. It’s absolutely essential. The last question here is from my last guests. Their names are Ram Iyer and Anand Shekhar And their question is: how do you help a person build critical thinking? And what is the role of critical thinking now in the AI?
Brenda Boyd (54:29)
I could have like planted that question. It is really a great question. And I so
Jason Gorman (54:31)
It’s it’s really perfect. It really worked out. Yeah.
Brenda Boyd (54:36)
I think how we build critical thinking is we teach people how to unpack things that like things like misinformation. We teach them what a good primary source is. And this isn’t something that I think should just be limited to college courses. I think everyone needs these skills.
at this point. Critical thinking means looking at something and not just accepting it right away and saying, well they said that the so that’s the way it is. It’s about unpacking that. And I think we need to question and look at look at things from different points of view and and consider how something might be different if conditions were made different, right? So
I think you help a person build critical thinking by helping them solve problems or putting them in situations where they have to solve a lot of problems. I come from a generation where we were free-range children and we went out and built forts and if our chain came off our bike we had to figure out how to put it back on, right? So so I think that building critical thinking means that you have to you have to be put into situations that push you a little bit.
and make you make you think about how to solve a problem in different ways because there’s often not one single answer, right? Like there’s often many ways to do something and do it well. but I think the role of critical thinking now is to be that to is to be discerning, is to be the person who can say, okay, well, even if AI cor correct, you know
created something, it can we have to have that discernment to determine is it good or is it not good? Like is this quality, is it not quality? And and my colleague Chris Sachs created a she created this great guide on evaluating the quality of AI outputs. And I think that that’s a it’s a super valuable tool because we do need to have
those questions. You know, and there’s there’s different frameworks for like evaluating media sources and evaluating information sources. And we need those kind of tools for AI too to see like because AI slop is a thing, right? And that’s just like it’s AI just being AI. It just does its own it just this is ha this is the default. And default settings don’t necessarily mean that they’re the best. So
we have to push for them and we and we’re all going to be learning together how to use our critical thinking skills to get out of AI what we need to serve what we’re trying to do. and instead of, you know, using it for automation to take away people’s jobs, how about we teach those people how to use it to do their job better, more efficiently, be able to be more productive
and work fewer hours, for example. So I think that the critical thinking can can really propel us forward as humanity in general.
Jason Gorman (57:46)
This has been a lovely, wonderful conversation. I’m so glad we did this. And I I really appreciate thinking in terms of quality because I think the the place where I’m in now, so many people are now, is this place of disorientation. I think there is something so grounding in everything that you’ve been talking about today. So I just really appreciate your perspective.
Really appreciate you coming on. Thank you so much for the conversation, Brenda.
Brenda Boyd (58:16)
Thank you, Jason. Thank you so much for having me. It’s been a wonderful time.