When Brick Maier talks about learning, he starts not with metrics or models but with a memory. He remembers kneeling beside a sixth grader’s desk in a classroom in Los Angeles, seeing that look of understanding spread across a young student’s face. That moment, he says, is what set everything in motion.
“The interaction I got to have with my students was that thing that set in motion why it’s important to design these experiences and to help people grow.”
Two decades later, that same drive still defines his work, only now the classroom is virtual, and the students are adults learning about renewable energy systems. As Director of Enterprise Learning and Development at HeatSpring, Brick is helping to shape how thousands of professionals prepare for certifications and careers in the clean energy sector.
Listen to the podcast on your favorite directory:
From the Classroom to Clean Energy
At HeatSpring, Brick oversees a vast catalog of around 350 courses in solar, geothermal, battery storage, and more. Half are free, half are paid, and all share one goal: helping learners master complex technical skills that have direct real-world consequences.
He describes HeatSpring’s learners as “builders of the new grid”, adults who come from all walks of life, some reskilling for clean-tech careers, others deepening their expertise. The challenge is not just to teach but to design experiences that truly work for them.
“We have folks coming in from all different pathways,” he explains. “We want them to succeed, to pass that certification test, to perform better in the field, and to see themselves grow.”
This isn’t theoretical learning. It’s education with boots on roofs, hands on equipment, and outcomes that matter to livelihoods and industries alike.
Designing from Reality, Not from Scratch
When Brick collaborates with subject matter experts (SMEs), many of them are practitioners, not teachers. They have decades of field experience but little time for instructional theory. So instead of starting from a blank page, Brick meets them where they are.
He helps them think about how a learner might experience their expertise, from linear lectures to field footage, from peer conversations to software demos. This modular approach allows each course to blend structure with authenticity. “The idea,” Brick says, “is to help them see they already have access to all these different ways of teaching, not just the PowerPoint.”
For the learners, this variety means the lessons feel alive. They can see a combiner box being opened, a heat pump installed, a diagram come to life on screen. The gap between learning and doing starts to close.
Lessons from Amazon: Data and Discipline
Before HeatSpring, Brick spent seven years at Amazon in learning design and knowledge management roles. That experience left him with a clear conviction: if you can’t measure it, you can’t call it learning.
At Amazon, he learned the rigor of linking learning outcomes to measurable behaviors, not vanity metrics. He also saw the chaos that emerges when knowledge systems lose coherence. “We had two glossaries defining the same acronym differently,” he laughs. “That’s how trust erodes.”
He brought those lessons to HeatSpring, balancing Amazon’s data-driven discipline with the flexibility of a smaller organization. It’s a space where rigor meets pragmatism. “We don’t add rigor where it’s not required,” he says. “It’s a battleship versus speedboat situation.”
The AI Breakthrough: Reclaiming Time for Real Teaching
For years, one of the hardest parts of course design was building assessments. Writing good questions, the kind that actually diagnose understanding, takes time. Too much time. Then AI changed that.
Brick piloted a simple script on his computer that analyzed course transcripts and generated draft quiz questions. After refining the prompts and testing with instructors, the results were remarkable.
“We saw about a 90 percent reduction in time.”
The AI-generated questions weren’t perfect, but they were good starting points. Instructors could edit, improve, and deploy them in minutes instead of hours. That efficiency meant more formative assessments across courses- not just final exams, but regular pulse checks that help students learn as they go.
Students now engage more frequently, instructors save time, and learning data becomes richer and more immediate. It’s a small shift with a large impact, one that wouldn’t have been feasible without technology as a partner.
Adapting Without Over-Engineering
Even with all the automation and AI tools available, Brick is careful not to overcomplicate processes. He tells a story about a former Amazon colleague who built an elaborate system in Asana, full of triggers and automatic notifications. It looked brilliant until every new task generated a flood of emails.
“We realized we’d automated ourselves into chaos,” he says, laughing. “Automation should simplify, not overwhelm.”
That philosophy guides his work at HeatSpring: build just enough structure to ensure quality, then stop. It’s a lesson that applies not just to software but to learning itself. Sometimes progress means dialing things back.
The Future: Probabilistic Learning and the Curious Student
When asked about the future, Brick hesitates to make predictions but offers a thoughtful warning. AI, he says, isn’t just changing how we design learning. It’s changing how people experience it.
Soon, learners will have AI agents that can interact directly with course content asking for clarifications, diving deeper, even generating side explanations on demand. That’s exciting, but it also means the learner experience will vary widely.
“We’re heading toward probabilistic learning,” he explains. “Not everyone takes the same path, but they all reach the same destination.”
The challenge, he believes, will shift from designing a single linear experience to designing systems that can adapt dynamically while still being reliable, safe, and accurate.
He tells a story from his time at Amazon about a veteran engineer who became an informal mentor to his team. Some people would come to him with the same question over and over; others returned each time with new, sharper questions. The difference, Brick says, was growth.
“The people who grow ask a different set of questions every time.”
That, to him, is the core of future learning- not just knowing how to use AI, but knowing how to ask better questions because of it.
Evidence Over Enthusiasm
For all his curiosity about technology, Brick remains grounded in proof. What matters to him is not the promise of innovation but the evidence of impact. Are instructors using formative tools? Are learners engaging with them voluntarily? Are skills being applied better on the job? Are outcomes improving?
If the answers are yes, then it’s working. If not, it’s just hype.
In an era obsessed with acceleration, Brick’s approach is a reminder that learning, at its best, is still about connection, reflection, and craft.
Listen to the full conversation with Brick Maier on The LX Ops Show or wherever you listen to podcasts
Podcast Transcript– Jason Gorman with Brick Maier
Jason Gorman (00:00)
Hello and welcome to the Learning Experience Operations Show. I am very excited today to be joined by Brick Maier. Brick has extensive teaching education, and technology experience, and currently he is the Director of Enterprise Learning and Development at HeatSpring. Welcome to the show, Brick.
Brick Maier (00:16)
Hello Jason, it’s great to be here.
Jason Gorman (00:17)
Great to have you here. So I want to start with a simple question, but it’s possibly the most important question. You’ve been doing this work for a little while, but what was the thing that got you really interested in doing this? What sparked your passion and what has kept you into it?
Brick Maier (00:33)
I probably go fairly far back in my career where I started as classroom teacher in inner city Los Angeles, where I was a volunteer teacher for two years as a middle school teacher, self-contained sixth grade classroom, taught all the subjects, and was really young and didn’t know a lot about pedagogy, teaching, kind of just making mistakes every day, trying to learn.
how to teach and be a teacher. But the interaction I got to have with my students was that thing that set in motion why it’s important to design these experiences and to help people grow. think when you’re in a classroom and you’re working with students, you can see the growth and you can see them interacting with each other and you get to build trust. And there’s a lot of feedback that’s just happening right in that moment.
And if I take that and fast forward to 2025 and distance learning and online learning, we try and create a learning environment in a classroom. But we’re measuring those things through different signals that I got firsthand in person, kneeling next to my students desk when they got something and they understood something. And I was like, okay, great. And then so that spark for me
is what’s kept me going. 20 odd years later, I’m still fascinated by helping people learn. I just have a different toolbox for measuring how they’re learning. honestly, like the, the stat spark of excitement looks a little different for an adult learner than a sixth grader, but, ⁓ still the same, fundamentally.
Jason Gorman (02:05)
Yeah.
Yeah. And so maybe you could walk us through what your work is like now. What are you doing on a day-to-day basis? Who are the learners that you’re serving? Tell us a little bit about what you do.
Brick Maier (02:17)
Yeah. So HeatSpring is an online learning platform primarily for renewable energy. And so we work a lot with workforce development and folks getting certifications in solar or building professionals, BPI certifications. We have a number of nationally recognized certifications that our courses people prepare for passing tests and getting certified. And so
They’re adult learners. have about 350 courses, half of which are free and then half of which are paid and bundled and are really rigorously designed to prepare them to pass a nationally recognized test. And my role in that is working with a lot of instructors and subject matter experts on the building out the course and building out the assessments and helping them
hone their expertise into a course that can be delivered and picked up by a student so that they successfully pass that test. Or someone who’s curious about clean tech, renewable energy. I mean, we have folks coming in from all different pathways and a lot of those folks might come in at our free course offering building out fundamentals in solar or geothermal
heat pumps, batteries, like we have a whole array of courses and each of those has its own outcome based on what our students want.
Jason Gorman (03:36)
And it sounds like you are involved in everything from really determining what the offerings should be to match the need to working with subject matter experts to creating these programs. Is that accurate?
Brick Maier (03:50)
Yeah. I work with Brian and Brit and others at HeatSpring to determine the mix and that kind of forward looking like course development pipeline, like where it takes us a while to build a course and or build out an offering in a certain area. And so we talk about where should we focus our efforts or build out. And then, yeah, then I get into the block and tackling of like, all right,
Here’s an instructor with a certain expertise. They have this body of knowledge or materials that they’re ready to turn into a course. And I work with a wide spectrum of experience in teaching. Some haven’t taught at all, and this is their first time. And so I provide a lot of scaffolding on course development. Others have a lot of experience delivering instructor led training in person. So they might have a lot of slide decks.
But this is their first time translating that into an online offering. And then I’ll have another subset of folks. They’re like, got courses on all major platforms, total pros. And I’m mostly just walking them through our system and giving them tips and tricks. So I have to adapt to what their background is.
Jason Gorman (04:54)
I would love to learn more about your system. And I think maybe as a way to frame that, because I think the way that we work now has everything to do with the jobs and the roles we’ve had in the past. So I’d be really curious to hear about some of the jobs you have had in the past, some of the roles you’ve had, and what the critical mindsets, processes, tools
that you developed and are really part of the way that you do your work, what are those things and how do you bring those things to bear now?
Brick Maier (05:24)
Prior to HeatSpring, I spent about seven years at Amazon as in either knowledge management or learning design in various teams within the monolith, which is Amazon. I never worked in delivery. So whenever I told people I worked at Amazon, they’re like, I just got a package from… I was like, I was somewhere else. I didn’t touch delivery at all. So I most recently came from AWS, which is their cloud offering.
Jason Gorman (05:38)
Mm-hmm.
you
Brick Maier (05:52)
and worked in learning a design there. But in that time, and then prior to that was a brief stint with Microsoft helping build out a program that piloted a new piece of storytelling software within a creative writing education context. And that was an interesting period because it was short term, but it served as a bridge from when I was primarily a filmmaker and teacher.
of filmmaking to teachers. So I spent time in LA training about 500 teachers how to bring filmmaking and animation into the classroom with off the shelf everyday accessible tools, but centered on reading and writing. So I did that for a number of years, even had a Kickstarter, sold kits, like was an entrepreneur, had a website, package shipped, all these things all over the world. But then that came to a natural conclusion and I
started entering into digital storytelling with this Microsoft project. And I started to see like, you can transfer storytelling, writing, critical thinking in this way. And then that ended up and I was like, okay, I worked with a major tech firm and it seemed to go, okay, I’m going to try and like work at a big tech place. And so I applied for Amazon, got that. And I will say that the first few months at Amazon felt like
imposter syndrome, because I came as a teacher of record. My background is a teacher, filmmaker, working with nonprofits and doing this kind of work. And then I felt like David Attenborough is narrating my walking into a conference room he doesn’t know where to sit or look. He’s not quite sure if he should be typing in this meeting or even at his laptop. You know, it just felt like
Jason Gorman (07:10)
Mm.
Ha ha.
Brick Maier (07:30)
there’s going be a red button that they hit and that the chutes can open and I’m just going to drop through the floor and they’ll be like, sorry, you know, we, we, this was all a test and you didn’t pass, but that was just kind of typical of being new to an environment. And I realized that the critical thinking, looking at something from an oblique view, the creative storytelling was something that was unique that I was bringing to Amazon. And then what I learned at Amazon. And it was like drilled in was like the data driven approach to.
Jason Gorman (07:43)
Mm-hmm.
Brick Maier (07:58)
Um, is what you’re designing, building out the skills that you said it was going to, are there learning outcomes you said that are happening at the front, measurably demonstrated at the end. And that was really there. That was the litmus test for things we built there and that kind of data driven approach, like meaningful data. I don’t know. There’s sometimes you can say data and it’s like, you have a wall of numbers and it looks like an eye chart and you’re saying 7 % better.
32 % faster and 4 % less errors. And I think there’s ways to have word salad of data, but the actual what are we trying to accomplish with this learning experience? How will we know it’s going well midway through? And then once they finished it, what are we looking for two or four weeks out in their application of those skills that it’s measurably better than that. And doing that for six years across teams.
Jason Gorman (08:28)
you
Brick Maier (08:49)
And hundreds of projects helped me burn that in. think I bring that to heat spring as well. It’s just a data-driven approach and assessment-driven approach. The questions that you ask to identify gaps in knowledge are what will reveal whether someone understands it or doesn’t. And so we’ve learned that the quality of the questions you’re asking and trying to get at gaps
are what determine how good the learning experience is. If you’re not really putting formative and summative assessments in front of students to interact with, you’re flying blind a little bit in not knowing the quality of the learning experience.
And that was something that Amazon built in. I worked on two projects that were like internal learning libraries and internal learning universities to build a collection of resources that people could come to. And a lot of that centered on just principles of standardizing the process of making learning artifacts so that we’re all working from the same page.
Jason Gorman (09:47)
Yeah.
Brick Maier (09:49)
So that the library ended up having a similar output and format and just simplifying, just reducing redundancy. Amazon had this internal Wiki that anyone could write to and build to, which was great for speed, but terrible for source of truth. And you’d get two glossaries of acronyms for the same program. And they’re defining acronyms differently. And you’re like, well, what?
Jason Gorman (10:10)
boy.
no.
Brick Maier (10:13)
Who’s is it? And then the program manager who’s left and abandoned out. It’s the same thing for any program. It’s how are you updating it and keep you up to date? that we, I experienced the, what happens when something falls out of date and you lose trust on, the content. So those are some of the things that I think I.
try to bring into heat spring, which is fundamentally different where smaller operation, where 350 courses, we can manage those things a little more easily than something like Amazon, which was just all about scale and speed and adaptation.
Jason Gorman (10:37)
Yeah.
Right. So how are those? I mean, you mentioned lots of great things here. Data-driven, assessment-driven approach, one where you’re using critical storytelling, leveraging your film background, but also your teaching skill, the skills that you had as a teacher in the classroom, understanding pedagogy. These are all things that I’ve written down. Asking good questions.
It sounds like of students, but I suppose also of subject matter experts, right? To help kind of figure out exactly what you’re teaching. And then these systems to run things in an organized way that it just doesn’t get wonky and out of control and chaotic in terms of how things are standardized. So ⁓ how are those things harmonized?
In your work now at HeatSpring, how would you characterize how those come together in your work?
Brick Maier (11:30)
One element that I built out that’s been helpful in my initial kind of kickoff call with a subject matter expert is a kind of toolkit or framework for thinking about a course and trying to meet them where they’re at with the resources they have. And a lot of times,
we work with a lot of folks who are incredibly smart and have built out years of experience in a field where they’re not in front of a computer. they, they’re well versed in software. They can use it. They’re giving me some of these.
backgrounds like they require that they know how to use software, but they’re fundamentally doing their work in the field and they’re on a roof Yeah roof battery, I mean you think about utility scale solar utility but the battery storage with a bunch of storage containers or CNI like rooftop solar or geothermal like a drilling rig is backing into a field and drilling and so
Jason Gorman (12:04)
Yeah.
In the field, Yeah, roof.
Brick Maier (12:24)
this is their expertise. They’ve been on however many sites and they’re trying to train someone that’s like near peer expertise, their peer or someone who’s coming up and they’re trying to build either a fundamental course or an intermediate level two, level three kind of course. And so when I meet with them initially, I’m trying to understand how is your expertise currently captured? Is it just in your head or have you documented it? And a lot of times they have it in a PowerPoint presentation.
or they’re thinking like, I’m going to deliver this in a linear slide deck, chunk it up. And so my help, my framework and what I try and show them is like, let’s start with that slide deck. That’s a really great model. People understand it. We can build toward that with export that as a video and heat spring is kind of a video first streaming platform. So you’re going to interact with a lot of our content through a video. However, there’s a lot of other ways you can deliver knowledge. So I’ll say, okay, let’s start with this.
slide deck, but do you in your pocket, your smartphone, have you ever filmed or captured photos of you out in the field demonstrating something? And this is something we’re kind of pulling in from YouTube really, where someone’s like, hey, what’s up everybody? I’m just going to show you how to do And those videos are incredibly powerful for learning something in the field with your hands looking at, and that’s so relevant for what we do. So I’ll ask them like, do you have that? And then a more formalized version of that, like,
within your company or do yourself have drone footage. What does the field footage look like is on your phone is from a drone. And from there we can start to say can we build some vignettes that go from the slide deck into the field that show the application. And then that way the learner is seeing it both in that linear slide deck form and the messier real world, opening up a combiner box or
Jason Gorman (13:50)
Mm. Mm.
Hmm.
Brick Maier (14:10)
some switch gear and like looking at this panel. And we work with a lot of products, companies that sell products. And so they’ll build kind of customer education courses and they’re in a warehouse and we’ll do that same model. So it’s like slide deck, field footage. And then this format that we’re in now, this kind of podcast style is a really powerful way to unpack expertise through conversation and dialogue that isn’t the main delivery of a course. We recognize that you can really pack a lot into
Jason Gorman (14:19)
Mm-hmm.
Brick Maier (14:37)
lecture notes, deck, but to have a place for anecdotes, reflections, lessons learned, where two peers, and the important thing there is we found and would recommend is they’re on a similar expertise level plane. That it’s not necessarily like mentor mentee. It’s two folks who’ve got decades of experience who are chopping it up. They can go right into it. And that helps.
Jason Gorman (14:57)
Hmm. Hmm.
Brick Maier (14:59)
append that to the end of a lecture. And so we start to kind of like when at the end of that call, I can say, course on heat spring can be the foundational backbone as a slide deck. can be augmented through field footage. can drop in podcast style exchanges where relevant. And then if you’re software-based in what you’re teaching, then we’re going to do a lot of software tutorials, screen capture, narration of how you
design single line diagrams in this specific software. and then if they have a CAD model, we can drop that in, like a screen recording modality. So that’s a way I think about deconstructing a course and presenting it back to them and saying, you actually have access to all these different ways of teaching that are beyond just the PowerPoint that.
Jason Gorman (15:36)
Yeah.
Brick Maier (15:43)
scale and their level of effort. cause some of these things take a lot of time to make others are really quick and trying to be helpful in navigating being the other thing I should say, Jason is that our subject matter experts are not full-time teachers. So when they’re like, I’d like to create a course for heat spring, they’re going to do this on nights and weekends and when they have time. And so I’m trying to present solutions to them that are
Jason Gorman (15:54)
Yeah.
Mm-hmm.
Yeah.
Brick Maier (16:06)
finding that balance of effective pedagogy and not making them crazy around spending. I can’t believe I’m still working on this. Cause if you’re a dedicated learning designer, you could spend, I think ADT says like 70 hours for one hour of e-learning. And that number has come down with AI, but it’s still slow. And I’m dubious of anyone who’s talking about how AI is like a one press button and you’ve got your course and I don’t think so
⁓ cause you you still need a lot of revision and, that’s kind of the balance, what we’re trying to find and getting their expertise out of their head and into a format while managing their real life time constraints.
Jason Gorman (16:34)
Yeah.
Yeah, and it sounds like, you talked about the last point that we were talking about, the last thing that you said was that you learned at Amazon systems to run learning operations. And it sounds like you are starting in your conversations with subject matter experts who are not expert teachers necessarily with a of a packaged system. It’s not a blank page. It doesn’t have infinite options. There’s modularity.
Those modules are arranged in ways that you know work, because you’ve seen them work. You can show them examples of how they’ve worked. And it sounds I like the idea of this being a toolkit or a framework, because it’s something that you’ve done a bunch of times, that their peers have done, maybe possibly more importantly, a bunch of times, that they’ve been successful, and that your toolkit and framework has all the necessary elements in the order that you’d want to have it.
probably would imagine with some flexibility, but it gives them something to get their hooks into, to understand what is this thing that we’re doing and what do I need to do to make it work? It sounds like you’ve created a nice package for them.
Brick Maier (17:42)
Yeah.
Yeah. And it’s, that, that initial overview helps them see what their course could be. And then I’ve got a couple other secondary calls or support resources that are more granular that are like, okay, I’m to leverage some of these new AI workflows to help analyze what you’ve given me. And then I can give back to you a proposed course outline terminal and enabling learning objectives. and these are all new vocabulary terms to them potentially.
They may not know why those are so critical, why I would present those to them as like, we got to think about these things first. And so let’s analyze what’s come back and does this sequencing make sense? And these outcomes, do those match? are these objectives that are kind of embedded into each of these modules, we’re to be working backward in our content to meet those.
Are those right? Because if those aren’t right, then we’re going to wander off and our course no longer measures what we said was going to measure. And then getting into actual artifacts that I start presenting to them in this kind of modified, compressed, simplified addy model. One of the things that I
didn’t bring from Amazon or like maybe a more traditional learning and design team at a corporation was a very formal, rigid structure, like an Addy model. Not to say there’s anything wrong with that, but I have this like battleship versus speedboat tension around how much documentation or rigor do you need? And sometimes Amazon and big companies like, no, it needs to be standardized. It needs to be this way.
Because we’re scaling this out across thousands of employees and thousands of instruction. We don’t follow this. We’re to go off the rails real fast. Heat spring, it passes through me or Brian or Britt. We can see it. And so we don’t need to have as much. We’re a little more flexible and like placing us in the loop. I don’t want to dismiss adding model, but I just, don’t need to add rigor where it’s not required.
Jason Gorman (19:33)
Yeah, I think it’s a really important point because my hope is that there are people from lots of different kinds of organizations who might hear this conversation. Their situations are going to be different. Everybody’s going to be in a different spot. So if you’re on a team of 15 and you have a 30,000 person organization that you’re building internal leadership training for or something, you’re going to need a lot of standardization. But
if you are the one person that does the stuff in the place, then it’s just different. The burden is not as much.
But it does sound even even still though. I mean I would certainly encourage anybody to write things down to whatever extent makes sense and you do have this process you have this toolkit you’re not winging it each time You do have a way that you’re entering into it So I hear what you’re saying because you don’t want to over engineer that stuff because then just writing things down becomes its own
own major project.
Brick Maier (20:27)
I have this
funny anecdote around that. A teammate at Amazon who was learning Asanaand come from the military and then worked within a pretty large system. And so really just had a systems thinking that was more built out than I had ever experienced.
And so she went in and did the most you could do inside Asana to have these auto triggers and logic gates and things. And it was really funny when we’re initially testing it out, like some, one person on our team would hit add task and then so many emails would show up and all these things would happen. And then we’re like, okay, let’s dial it back.
Let’s not have it do all of those things. It was kind of funny.
Jason Gorman (21:04)
Ha
Well that’s a really good point, because you mentioned AI. And I would say when we think about AI and we think about automation, we are in this moment of just incredible acceleration for these reasons and other reasons. But just the world is changing really, really fast. And I think that does put some pressure on you and me and people that do the kind of work that we do to
to update what we do on a regular basis. I feel like we are kind of challenged to be more forward thinking and responsive to the way that the world’s changing. And I’m curious whether you feel there have been any challenges in your work in the last, I don’t know, period of time, recent to medium term history
where maybe something that you had been doing for a long time, you recognized, this isn’t quite going to work as well as it has. Or the opposite, suppose, of that could be, ⁓ this wasn’t really possible to do before, and now it is. And I have to kind of rework some things to work that in.
Brick Maier (22:12)
Yeah, I the ladder of new capabilities being unlocked or the level of effort in generating a really great assessment is really high. If you’re a human writing that question inside word or something, you’re just like, I’m going to open up this document. I’m to write a question and I’m going to let my expertise show up in the form of that question.
Jason Gorman (22:17)
Mm-hmm.
Brick Maier (22:35)
And that’s where that was been a fundamental unlock for HeatSpring in analyzing the content embedded in a course through an AI enabled assessment lens. And that’s something that I built out. I piloted it locally on my machine through a Jupyter notebook and confirmed that my API calls to the LLM were returning something that was usable.
And then looped in our development team to see what I was doing. And they saw the script that I had written and like, okay, we can adapt that logic. And then we tested it in our staging. We looped in a couple of instructors to say we’re, we’re having AI analyze your transcript because we’re video driven and our instructors are following our best practices. Everything that they narrate.
is a close approximation of what’s being delivered in terms of expertise. There’s not gaps. And so we’re going to send that and have it analyzed based on some customized prompts we built to give us back an assessment. Can you look at these and tell us the quality? And we were, that was an interesting point when we were testing this out. This is the kind of a go, no-go situation. Cause if these initial pilot, these instructions, like these assessments are
Jason Gorman (23:25)
Mm-hmm.
Brick Maier (23:45)
wildly off, they don’t work. don’t know how, but that’s not what it came back to. These are great. These are so helpful to save so much time. a lot, and there’d be feedback like, was redundant or that seems a little simplistic. And that was us tuning, what we were asking the LLM to do. There wasn’t an issue of like, okay, we’re going to have to like incorporate Bloom’s taxonomy and give some sliders on complexity now.
Jason Gorman (23:46)
Uh-huh.
Wow. Wow.
Mm-hmm.
Brick Maier (24:08)
And we have to look at stem construction distractors and we’ve honed this thing into a nice sculpture of a prompt and we like it. And if there’s anything that’s special sauce in the age of AI, it’s like our own hard work that we put into that prompt that we think returns really great, consistent results for us. And so that for us has been AI unlock on our platform. We scaled that out. We embedded it into our
Jason Gorman (24:13)
Hmm.
Brick Maier (24:31)
platform, we ran it across our 350 courses so that we didn’t just have it blanket create assessments for every course all across it, but we’ve now enabled an instructor to go in at the assignment level and generate knowledge checks wherever they want. Then they can edit and those can show up right after video. A knowledge check for us is a version of a formative assessment, low stakes, engage with the content, show what you know.
Jason Gorman (24:48)
wow.
Brick Maier (24:57)
Explain why you don’t understand it with feedback for right running and answers. Score it immediately. And that is different from what we were doing previously prior to AI, which is high-stakes summative assessment quiz or test at the end, which are valid. Those are important, but we wish never would have done the formative assessments at this scale without AI. So that’s been a great.
Jason Gorman (24:57)
Mm-hmm.
Because it was
too time consuming before? Yeah.
Brick Maier (25:20)
Yeah, no one wants to write assessment questions. No one wants to do that.
it’s just takes so long. And you’re like, I just spent 25 minutes on three questions or two. And, as opposed to like, we had like a 90 % reduction in time where we would come back and just be like, Oh, here are these three questions. And then rather than them drafting from scraps, they’re now modifying the stem or the distractors and then deleting the ones that weren’t high quality.
Jason Gorman (25:35)
Wow.
Wow.
Brick Maier (25:46)
And the instructors really liked it. Cause it’s way faster. We actually had some who were like, can we export these assessments? Cause they wanted to use them in other contexts. And we haven’t quite figured out how to bulk export those yet, but I also took that question as an endorsement of quality. They just wanted to use them elsewhere.
Jason Gorman (26:02)
Definitely.
So usually my next question is, and a lot of times this is hard to answer in our industry, but what’s the measure of ROI or success? There’s a clear measure. is it 90 % time savings? It also sounds like you’re out of the middle of that process. If you had been sitting down and writing them in Word, and now they are going by themselves into the system, generating and tweaking
and okaying the questions without you even needing to be there, you are saving 100 % of your time on that task.
Brick Maier (26:35)
Yeah, yeah, I know. that’s,
we’ve had it rolled out for, I think, seven or eight months now at this stage. And so we can kind of see on the back end, like where these knowledge checks are starting to show up. And, we have legacy instructors who have to go back to a course and add them, which some are really engaged and want to update their course and they’ll do that. Others are like, no, that’s good. And that’s just, we don’t go in and add them for their course.
But what we do notice is the new courses in that last seven months, a lot of the new instructors are adding them in way more than we thought they would initially when we rolled it out, we’re like, every third video, just a pulse check, pick, and they’re like every video they’re adding these two to three questions takes about a minute of time to put that in. They’re skippable for the student. we don’t lock the UI and force them to go through these, but
if you choose to engage, it’s going to be at a minute or two in to your time in between videos. And, we just saw that instructors are adding them. And we thought that was also another metric of this and then students are taking them. We see that they’re engaging with them. They’re not skipping them. And, ⁓ we don’t score in the same way as a summative assessment where like pass rates and number of attempts and things like that, but
we can see how they’re engaging with the form of assessments.
Jason Gorman (27:45)
I think you used the word unlock. That’s a great unlock. As I said before, I think a lot of different people are going to be in lots of different kinds of organizations who might hear this. And if so much, most of us are writing, planning
creating assessments as part of the things that we’re doing. And if somebody is in the process right now where they are really facing the same challenge, maybe it’s taking too much time or they’d love to save some time, where they want to implement a solution similar to yours, what advice would you give them in terms of getting started putting that into place?
Brick Maier (28:21)
Short cycles of iteration have been really helpful in testing a concept. with these, the capability of any of the frontier LLMs is interchangeable how good they all are. We have our preference for some, like whatever your favorite go-to model is, they’ll be really good at responding to that test and looking at the quality and understanding is this good? And one thing that was…
required for me to understand if it was good or not was to have an expert in the loop. I could go and have the AI look at a battery energy storage system course and it would come back with some questions and I’d be like, I don’t know if these are good, but I would loop in the instructor who wrote the course and you’d be like, okay, yeah, these are great. This one is a little simple. I wouldn’t ask that one because it’s not relevant, but by and large eight out of 10 of these are passable.
Jason Gorman (28:58)
Sure. Yeah.
Brick Maier (29:12)
That’s great. So the advice is quick iteration and then having expert who can be that evaluator. Because with the AI tools, you gotta find a way to maintain a critical thinking, human creative connection to what’s being outputted and not just trust it
in the initial testing phases, because it’ll give you stuff that looks really good. Always. It’s always going to read well initially. And then you read it again. Like, that doesn’t make sense. And, and so you have to really go through a fine tooth comb initially and really not trust it at first. And there are places where we don’t use AI. We just determined it’s not worth it. It’s not good but for the assessments, was very good. And we got there by short iterative loops of testing with a human
Jason Gorman (29:37)
Yeah.
Yeah.
Mm-hmm.
Brick Maier (29:59)
expert evaluating it and giving us feedback on all the things that we were interested in, like the quality of the stem and the diversity of questions that referenced the Bloom’s taxonomy, what kind of questions are you asking? So those are the kind of things that helped us arrive at our solution.
Jason Gorman (30:19)
That’s terrific. So usable and I think pretty universal kind of use case to what so many people have to do in their jobs. we think about this from an operating perspective. The workflow of starting with a raw material, creating some kind of draft, getting feedback on that draft
revising the draft, pushing out the final, publishing the final version. It’s actually quite a lot of steps with a number of people involved. And I think you don’t know that until you’re really doing the work and you’ve done the work and you realize, you can’t shortcut these things too. But it seems you’ve really found some nice ways to get a great level of quality much faster. really, this is very, very useful, very specific
tips here, so really appreciate that. A couple quick questions for you as we’re getting close to the end here. So one is, what are your predictions? Like I said, things keep changing. What do you think the future is going to look like for people like you and me in the future in terms of our jobs?
Brick Maier (31:23)
It depends on the day and kind of what my blood sugar level and if I’m hungry or angry or grumpy, ⁓ no, no, I think, it’s an interesting question to answer in this moment because we are in, at a peak of something, we’re at a level of AI hype.
that you can get caught up in predicting a future when it’s the hype, that’s the momentum. And so like the enthusiasm for the future, is outweighed by the evidence for the actual change. But AI has actual change happening right now. So I’m not dismissing the ways that we’re interacting in.
What changes for learning designers or folks who think about instructional design or just building a learning experience is increasingly there will be an interaction layer on top of what we build that is independent of what we can control. is, and right now it could be like, you you’ve got your two windows open and you’re kind of chatting with chat GPT about something and it sends it back, but
fairly soon, browsers are just going to have an agent view of what you’re looking at. And that little chat window is now going to be looking at your course, the course that maybe we designed. And that interaction is happening outside of our course. They’re just querying other things or trying to understand better. And that kind of interaction is great for the learner, but changes how we potentially build
Jason Gorman (32:31)
Mm-hmm.
Brick Maier (32:47)
courses and interact with our students. And so I don’t know how the habits of learners will change or what expectations they’ll bring to a course over the next few years in terms of that little, assistant tutor, helpful thing
that is there to answer your questions or dive deeper or go horizontal in this way or unpack something. And to the extent that you can build that in to a course for them, I think is helpful. We’re being conditioned, I think, to start using these interfaces more and talking with our content in a way that’s different than the static.
I view our courses now as like the Gutenberg press, just we’re typesetting things and we’re stamping it, we’re pushing it out and we’re like, well, we’ve got to update that later. So we have to have systems in place to change it. And we’re, moving toward a future where it’s more dynamic. learning is more probabilistic and not deterministic. And I think in large corporations, when you have an e-learning mandatory learning that everyone has to go through that’s
Jason Gorman (33:29)
you
Brick Maier (33:50)
deterministic and it can be depending if it’s coming from HR or something we need to make sure everyone goes through this exactly and so we can say they got it, they learned it and they can do it. And that I mean for safety like I was saying some of the courses we don’t use AI for like electrical safety like a hallucination on something related to electrical safety is a risk we are not willing to take. So we are only going to work with experts who are you know certified master electrician journeyman electricians
to vet that kind of content. ⁓ But we are, I think, heading toward a future where high 90s percentile probabilistic learning, where not every learner has the same experience, but ends up at a point where they can take a summative assessment, where they can demonstrate they’ve learned the things. Does that make sense..? AI is going to give us a different path. And we can have an objective
Jason Gorman (34:19)
Mm-hmm.
Brick Maier (34:41)
test that we have them take. And then that’s going to be from a learning perspective, that assessment is our way of saying, well, they’ve demonstrated the outcomes that we set forth. They have these skills and competencies. We build a course and we know that we built it in a way that they can approach it from a number of different paths. AI will do that, I think, better and better.
And then beyond that, beyond like two years from now, immediately there’s this Socratic learning mode that people are testing out and it’s okay. It’s, it’s going to get better. when I looked at it, I think you tested it out as well. it’s trying to be a helpful inquisitive prompter, but it also kind of, you get question fatigue and you’re like, do you keep asking me? so
Jason Gorman (35:11)
Yeah.
Yeah.
Brick Maier (35:24)
they’ll dial it in and
to the extent that voice interaction is gonna become more common and synthesized AI voices for those who wanna speak out their learning, they’re gonna get that. People wanna type it out, they’ll get that. Dynamically generated AI videos I imagine sitting on a pile of knowledge
Jason Gorman (35:36)
Mm-hmm.
Brick Maier (35:48)
And then the LLM analyze it and just constructs it in the way that the student wants. And it can adapt the course to how they’re going through it. And then my mind, think about, I think less about what learning designers are doing. And I’m just kind of curious about who’s going to succeed as a student in that environment. Cause I think back on this, one of my favorite anecdotes from Amazon, which was
I saw this one guy, he’s a great teacher and he would, data engineer 20 years at Amazon, actually 25 years at Amazon. And he just had, everyone had like nine monitors and total he’s in a BI team. so they all these things, but he just had one laptop, which I think is a sign of like a boss movie. Just one single laptop, no external monitors. And he’s just working inside the matrix doing stuff. But anyway, they come up to him and ask him a question, he pushes his laptop back.
Jason Gorman (36:33)
Thank
Brick Maier (36:36)
answer a question. And I noticed that the steady stream of people coming by and I asked his manager, what do you see? Like when someone’s onboarding into their role as a BI data scientist and they’re trying to understand these databases and they’re going to John all the time. And he’s like, the people who don’t grow in their role ask the same question.
Jason Gorman (36:56)
Mm.
Brick Maier (36:56)
And they’re just
looking for John to help them solve that question. He’s like, but I can also see other people who go and learn and ask a new set of questions every time. And then over a 90 day or six month or 12 month period, those two people are fundamentally different places in their growth. And I think of that when I think about AI, because I think that the people increasingly who can be actively engaged in like critical inquiry and like asking a different set of questions and growing.
They’ll get a lot out of these new ways of interacting. But I also know there’s a lot of people who are going to kind of maybe be a little more passive or maybe haven’t been coached on how to ask better questions. then think that’s where I think of how do we train the students of the future to be inquiry based and to be just wildly curious about something. And, and I think about like back to my classroom where, know, we’re at a point now we’re like,
Jason Gorman (37:31)
Hmm.
Mm-hmm.
Brick Maier (37:50)
You don’t ever need to write anything ever, ever again. Hooray. Never pick up a pencil, never write anything. And how can I do the least amount of work possible to get this grade? Well, it’s there. Never write. However, if you do want to grow and learn and be curious and build out a whole new world of things, then let’s get really good at asking this question, reading what comes back
Jason Gorman (37:57)
you
Yeah.
Brick Maier (38:14)
doing your own writing and reflection offline, non cloud-based. I don’t know how often you do this where you just write with a notebook or think outside of the LLM, but it’s a different part of your brain. I don’t want that to atrophy. So yes, these tools are great, but we need to be curious and help encourage people how to be curious. I don’t know. That was a long winded answer, but it’s.
Jason Gorman (38:25)
It is.
No, I actually
really appreciate it. And I love thinking about John at Amazon it sounds like people were coming to John and externalizing their understanding by asking him, getting the answer and then going to implement the answer. But if some of them were not actually learning why that was the answer. So they had to come back and ask again. But then the second set of people were learning. This does seem like the perfect ⁓ analogy for AI because
I think that you’re right. I do actually have a notebook. And when I am being responsible in my work, I’ll start in my notebook. And I find that I’m just a stronger thinker. I make better decisions. I can ask better questions. I can be more critical of what I’m getting from it. So yeah, totally, totally makes sense. It’s a wonderful answer. So two things here to wrap up.
One thing that I will ask you in a minute, but I’m gonna ask you first, we’ve had one guest before you, his name’s Stephen Hyde. Stephen works in University of North Carolina. He’s a leader there in executive development. And I asked him to ask a question of our next guest, and you’re our next guest. So here’s Stephen’s question.
His question is, when did it hit you that your work matters just as much as what you deliver? And what, if anything, shifted after that?
Brick Maier (39:50)
All right, let me think about that question.
So the process of the work is as important as the output of the work.
Jason Gorman (39:57)
Yes.
Brick Maier (40:01)
I don’t know if there’s a specific moment that I realized it, like this epiphanal, like, aha. But the general realization of the process is what matters helped me take my foot off the gas of delivering and being bound by a milestone.
Deliverables schedule because those do those milestones that are important to keep everyone on the same page you can see but if You’re so focused on getting the thing out the door You can lose sight of what’s going out the door, it’s like no no, got QA We got a T testing. It’s like it’s good. It’s good. It’s good but I’ve experienced that particularly place like Amazon where it’s just like That that date is very important and we better have it ready for that date
Jason Gorman (40:20)
Yeah.
Yeah.
Brick Maier (40:41)
and you don’t want to show a little red dot on your, you don’t want to be a blocker. So get it going. And to find a way to use the process for critical thinking and understand that like you’re going to be problem solving, you’re going to get stuck, you’re get unblocked, you’re going look at this way, you’re going to look at Like that is when the, yes, the, the, the, the date, the literal delivery of it might shift, might be earlier, might be later. But like if you’re kind of present to, uh,
Jason Gorman (40:44)
you
Brick Maier (41:08)
what you’re building and pulling back and being willing to pivot. I don’t know, those happen in the process. And if you’re really so rigidly outcome focused, you can kind of lose that. I’ve found that sometimes you can lose that.
Jason Gorman (41:20)
Excellent. And now it’s your turn. What’s a question you would love for our next guest to answer?
Brick Maier (41:26)
Well, it’s funny that there’s a specific one that’s an extension of the AI conversation we just had.
The question is around student learning and their experience of learning, what will you look for to see that it’s kind of fundamentally changed with these new tools? Like what evidence will you be looking for to know that it’s changed? Cause right now, if you go on YouTube or
Jason Gorman (41:44)
Mmm.
Brick Maier (41:51)
social media or LinkedIn, you’ll see a lot of hype and promotion. And that’s just the nature of those things. They’re important. It’s important to people put their thoughts out there, but I haven’t seen a lot of evidence that the learning experience has changed. we’re changing the way we build things. like, the student interview or the student application, the application of what you talk about. What are you seeing that’s different?
I’d be curious about that response. ⁓ this is one that I think about a lot as I myself am a kind of an AI advocate, skeptic, want to use it, but also don’t want to trust it blindly. My best way to be skeptical of it is to try and use all these new things rigorously and then like, know when it’s like not very good and when it’s actually great.
So.
Jason Gorman (42:37)
Good stuff, really appreciate it. I so appreciate you coming on, sharing your background, getting into the weeds of some of what you do, sharing your kind of philosophies and opinions about where we are now and where we’re going. This has all been just excellent. There’s some things that I know that I’m gonna be able to take away and use this from this in my work. So that’s the point, really appreciate it. Thanks for joining us, Brick.
Brick Maier (43:00)
Well, thanks Jason for organizing this and you came up with a great set of questions that were a good starting point for reflection, which is really helpful. So thank you.
Jason Gorman (43:11)
Yeah, absolutely, my pleasure. And the world’s gonna continue to change, so maybe we could do this again in six months or a year and see if, see when the world’s totally different and see how right or wrong we if…
Brick Maier (43:20)
we still have jobs or if we’re just all eating like, dates on fluffy
pillows and be like, isn’t it great that we don’t have to do any work anymore? I imagine we’ll both still be working.
Jason Gorman (43:27)
All right. Yes, I certainly hope so. Anyway,
thanks again. Really a pleasure having you on.
Brick Maier (43:34)
All right, talk to you later.