Assessing understanding in the age of AI

Generative AI has created a strange problem for higher education.

For a long time, we have treated the finished product as evidence of what a student knows. A student writes a paper, completes a case study, develops a presentation, or submits a project. We evaluate the work and, from that, make some reasonable assumptions about what the student understands.

That assumption has become much harder to make.

A student can now use ChatGPT, Claude, Gemini, or any number of other AI tools to help brainstorm an argument, organize a paper, rewrite paragraphs, analyze information, generate code, summarize readings, or produce a remarkably polished first draft.

And this is not really an AI-detection problem.

It is an assessment problem.

The question has changed

I work with faculty on educational technology and AI, and one of the questions I hear repeatedly is some version of this:

How do I know whether the student actually understands what they submitted?

That is the question I think we should be spending more time on.

Trying to determine exactly how much AI contributed to a document can turn into a dead end pretty quickly. Even if an instructor strongly suspects AI use, that still does not tell us what the student knows.

Maybe the student generated the entire thing and barely read it.

Maybe they used AI to improve their writing but did all of the intellectual work themselves.

Maybe they used it to brainstorm, challenge an argument, clean up grammar, or help organize ideas they already understood.

Those are very different situations, yet the final documents might look surprisingly similar.

So instead of asking a piece of software to determine who or what wrote every sentence, I think there is a more useful question:

Can the student explain and defend the work?

That gets us much closer to what we actually care about.

The paper still matters

I do not think the answer is to stop assigning papers.

The same goes for presentations, projects, case studies, coding assignments, design work, and other traditional forms of assessment. There is still tremendous value in asking students to research something, develop an argument, create something, and communicate it well.

The problem is assuming that the finished artifact tells us everything we need to know.

It doesn't anymore.

Suppose a student submits an excellent research paper. Instead of ending the assessment there, ask the student a few questions about it.

Why did you choose this evidence?

What was the strongest source you found, and why?

You make this claim on page four. Walk me through how you reached that conclusion.

What evidence did you find that challenged your argument?

If you had another week to work on this, what would you change?

Those questions are difficult to answer convincingly without understanding the work.

They are also much more educationally interesting than asking whether an AI detector thinks a paragraph was machine-generated.

There is nothing new about oral defense

Academia has been doing this for a very long time.

Doctoral students defend dissertations. Graduate students sit for comprehensive examinations. Universities around the world use viva voce examinations. Professional programs use oral examinations, simulations, clinical questioning, and practical demonstrations.

The basic idea is simple: if someone claims to know something, give them an opportunity to demonstrate that knowledge.

What has prevented this from becoming a normal part of undergraduate assessment is largely practicality.

If I have 30 students, I might be able to meet individually with each one.

If I have 100 or 200 students, conducting individual oral examinations becomes almost impossible.

That is where I think AI can actually help assessment rather than undermine it.

Using AI to make oral assessment practical

This idea became one of the foundations for Rocketproof.

A student first completes the normal assignment. The paper, presentation, project, analysis, or other artifact still matters.

Then the student completes a short oral defense based on that specific work.

The important part is that the questions are grounded in what the student actually submitted. If a student made a particular argument, cited a particular source, reached a certain conclusion, or made an unusual design decision, those are exactly the kinds of things worth asking about.

The resulting conversation becomes another piece of assessment evidence.

Faculty can review the recording, transcript, rubric, suggested scoring, and supporting evidence together, then make the final grading decision themselves. That faculty control is important. The AI can help organize evidence and reduce some of the mechanical work, but the instructor remains responsible for the grade.

That is a much more interesting use of AI to me.

We are using AI to help a student demonstrate what they know.

What does an oral defense actually tell you?

Imagine that a student submits a polished paper arguing for a particular approach to coastal restoration.

The paper is excellent.

Now ask:

"Your argument depends heavily on this particular source. Why did you consider it credible?"

A student who understands the research can probably answer that without much difficulty.

Then ask:

"What is the strongest argument against your position?"

Now we are getting somewhere.

Or:

"You recommend this approach in your conclusion. What would make you change that recommendation?"

That question requires the student to think beyond simply remembering what was written.

None of these questions proves that the student never used AI.

That is not really the point.

They give the instructor something much more useful: additional evidence about whether the student understands the ideas represented in the work.

This works differently depending on the discipline

One thing I find interesting about oral assessment is how easily the basic idea transfers between fields.

In a history course, a student might defend why one primary source deserves more weight than another.

In an emergency management course, the instructor might change one variable in a scenario and ask the student how that affects the response.

In business, a student might defend a recommendation and then respond when one of the assumptions behind the recommendation changes.

A computer science student could explain a section of code, identify a design decision, or walk through what would happen if part of the system failed.

A nursing student could explain the reasoning behind a decision in a simulated patient scenario.

A design student could explain why a particular design choice was made and what alternatives were considered.

A student in a language course could simply have the conversation in the language being assessed.

The exact format changes, but the underlying principle stays pretty consistent: the student has to do something with what they supposedly know.

Rocketproof was designed around that idea, which is why it supports different assessment workflows rather than treating every assignment as an oral version of an essay.

It can also change how students approach assignments

There is another benefit here that I think is easy to overlook.

Assessment influences behavior.

If students know that the only thing being evaluated is the final document, their attention naturally goes toward producing the best possible document.

That made sense when producing the document required doing most of the intellectual work.

Generative AI has weakened that connection.

But imagine telling students at the beginning of an assignment:

"You can use AI according to the policy for this course. Just remember that you are going to have to explain and defend what you submit."

That changes the incentive.

Now it matters whether the student understands the sources.

It matters whether they can explain the argument.

It matters whether they recognize weaknesses in their reasoning.

It matters whether they can respond when someone asks a question that was not in the original prompt.

In other words, understanding becomes valuable again.

This is also why I am skeptical of an AI-detection arms race

I understand why AI detection became one of the first responses to generative AI. Faculty were suddenly receiving work that looked very different from what they had seen before, and they needed some way to respond.

But detection has a fundamental limitation as an assessment strategy.

It is trying to answer a question about the origin of the text.

Faculty ultimately need to answer a question about the student.

Those are not the same thing.

Rocketproof was deliberately designed around verification of learning rather than AI detection. The platform does not need to prove that a student used ChatGPT. It gives the instructor another source of evidence for determining whether the student understands the work.

That approach also works regardless of a course's AI policy.

An instructor might prohibit generative AI for a particular assignment.

Another might allow it for brainstorming and editing.

Another might actively require students to use it.

In every one of those cases, it is still reasonable to ask students to understand what they submit.

What about students who are nervous about oral exams?

This is a legitimate concern.

A student can understand something perfectly well and still be uncomfortable speaking on camera. Students may also have disabilities, language differences, anxiety, or accommodations that affect how they should participate.

That means oral assessment needs to be designed with flexibility from the beginning.

Rocketproof supports options such as typed responses, audio-only participation, time extensions, pause and resume, practice opportunities, and other accommodations.

I also do not think every oral defense needs to feel like a dissertation defense.

For many assignments, this can simply be a short academic conversation.

"Tell me why you made this choice."

"Explain this part to me."

"What did you find most difficult?"

"What would you do differently?"

That feels very different from trying to catch a student doing something wrong.

The instructor still needs to make the judgment

This part matters to me a lot.

If the response to AI-generated student work is simply to have another AI grade everything, I am not sure we have solved the underlying problem.

Faculty judgment still matters.

AI can generate questions. It can transcribe a conversation. It can identify relevant moments in a response. It can compare evidence against a rubric and suggest where an instructor might want to look more closely.

Those are useful jobs for AI.

The final academic judgment should still belong to the instructor.

That is why Rocketproof brings the student's submission, oral defense, transcript, rubric, suggested scoring, and supporting evidence into the faculty review process and allows the instructor to adjust or override the result.

The technology should make it easier for faculty to see the evidence. It should not make the academic decision for them.

Maybe we have been asking the wrong question about AI and assessment

A lot of the conversation around generative AI in education has focused on preventing students from using it.

I am increasingly convinced that this is too narrow a way to think about the problem.

AI is going to be part of the environment our students learn and work in. The tools will continue to improve. The line between AI-assisted and entirely human-produced work will probably become harder to identify, not easier.

That means our assessments need to hold up even when AI is available.

For me, the more useful question is becoming:

What evidence would convince me that this student understands what I am assessing?

Sometimes the answer will still be a paper.

Sometimes it will be a presentation, project, simulation, demonstration, discussion, or exam.

And sometimes the simplest answer may be to ask the student a few good questions and listen to how they respond.

That is the problem we built Rocketproof to help solve.

The goal is not to catch students using AI.

The goal is to make student thinking visible again.

If you are reconsidering how you assess student learning in the age of generative AI, you can explore the Rocketproof interactive demo or contact us about running a pilot in your course or program.