Peer Review in the Age of AI: How to Assess Evaluative Judgment When AI Can Write Feedback

For decades, higher education faculty assigned peer review with a clear pedagogical purpose. When students evaluate a classmate's draft, they do more than assist with grading. They examine work addressing their prompt, analyze arguments, identify weaknesses, and consider how another writer approached the problem.

Generative artificial intelligence disrupted that mechanism. Today, an undergraduate assigned to review three classmate papers can paste each draft into ChatGPT, request feedback, and submit the output within minutes. The resulting comments read smoothly, praising the author's structure and recommending clearer thesis statements.

The student author receives polite, homogenized feedback. The student reviewer never reads the paper, never evaluates evidence, and never makes an independent intellectual judgment. The instructor awards credit for a polished critique that represents zero student learning.

When software generates thoughtful-sounding prose on demand, written reviews stop serving as reliable evidence of evaluative thinking. If universities want peer review to remain an engine of critical thinking, instructors must evaluate whether students can explain, justify, and defend their evaluations.

Developing Evaluative Judgment in the AI Era

Higher education research demonstrates that peer review benefits the reviewer even more than the author. Educational theorist Royce Sadler defined evaluative judgment as the capability to recognize quality and determine how work compares against disciplinary standards. David Boud and Phillip Dawson expanded this framework, showing that students develop evaluative judgment by repeatedly appraising work produced by others.

When a student reads a peer's essay, lab report, or design proposal, they must interpret the rubric in context: Does this paragraph support the central claim? Is this sample size adequate? Does this engineering schematic respect safety tolerances?

Answering those questions forces students to internalize disciplinary criteria, carrying a clearer understanding of quality back to their own work.

When students offload that critique to an automated tool, this developmental loop breaks. Language models produce text mimicking academic feedback, but the student never exercises judgment. The software performs the analysis; the student acts as a courier between two text boxes.

The Pedagogical Breakdown of AI Peer Critiques

AI-assisted peer review introduces three specific hazards into the classroom:

First, language models generate anodyne critiques. A standard model reviewing a history paper will reliably produce polite paragraphs complimenting the author's passion. It rarely points out when a central thesis contradicts primary source evidence. Real peer feedback requires precision and skepticism; automated critiques deliver diplomatic platitudes.

Second, synthetic reviews create epistemic harm for the recipient. When a student author submits a draft, they need to know where a human reader lost the thread of their argument. Receiving synthetic praise leaves the author with false confidence and unaddressed conceptual gaps.

Third, outsourcing review creates privacy hazards. Pasting unpublished classmate essays into public foundation models frequently violates institutional policies and student intellectual property boundaries. An undergraduate's research should not be indexed by commercial AI systems simply because a peer wanted a shortcut on a review assignment.

The Solution: The Peer Review Defense

The fundamental principle of authentic assessment in the modern era requires evaluating the student directly: assess the student, not just the submission.

In written assignments, this principle led to the return of the oral defense. When a finished essay no longer guarantees authorship, asking the student to talk through their decisions, explain their sources, and address counterarguments verifies true understanding. To explore how oral defense functions across modern courses, read our guide on what an oral defense is and why it is returning in the AI era.

That same logic applies directly to peer review. If software can generate the critique, instructors cannot treat written comments as sufficient evidence of learning. Instead, faculty can introduce a short, structured Peer Review Defense.

In a Peer Review Defense, the student submits their evaluation and engages in a short oral conversation about their critique, examining both the peer's submission and their own markings.

To complete the defense, the student answers source-grounded questions:

• Which specific claim in your peer's literature review seemed unsupported by empirical evidence?
• You scored their methodology section as proficient rather than exemplary. What experimental control did they omit?
• Walk through the suggestion you made in paragraph three. How would that change strengthen the author's argument?
• Where did your peer's analysis surprise you, and did reading their draft alter how you approached your own project?

A student who pasted their classmate's paper into a chatbot cannot answer those questions in real time. They do not know which evidence appeared on page four because they never read it. A student who read the manuscript can articulate their thinking naturally. For a broader look at verbal reasoning, see our analysis of student explanation as evidence of learning.

How a Peer Review Defense Works in Course Workflows

Rocketproof makes this workflow practical at scale through its built-in peer review workflow mode. The platform structures the defense as an automated conversation directly after the student completes their evaluation.

Here is the operational workflow from assignment creation to final grading:

1. Assignment Setup: The instructor configures the peer review assignment within their learning management system or Rocketproof, attaching a rubric with focus areas such as source credibility, argumentation logic, or constraint analysis.

2. Peer Evaluation Submission: Students read their assigned classmate's work, mark rubric criteria, and provide written feedback. Rocketproof records the peer's original document alongside the student's written critique.

3. The Interactive Defense Session: Immediately after submitting their review, the student enters a short, four-to-six-minute conversation using their browser microphone. Rocketproof grounds questions directly in both texts: the classmate's draft and the student's critique, asking precise, source-aware questions derived from the actual content. To understand how adaptive probing works, see our guide on source-grounded questioning. For example, the system might ask: "You wrote that the author failed to address alternative explanations for their survey results. Point to the specific data point in Table 2 that you found ambiguous, and explain what additional variable they should have controlled for." The student responds verbally, and the engine asks follow-up questions when a point needs deeper justification.

4. Faculty Review and Grading Sovereignty: Rocketproof compiles the complete record for faculty review: the peer's draft, the student's critique, the synchronized transcript with audio recording, and a concise summary highlighting how the student justified their critique. The instructor retains complete grading sovereignty, reviewing each defense in two to three minutes. To learn more about instructor oversight, read our article on faculty-controlled AI assessment, and see our framework for scalable authentic assessment.

Disciplinary Applications: Engineering and Business

Peer review is common across professional programs where students must learn design critique and technical evaluation. Here is how a peer review defense functions in two professional disciplines.

Engineering Design and Capstone Projects

In senior mechanical engineering capstone courses, student teams review technical specifications drafted by other groups. A written critique might state that a competitor's gear assembly lacks thermal expansion tolerance.

During the defense, the platform probes that claim: "You noted that the thermal expansion calculations in section 3.2 are insufficient for high-temperature operating environments. Walk me through the thermal coefficient they selected, and explain why that material choice poses a failure risk under extended operational load."

The student must articulate the engineering principles behind their observation, verifying that the reviewer understands thermodynamics rather than reciting a generic checklist.

Business Case Analysis and Strategy

In an undergraduate management course, students critique strategic expansion plans written by their peers. A student reviewer might recommend that an author reject a proposed market entry into Southeast Asia.

The defense examines the reasoning behind that recommendation: "Your critique argued that the author underestimated regulatory entry barriers in their market analysis. Which specific regulatory requirement mentioned in the case study led you to that conclusion, and how should the author adjust their financial projections to account for compliance costs?"

The student demonstrates whether they understand international market dynamics, regulatory risk, and cash flow modeling.

Setting Pedagogical Expectations

Adding a defense component to peer review requires clear communication with students:

Distinguish Mechanics from Evaluative Thinking: Instructors should clarify that mechanical proofreading is distinct from evaluative critique. Software can spot misspelled words, but human evaluators must assess argumentative coherence, ethical implications, and methodological soundness. Faculty should explain that the defense grade reflects their ability to explain conceptual critique rather than minor grammatical edits.

Share Defense Criteria in Advance: Students should know before reading that they will answer for their critique. When students know they will defend their evaluations, their reading behavior changes immediately. They take active notes and examine footnotes. Knowing they must speak about the work transforms passive skimming into rigorous critical reading.

Calibrate Quality in Class: Before assigning the first peer review, instructors benefit from reviewing an anonymized draft together as a class. Walk through what constitutes an actionable critique versus a superficial comment, modeling the types of follow-up questions students will encounter during their defense.

Accessibility, Privacy, and LMS Integration

Authentic assessment requires careful attention to accessibility standards, data governance, and technical compatibility:

Universal Design and Accommodations: Rocketproof includes an accessible interactive text mode for students with documented speech disabilities or approved accommodations. In text mode, the student responds via typed text while facing the exact same source-grounded questions and rubric standards. The system evaluates their analytical reasoning without penalizing communication formats. For a comprehensive review of inclusive design, read our guide on accessibility in oral assessment.

Student Privacy and FERPA Compliance: Peer review involves sensitive student work. Rocketproof operates with an enterprise data architecture that respects student privacy. Student submissions and audio recordings are never used to train public machine learning models, remaining isolated within institutional boundaries in compliance with FERPA requirements.

LMS Integration via LTI 1.3 Advantage: Rocketproof integrates into institutional learning management systems through LTI 1.3 Advantage standards, supporting Canvas, Blackboard Learn, D2L Brightspace, and Moodle. Faculty create peer review defense assignments directly within course modules, rosters synchronize automatically, and grades pass back into the gradebook. To explore setup details, see our guide on LMS-integrated AI assessment with LTI 1.3.

Frequently Asked Questions

Does adding a defense increase grading time for instructors?

No. In traditional peer review, faculty spent substantial time reading lengthy written critiques to determine whether students put genuine effort into their reviews. Rocketproof provides a summary of the defense alongside the transcript and audio timestamps, allowing an instructor to verify student comprehension in two to three minutes per student.

How does this prevent students from using AI during the defense itself?

The defense is a live, dynamic conversation. The system generates follow-up questions tailored specifically to the student's spoken response and the text of the peer's paper. Because the student must respond verbally within a designated time window, they cannot copy questions into an external chatbot and read an answer aloud without noticeable pauses and disjointed responses.

Can students still use AI as a formative editing partner?

Instructors retain full authority over their AI policies. Many faculty permit students to use generative tools to brainstorm ideas or polish grammar during early drafting phases. However, the Peer Review Defense ensures that the student reviewer personally understands disciplinary standards and can defend their evaluation without relying on automated assistance.

Cultivating Independent Evaluators

The goal of higher education extends beyond generating written documents. Universities prepare students to think critically, evaluate evidence, and make reasoned judgments in their disciplines.

When generative AI can draft both the research paper and the peer critique, relying on written text alone leaves instructors uncertain about what students actually understand. A Peer Review Defense restores accountability to collaborative learning. By asking students to speak for their assessments, faculty protect the integrity of the peer review process, foster genuine evaluative judgment, and ensure that students leave their courses capable of evaluating quality in the real world.

To explore how oral evaluation strengthens academic integrity across disciplines, learn more about AI oral exams and oral defense, try our interactive demo, or join the waitlist for your department.