A student submits a better answer than they could have produced alone.
The answer is coherent. The reasoning is neatly structured. The examples are stronger. The grade improves.
A week later, the student faces a related problem without the tool.
What should we expect them to know now?
That question is the centre of “Distinguishing performance gains from learning when using generative AI,” a 2025 Nature Reviews Psychology Comment by Lixiang Yan, Samuel Greiff, Jason M. Lodge, Dragan Gašević and colleagues.
The paper makes a distinction that education has always needed but generative AI makes impossible to ignore: performance while assistance is available is not the same outcome as learning that persists when assistance changes or disappears.
The focal publication is a Comment, not an empirical trial. It synthesizes arguments and prior evidence to warn against inferring learning from assisted output quality.
That source class matters.
The paper’s claims become strongest when read alongside emerging controlled studies that directly measure what learners can do after AI support is removed.
What the Comment argues
Generative AI can make learners faster, more fluent and more successful at immediate tasks.
That can be valuable.
But many forms of learning depend on cognitive activity that assistance can either support or replace: retrieval, explanation, error detection, comparison, planning, integration with prior knowledge and monitoring of understanding.
If a tool supplies an answer before the learner has to perform those operations, the observable product can improve while the underlying capacity remains unchanged.
The Comment therefore asks educators and researchers to separate performance gains from learning gains.
Performance concerns how well a learner completes the present task under the present conditions.
Learning concerns relatively durable changes that can be demonstrated later, often through retention, transfer or unaided performance.
The distinction is simple enough to state.
It is surprisingly easy to violate in research and educational practice.
Better output is not the wrong outcome
The argument is not that performance assistance is useless.
External tools have always extended human capability. Dictionaries, calculators, search engines, worked examples and collaboration can all improve performance.
Sometimes the appropriate goal really is joint performance. A professional may need to produce a safer report, not memorize every sentence. A learner with a disability may need an assistive system to access a task on equitable terms. An expert may use automation to reduce routine burden and preserve attention for higher-level judgment.
The mistake is not accepting assistance.
The mistake is measuring one outcome and naming it another.
If a course promises durable conceptual understanding, an AI-supported assignment grade is not sufficient evidence of that understanding.
If a workplace only needs reliable joint human-AI performance, unaided recall may be less important.
Goal clarity comes before tool judgment.
The most useful evidence appeared around the same time
A major 2025 field experiment in high-school mathematics gives the Comment’s distinction empirical weight.
In work published in PNAS, researchers studied nearly 1,000 students using two GPT-4-based systems. A standard chat-style system, GPT Base, substantially improved performance on practice problems while students had access to it. A more constrained GPT Tutor also improved assisted practice performance and was designed to provide teacher-created hints rather than simply give solutions.
When the systems were removed for a subsequent exam, the GPT Base group performed 17% worse than the control group. The negative learning effect was largely mitigated in the scaffolded GPT Tutor condition, although the tutor group did not show a positive exam advantage over control.
The study is not proof that ordinary AI use always harms learning.
It is proof that a very large assisted-performance gain can coexist with worse later unaided performance under a particular design.
That is exactly the inferential error the Comment asks researchers to avoid.
Guardrails can change the learning process
The difference between GPT Base and GPT Tutor is important.
Technology does not enter learning as a binary variable called “AI.”
Interaction design changes what the learner has to do.
A system that immediately supplies final answers can reduce retrieval and problem generation. A system that asks a learner to attempt a step, provides a hint, exposes an error and asks for revision can create a different cognitive sequence.
A 2025 randomized controlled study in Computers & Education provides another example. In a visual learning analytics task with 117 higher-education students, proactive generative-AI agents that used scaffolding questions produced better comprehension than passive agents or standalone scaffolding, with benefits persisting into post-intervention assessment.
Different task, different population, different system.
That heterogeneity is the point.
The scientific question is not “Does generative AI help learning?” in the abstract.
It is “What does this system ask this learner to do, what cognitive work remains human, and what can the learner do later under a meaningful criterion?”
The performance-learning distinction predates AI
Learning science has long shown that conditions making practice feel easier can produce weaker long-term retention than conditions that require effortful retrieval or generation.
Repeated study can increase immediate fluency and confidence. Retrieval practice can feel harder while improving later retention. Worked examples can accelerate early performance but need to be faded as expertise grows. Guidance can be beneficial for novices and excessive for advanced learners.
Generative AI intensifies these older problems because the assistance is unusually adaptive, fluent and comprehensive.
The tool can perform not only arithmetic or lookup but also planning, explanation, summarization, argument construction and self-evaluation.
Those are precisely the operations educators often hope learners will practice.
Metacognition is part of the problem
Learners also have to know whether they are learning.
The PNAS mathematics study reports evidence that students did not necessarily perceive the later cost associated with copying or relying on solutions.
This creates a metacognitive trap.
Assisted success can feel like personal mastery because the learner experiences the task becoming easier in real time. The external contribution is psychologically absorbed into the experience of competence.
When the tool disappears, the learner discovers that the performance state did not fully transfer.
This is related to older findings in learning research where rereading can increase familiarity and confidence without producing equivalent long-term retrieval strength.
AI does not create the metacognitive illusion from nothing.
It can make the illusion more convincing.
What should count as evidence of learning?
A strong study needs a criterion matched to the claim.
If the claim is retention, test after a delay.
If the claim is transfer, change the problem structure or context.
If the claim is independent skill, remove or alter the assistance.
If the claim is human-AI collaborative competence, keep the tool available but test whether the user can detect errors, select when to delegate and recover when the model fails.
There is no single correct post-test.
There is only a test that is more or less aligned with what the programme says the learner is becoming able to do.
“Without AI” is not always the final destination
One response to the performance-learning problem is to insist that all meaningful learning must eventually be demonstrated without external tools.
That is too simple.
Modern cognition is distributed across notebooks, calculators, software, institutions and other people. Expertise often includes knowing how to use external systems intelligently.
The relevant question is which capacities must remain available internally for the person to use the external system responsibly.
A pilot need not calculate every navigation variable mentally, but must retain capacities needed when automation fails. A clinician can use decision support, but must understand enough to recognize implausible recommendations. A writer can use language models, but may still need argument structure, source judgment and a sense of what they actually mean.
The Institute’s synthesis is to distinguish capacity worth retaining from work worth offloading.
The Comment motivates this question but does not answer it empirically.
The paper’s strongest limitation is its genre
Because the focal source is a two-page Comment, it should not carry the evidentiary weight of a systematic review or large experiment.
It proposes a useful distinction, draws on established learning theory and references emerging research. It does not itself estimate the size of AI’s effect on retention across populations.
Its broad language about deep cognitive and metacognitive processing should therefore be interpreted as theoretical framing rather than a quantified universal law.
The controlled studies published around it improve the evidence base, but they remain heterogeneous.
High-school mathematics is not university writing. Visual analytics is not language learning. A constrained tutor is not a general chatbot. Short-term post-tests are not career-long expertise.
A responsible synthesis preserves those differences.
Study design has to separate assistance from acquisition
A credible AI-learning study needs more than an intervention group and a better-looking final product.
At minimum, the design should make the timing of assistance visible. What happened while the learner had access to the system? What happened after the support was reduced, changed or removed? Were both groups given the same opportunity to practise the underlying operation, or did one group simply receive more explanation? Was the post-test close enough to the training task to reward imitation, or different enough to test transfer?
The strongest designs can separate several outcomes rather than compressing them into one score: assisted task performance, delayed retention, transfer to a structurally related problem, error detection, confidence calibration and performance when the tool gives a plausible but wrong answer.
This matters because an intervention can succeed on one dimension and fail on another. A tutor might improve immediate completion and conceptual understanding while leaving unaided speed unchanged. A writing assistant might improve structure while weakening recall of source details. A coding copilot might increase throughput while simultaneously teaching useful patterns to novices. None of those profiles can be summarized honestly as “AI helped” or “AI harmed.”
For the Institute, the evidentiary unit should be the capacity profile, not the headline direction of a single outcome.
Fading support is a design hypothesis worth testing
Learning sciences have long used fading: strong support early, less support as competence develops.
Generative AI makes fading both easier and harder. It is easier because an adaptive system can vary hints, questions and examples in response to a learner. It is harder because the system can always produce the finished answer, and users have strong incentives to request it when under time pressure.
A useful design question is therefore not merely whether guardrails exist, but whether the system changes the locus of work over time. Early on, it might model a reasoning process. Later, it might ask for an attempted explanation before responding. Eventually, it might provide only diagnostic feedback or a counterexample.
This is a hypothesis about architecture, not a proven universal recipe. The PNAS comparison between GPT Base and GPT Tutor suggests that interaction constraints can change later outcomes, while the visual-analytics RCT suggests that proactive scaffolding can support comprehension. Neither study establishes an optimal fading schedule.
The research opportunity is to test fading directly: when should assistance retreat, which components should remain, and how should the answer depend on prior knowledge?
Cognitive offloading is not one behaviour
“Offloading” can sound like a moral diagnosis, as though every external aid represents lost capacity.
In reality, people offload for different reasons and with different consequences. Writing a deadline in a calendar preserves memory for other work. Using a calculator can free working memory for modelling. Asking an AI system to generate a first-pass bibliography may save time while increasing the need for source verification. Asking it to solve every practice problem may remove precisely the retrieval and generation the lesson was meant to cultivate.
The relevant variable is not whether cognition is externalised. It is which operation is externalised at which stage of learning.
That distinction also protects accessibility. A learner who uses text-to-speech, translation or motor assistance is not failing to learn merely because some task component is delegated. Educational design should identify the target capacity and avoid confusing access support with replacement of that target.
Institutions should measure what remains when convenience is removed
Schools and universities often collect the easiest data: submission quality, completion rate, time on task and grades. AI can improve all four without answering the learning question.
If an institution genuinely cares about durable capability, some assessment moments need to reveal what remains under changed conditions. That does not require a return to surveillance-heavy closed-book testing. It can mean delayed oral explanation, transfer problems, source critique, error diagnosis, reconstruction of an argument from notes, or comparison between an AI-assisted draft and the learner’s later independent reasoning.
The same logic applies in workplaces. Productivity metrics can rise while expertise becomes brittle. If junior staff no longer practise the intermediate judgments that senior expertise depends on, the organisation may discover the loss only when an unusual case exceeds the tool’s competence.
This is a civilization-level version of the Comment’s modest methodological point. Systems optimise what they count. If they count polished outputs but not retained human capability, they should not be surprised when the two diverge.
What the evidence does not show
The 2025 Comment does not show that generative AI necessarily harms learning.
It does not show that higher assisted performance is meaningless.
The PNAS field experiment does not generalize automatically to every subject, age group or AI design.
The visual-analytics RCT does not show that proactive scaffolding will improve every form of learning.
No current study justifies a universal rule that AI should be removed whenever learning matters.
Nor does evidence of successful collaboration prove that unaided human capacity can safely disappear.
The open question is architectural: which cognitive work should assistance preserve, prompt, fade or take over under a given learning goal?
What changed with this Comment
The paper gives researchers a cleaner dependent variable.
Instead of asking whether students produce better work with AI, it asks whether the person has changed in a way that persists beyond the immediate assistance.
That change may be knowledge, strategy, calibration, transfer, error detection or the ability to coordinate effectively with an AI system.
Whatever the target, it has to be measured separately from the polished artifact on the screen.
The difference between performance and learning is not a semantic refinement.
It is the difference between a tool that helps someone cross one bridge and an educational process that changes what bridges they can cross next.
Research references
Yan L, Greiff S, Lodge JM, et al. Distinguishing performance gains from learning when using generative AI. Nature Reviews Psychology. 2025;4:435–436. https://doi.org/10.1038/s44159-025-00467-5
Bastani H, Bastani O, Sungu A, Ge H, Kabakcı Ö, Mariman R. Generative AI without guardrails can harm learning: Evidence from high school mathematics. Proceedings of the National Academy of Sciences. 2025;122(26):e2422633122. https://doi.org/10.1073/pnas.2422633122
Yan L, Martinez-Maldonado R, Jin Y, et al. The effects of generative AI agents and scaffolding on enhancing students’ comprehension of visual learning analytics. Computers & Education. 2025;234:105322. https://doi.org/10.1016/j.compedu.2025.105322
Roediger HL III, Karpicke JD. Test-enhanced learning: Taking memory tests improves long-term retention. Psychological Science. 2006;17(3):249–255. https://doi.org/10.1111/j.1467-9280.2006.01693.x