The Struggle Is the Point: What Agentic Harnesses Do to How We Learn

I built a Rust workshop, borrow-native, on a bet that isn’t actually mine. Torrance’s “criteria compliance” critique names the failure mode this fights: explicit criteria blunt discovery learning the moment a learner can read the answer off the grading sheet. A real body of productive-failure research backs the fix. Let a learner generate a wrong answer before showing them the right one, and it sticks harder, provided the audience already knows enough to generate something wrong on their own. Mine already did. They just had a coding agent sitting next to them too, which is the part the literature never had to account for.

It worked for four modules straight. A naive attempt (defensive cloning where a borrow would do, a wildcard match instead of an exhaustive one) passed every automated test exactly as cleanly as the correct answer. Only a human reading the “why this is hard” reveal ever caught the gap. Exactly the shape I wanted.

The rubric was never the problem by itself

Then an agent-run review panel, three personas built to read like different real reviewers, found something I’d missed for four straight modules: every rubric (shared with the learner before the attempt, because hiding grading criteria is its own well-documented bad practice) states the correct shape directly, in writing, as a scored line item. “Correctly uses a lifetime-scoped borrow instead of cloning.” Not hidden. Not subtle. Right there before you write a line of code.

The reflex fix is to reword it. Say what the artifact should observably do, not what technique produces it. “The finished code compiles without an unnecessary allocation” instead of naming the borrow directly. That’s a real improvement for a human skimming the page slowly, and it is close to useless for the actual audience this workshop was built around.

A human reading a property-phrased hint still has to do the work of connecting the property to the technique. A coding agent does not. Paste the property into a frontier model with zero other context and it hands back the exact idiom the property describes, because deriving a technique from a stated property is not meaningfully harder than deriving it from a stated answer. Rewording the rubric didn’t close the gap. It moved the gap somewhere I could no longer see it, which is worse.

What actually got named, not fixed

Here’s what actually shipped, and what it doesn’t claim to fix.

The workshop adopted property-phrased criteria anyway, because it’s still a real improvement against casual skimming. Alongside it, every module now carries an explicit, unscored reflective question asking the learner to name what they actually learned versus what their agent demonstrated on their behalf. And the question got generalized into a standalone self-check: three questions, answered without looking back at the code, before accepting any diff as done. Name the concept the exercise was testing. Predict whether the diff would pass a strict check before running it. Name how a plausible, honest-but-naive attempt would have gotten it wrong anyway, and still passed.

A rubric wording fix protects a human skimming the page. It does nothing against an agent that already knows the answer the moment you paste the property in front of it.

None of that closes the gap mechanically. The workshop’s own decision record says so in writing: a coding agent can derive a technique from a stated property just as fast as from a stated answer, and no amount of rubric wording changes that. What the fix does is refuse to pretend otherwise. It gives the learner a real, honest tool for checking their own understanding, instead of a rubric that quietly implies the checking already happened.

The question worth stealing

If you’re building any agent-assisted learning design (a course, an onboarding flow, a certification exercise), there’s one diagnostic worth running before you ship it: paste the task’s instructions into a frontier coding agent with no other context, and see what comes back. If it hands back the lesson, you’re fine. If it hands back the finished answer, you have a spoiled exercise, and no amount of rewording the visible text will fix the parts of it that are checkable at all.

That splits cleanly along one line. A deterministic gate, a test suite that has to go green, is immune by construction, because passing it reveals an outcome, never the technique that produced it. A conceptual or idiom-level criterion is exactly where the bypass lives, because naming the target shape is what makes it checkable by a human in the first place. You can’t have transparent grading criteria and agent-proof grading criteria for that second category. Pick one, and be honest with your learners about which one you picked.

Say plainly what the fix protects and what it doesn’t. The struggle was never the diff. It was always the part a rubric can’t check for you.

That’s why the self-check isn’t scored. A graded honesty check is still a rubric, and a rubric is still gameable by the same agent. The three questions live in github.com/dermdunc/agentic-learning-discipline now, extracted verbatim from borrow-native, plus node bin/check.mjs, a small CLI that walks you through them instead of letting you skim past.

$ node bin/check.mjs
Agentic Learning Discipline - a check, not a grade.

Answer these before rereading the diff. If you have to go check first, that is
itself the answer to this question.

- Name the concept this exercise was actually testing, without looking at the code.
  (Not "it passed the tests" - the one sentence a reviewer would use to describe what this diff demonstrates.)
> a lifetime-scoped borrow
- Predict whether the diff would pass the strict lint/check before running it.
  (Not whether the test suite passes - whether the idiom would survive a linter's default settings.)
> yes
- Name the specific way a plausible, honest-but-naive attempt would have gotten this wrong, and why it would have passed every automated check anyway.
  (If you can't name one, you likely haven't engaged with why this exercise is hard, only with why your diff compiles.)
> it would clone and still pass
One last question, honestly, not for anyone else to see:
How many of the three could you answer without having to go look? (0-3) > 2
2 of 3 without looking. Might just mean you were moving fast - use judgment.

This is a real, complete run, hints and all, not a trimmed-for-readability version, so it’s exactly what you’ll see if you type the same three answers yourself.

It doesn’t grade you. It just makes it harder to lie to yourself in the thirty seconds before you move on.

Next time: what it actually costs to trust a model’s own verdict on someone else’s work, and why “the judge sounded confident” isn’t evidence.