Judgment, kept human

The question

AI is very good at removing effort. Often, that is exactly what makes it useful.

But some forms of effort are how people learn to reason, question, connect ideas, tolerate uncertainty, and make judgments on their own.

So the question I started with was:

Which parts of thinking should AI make easier — and which parts should humans still have to do for themselves?

This was never really an argument against AI. My own experience has been almost the opposite. I use it as an extension of my thinking: external memory, another perspective, a way to synthesize ideas, challenge assumptions, and expose blind spots.

The distinction I kept returning to was between:

Where it started

Before generative AI, understanding still required reading across sources, holding ideas long enough to connect them, questioning what you read, and constructing your own mental model.

AI can now perform much of that work for us. That creates enormous opportunity — and raises a different question:

What happens if the cognitive work we outsource is also the work that develops our ability to think?

My first instinct was an AI that did not optimize only for the answer, but also asked whether answering immediately was helping the person become more capable.

The first hypothesis

Productive struggle

The AI could adjust how much cognitive work it asked the user to do.

If someone needed an answer in five minutes, give it. If they were trying to understand something, create more friction.

Over time, the system could learn things such as:

• whether the user is learning or completing a task;
• their tolerance for frustration;
• whether they tend to persist or disengage;
• how they respond to hints, analogies, debate, or examples;
• what kinds of reasoning they avoid or rely on too heavily.

The mechanism became a sliding scale between direct assistance and deliberate cognitive effort.

What success might look like

The more interesting question became what the system should optimize for.

Not intelligence in the abstract.

Not the number of prompts completed.

Not how impressive the AI appears.

The word I kept returning to was judgment.

Someone might still use AI heavily, but over time they could:

• ask sharper questions;
• identify assumptions earlier;
• recognize weak arguments;
• reason more comfortably under uncertainty;
• consider alternatives before committing;
• make more decisions without needing the AI to hold their hand.

The goal would not be to use AI less, but to become less cognitively dependent on it.

If reliance on the system for ordinary judgment increased over time, that might be evidence it had failed.

Success would look like greater confidence and capability outside the interaction.

What changed

“Socratic AI” is not an empty category. Existing products already use questioning, scaffolding, and withheld answers.

Making an AI less eager to answer is not enough.

The more interesting possibility is an AI that measures success by the user's growing ability to think and decide independently.

That creates an unusual tension: the product may succeed by making the user less dependent on it over time.