bartkolendowski.com writing 2026-05-18 · 11 min
A row of dominoes mid-cascade, with a single red domino tipping the rest into a falling line

The Constraint Problem

How the Socratic Method Stops Agents From Producing Beautifully Wrong Work

May 18, 2026 · ~11 minute read · ~2,650 words

Everyone building with LLMs eventually hits the same wall: you give an agent perfect context, and it returns a polished, internally consistent artifact that is completely, irredeemably wrong.

Most tasks don’t fail this way. Some come back done first try; some need a few rounds of iteration to converge. But this third class doesn’t yield to either approach, and the more careful you were about setting it up, the more confusing the failure feels. What’s strange is that often the tasks that fail don’t even seem any more difficult than some tasks the agent has no issue with. Ask the agent to come up with a project for launching a new product, and while what it produces will look like a real project (10 milestones, 50 tasks, all internally consistent), you’ll eventually notice something: the project doesn’t work.

If you’re lucky this happens while you’re reading through it the first time, and you notice that a milestone doesn’t actually make sense, or that one of the tasks is simply not practical. If it’s your first time trusting an agent with a project like this, you’ll likely catch it later, when you’re on Task 8 of Milestone 3 and you realize that the proposed task is, for whatever reason, not doable (perhaps it depends on a vendor relationship you don’t have, or assumes a tool your team doesn’t use, or expects a pace your team can’t sustain).

But then something even worse happens. Inspecting things closer, you notice that this task is actually a dependency for Task 2 of Milestone 6. And Task 2 of Milestone 6 is a dependency for a few downstream tasks. Digging in deeper, you start to get a sense of dread: the project, as it stands, is actually worthless.

It’s tempting to think it’s the complexity of the task that broke things. But hand the same agent a year of transactional data scattered across a hundred PDFs in messy folder structures and ask it to reconcile against external accounts, normalize the format, and produce a clean 200-page audit report, and it’ll breeze through. That task is, by any reasonable measure, at least as complex as the project plan that just failed. But the audit task has built-in constraints the agent must satisfy when iterating towards its output: sums have to tie, accounts have to reconcile, formats have to validate. With the project plan, the agent has no comparable constraints to check its work against.

Where the constraints live

Constraints are the boundaries an agent’s output must respect: hard ones (a compiler will reject code that doesn’t compile) and soft ones (a written convention, a stated preference, a Socratic answer). Where these constraints live and who can access them determines whether an agent can successfully navigate a given task.

Coding has solved this. Decades of work have gone into making programming boundaries machine-readable: the rules for what ‘good’ code looks like live in compilers, type systems, tests, linters, and runtime errors. These are hard constraints the agent can bounce its work against. When you ask an agent to write a piece of code, the agent is able to propose a solution, run it, and get immediate feedback the moment it hits one of the dozens of constraints built into the system. It can iterate this way until it produces a solution that clears every boundary. No need for you to get involved beyond telling it your goal.

However, most real-world tasks do not have any built-in constraints. Compare the coding task above to asking an agent to put together a project to remodel a room in your home (remodels are convenient because the constraints are visible; the same structure plays out for project planning, hiring decisions, pricing calls, board communication), and assume you give the agent no context, just the bare ask. Unlike the coding example, the agent has no way to iterate to a final product; it can’t propose a solution and immediately test it against a hard constraint to see if it breaks.

In this extreme example, with no context given, the specific constraints necessary to produce a useful result aren’t reachable by the agent: which room, what budget, what timeline, what your tolerance for disruption is. So the agent calibrates against a different set entirely: what it learned during training, averaged across millions of examples of tasks of a similar shape. Its sense of “good” is internally coherent and often impressive. But it’s an average over the world’s tasks, not yours. It will put together a plausible project for remodeling a room because plausibility against the average is exactly what training optimizes for. What it won’t put together is a project for remodeling the room you want to remodel, in the way you want it remodeled. The gap between “plausible to the average case the agent has seen” and “correct for your specific situation” is where every confident-but-wrong output comes from.

Much of this can be alleviated by externalizing your knowledge so it becomes context the agent can access. But context alone almost always falls short for a simple reason: the explicit constraints you are capable of articulating in a vacuum are vastly outnumbered by the tacit constraints you’d never think to write down. Closing the full gap requires handling both.

Explicit knowledge

Explicit knowledge is everything you can articulate. It reaches the agent in three forms, doing different work.

The prompt. Familiar to anyone who’s worked with an LLM: what you write when you sit down to ask the agent for something. The prompt captures what’s top-of-mind: what you’re trying to do, what you remember matters, what you judge worth mentioning right now. It’s where most people put most of their effort, and it does real work. But it’s incredibly limited because it’s bounded by your memory and judgment in the moment of writing. When you’re writing the prompt you’re writing what you, at that moment, think you want. For tasks that have some sort of external constraint, this is often sufficient. For tasks that don’t (e.g., asking the LLM to put together a project for you), this is destined to fail.

External context. This is everything that lives outside the prompt and is available to the agent durably. On web interfaces it’s often accessed through ‘upload your files’ features or by allowing access to your calendar or mail; locally you can connect the agent to a repository of personal or business documents. The shapes vary; the function is the same: a body of context the agent can draw on automatically without being told to. Your calendar has a vacation booked next month, so the remodel can’t kick off during it. Personal docs note you’re expecting a baby in two months, so the timeline might need to slip. The home blueprint shows the room is downstairs, with no nearby washroom and far from the kitchen, which lets the agent flag whether that’s the ideal place for the baby’s room. The product catalog. The team doc. Prior decisions you’ve already made and don’t want re-litigated. These are things you’d never think to include in a prompt, not because they’re unimportant, but because they’re not top-of-mind when you’re writing the request. The agent draws on them automatically, so they act as constraints even when you’d have forgotten to mention them.

Framing. Stating your framing of this particular task in your own words before any generation. “I want the first milestone to focus on gathering context: interviewing stakeholders, reading what’s been written, mapping existing constraints. Subsequent milestones should build on that foundation, not parallel it. Make sure the final deliverable is portable and doesn’t depend on me to interpret.” A framing isn’t a separate document; it’s part of the prompt, scoped to this one task. Without one, the agent bounds the problem with its own framing, pattern-matched from training data. With one, it conforms to yours.

All three forms function as constraints. The agent’s output is forced to navigate what’s in the prompt, what’s in the saved context, and the framing you’ve scoped; conflicts surface as questions or corrections rather than confident inventions. Together, they get you most of the way there. Many people already have a light form of context (a few docs they’ve written up hastily, or they’ve linked their calendar to an agent), but what most people miss is that having a comprehensive body of context is the difference between an agent that still makes mistakes (albeit fewer) across many tasks, and one that works across most tasks.

Note. Concretely, comprehensive context is breadth across categories:

Built up over weeks and refined continuously; the body grows as you notice what the agent assumed silently and got wrong. The common failure mode is stopping at one or two categories and concluding the approach doesn’t work. The difference is breadth across all of them, not any single document.

But all three forms are bounded by what you can articulate. That’s exactly where they fall short.

Tacit knowledge

The harder problem is what you know but never wrote down, and often never could have, because you don’t think of it as knowledge. Tacit knowledge is the kind of thing that surfaces only when a specific situation forces it to. You know it’s true. You’d recognize a violation of it instantly. But you can’t reliably enumerate it in advance.

Take the room-remodel example. Without ever having articulated it, you know you don’t want the kitchen opened up into the living room. It’s a preference you’ve gradually formed over years of cooking and entertaining, but it isn’t written down anywhere and you’d never have thought to mention it. The moment the agent proposes opening up the kitchen, you recognize it as wrong. That’s tacit knowledge: present, load-bearing, invisible until it’s needed.

With comprehensive explicit context, the agent will produce something noticeably better than the version with no context at all. You’ll see fewer obviously-wrong outputs: fewer Task-8-of-Milestone-3 surprises, fewer milestones built on assumptions that don’t fit your reality. But the last bit is what separates a useful output from a misleading one. The output that’s almost right and quietly off in some way you can’t immediately name is the version that costs you the most: you commit to it, build downstream work on top of it, and only catch the problem weeks later when the consequence shows up.

One mechanism closes that gap: extracting hard constraints from your tacit knowledge in real time.

The Socratic method. This is what actually closes the loop. Instead of just generating an output, at the end of your prompt you tell the agent to surface questions that should be answered first. This is what draws out the tacit knowledge. “Does the project sponsor sign off after each milestone, or only at the end?” “Is the quality bar derived from first principles, or matched to what competitors do?” “What does ‘done’ look like here: shipped product, signed contract, internal handoff?” “Are there budget or timeline constraints that should rule out approaches before we even consider them?” Each question is targeted at a gap the agent has detected, a place where a boundary is missing and where the agent would otherwise silently fill the void with training priors. Your answer to each question becomes a firm constraint the rest of the work must satisfy. The mechanism works because it doesn’t ask you to enumerate your tacit knowledge in advance. It draws the relevant piece out exactly when the task needs it, in response to the specific question only that task could have surfaced.

A warning on implementation: if you simply ask the agent to ‘ask me questions,’ it will default to a generic, 20-point consultant questionnaire. You must constrain the Socratic step itself. Tell the agent: ‘Ask me a maximum of 3-5 highly targeted questions about constraints, preferences, or blind spots that would drastically alter the shape of this deliverable. Do not ask me for generic business context.’

Take a concrete business example. You need to draft a quarterly strategy memo for your board. The frame: you’ve hit a plateau in your core market, and the board wants the plan. You’ve asked the agent to do this before. What came back was four pages of polished prose, well-formatted, structurally coherent, and entirely wrong. It read like it had been written by a consultant who’d never met your board.

The framing step gives the agent the seed. Something like: “Quarterly memo for the board. Frame: we’ve plateaued in our core market and the board wants the plan. They value capital efficiency over growth-at-all-costs. Three strategic options maximum, each with investment, expected outcome, and at least one named risk. Tone: direct and operator-flavored, not consultant-flavored. Under three pages.”

The Socratic step then forces the agent to elicit the tacit context before drafting. You see questions like: “Are there strategic directions that are off-limits because of prior board conversations?” “Has any board member been signaling concern about a specific topic offline?” “What’s the realistic financial picture for the next 18 months?” “Is there a specific question the board has asked recently that this memo should answer directly?”

You answer. The head of the comp committee has been hinting comp needs to be on the table. One board member is privately skeptical of one of the directions you were going to recommend. The CFO has been quietly pushing cost cuts that aren’t yet in the public conversation. None of this is in any document the agent could have loaded. All of it changes the memo.

What you get back is a memo that engages with the actual situation your board is in. The structure is what your framing specified. The content reflects the constraints your Socratic answers revealed. The strategic option the skeptical board member doesn’t trust gets addressed head-on rather than glossed over. The comp question gets surfaced rather than ignored. The capital-efficiency frame runs throughout.

The agent didn’t write a better memo because it got smarter. It wrote a better one because it had strict operational realities to navigate.

Together (comprehensive explicit context and framing, plus tacit knowledge surfaced via the Socratic method), the agent has strong constraints across the full surface of a task. It can iterate against them the same way it iterates against compilers and tests when it writes code. The output is checked against your reality, not against the training distribution’s average. The third class of task (the polished-looking artifact that comes back confidently wrong) stops happening.

Take this approach and you’ll notice that most of the tasks you’ve previously thought to be intractable or ‘too complex’ are just as accessible to agent generation as the ones you’ve had success with.

The longer view

This is not a system you’ll build overnight. Comprehensive context takes weeks or months. Finding the prompts and Socratic patterns that work for you takes trial and error. But the work compounds in a way the model-of-the-day’s output does not.

Frontier models will keep improving. The naive reaction is to wait: build the context once the smarter model arrives. The right reaction is the opposite. Better reasoning models are better at leveraging the context you’ve built, not at substituting for it. Each new generation multiplies the value of your existing context library. Each Socratic pattern you’ve refined keeps paying off across model upgrades.

Most investments in agent tooling become obsolete with the next release. This one doesn’t. The work you do now is what every future model builds on.

← back to bartkolendowski.com writing · 2026-05-18