Build vs Buy AI Support: The Prototype Is Five Percent of the Work

Connect a language model to your documentation and you will have something answering questions sensibly by Sunday evening. That experience is why so many engineering teams conclude buying is unnecessary.

The prototype is roughly five percent of a production system, and the other ninety-five percent is the part that determines whether it is safe in front of customers.

What the prototype does not have

Retrieval tuning. Getting the right passage back for a real question, consistently, across a messy documentation set. Chunking strategy alone quietly determines answer quality: procedures split mid-sequence produce confidently incomplete answers, and you will not notice in a demo.

A refusal path. The prototype answers everything, which is exactly the behaviour that produces invented policies in production. Building an honest “I cannot confirm that” that fires reliably is harder than it sounds, because it requires a confidence signal you trust.

Escalation design. What triggers a handoff, what it carries, how the agent receives it, what the customer is told. This is the part customers actually judge and it is entirely absent from a prototype.

Permission scoping. If the system does anything beyond answering, every action needs a bounded scope, a limit, and a reversibility preference. Unbounded write access to customer accounts is not an acceptable production position.

Audit logging. Every action and every answer, traceable, for disputes and for your own quality review.

An evaluation harness. This is the big one, and it is the item most consistently missing from build estimates.

The evaluation problem

You cannot ship customer-facing AI without a way to detect whether it is getting worse.

That means a test set of real questions with known-good answers, a scoring method, and a process that runs it when anything changes: a model update, a retrieval change, a documentation edit. It also means keeping the test set current, because a test set that reflects last year’s product tests the wrong thing.

This is genuine ongoing engineering work. It is also invisible, unglamorous, and the first thing deprioritised when the team is busy, which is precisely when it matters most.

The maintenance nobody scopes

Once live, someone owns:

• Retrieval quality as documentation grows and changes
• The refusal threshold as answer patterns shift
• Escalation rules as the product changes
• Model updates and their regression effects
• Weekly answer sampling and quality review
• Cost monitoring and optimisation

That is a permanent part-time role at minimum. It does not appear in a build estimate because build estimates are about building.

The opportunity cost

This is the argument that usually decides it.

Every engineer week spent on support infrastructure is a week not spent on the product you sell. For most software companies, support automation is not a differentiator. Customers do not choose you because your support assistant is better architected than the alternative.

Compare the quarter of engineering time plus permanent ownership against a subscription, and then compare both against what that engineering time could have produced in your actual product.

When building genuinely makes sense

Three situations, and they are narrower than they look:

Support automation is core to your product. If you sell to support teams, or automation is a feature you resell, building is strategic rather than operational.

You have requirements no vendor meets. Unusual data residency, an air-gapped environment, or a domain where general vendors genuinely cannot ground answers. Be honest about whether this is true or whether it is a preference.

You already run substantial machine learning infrastructure. Retrieval systems, evaluation harnesses, and monitoring already exist, so the marginal cost of one more application is genuinely low.

If none of those apply, the build is usually a preference for control expressed as a cost argument.

The honest middle path

Some teams build the retrieval layer and buy the rest, or buy first and build later once they understand the requirements properly.

Buying first has a specific advantage: three months of production experience tells you what you actually need, which is a much better specification than the one you would write today. If you then build, you build the right thing.

A test worth applying

Ask your engineering lead two questions:

1. Who owns the evaluation harness in eighteen months?
2. What are they not building instead?

If the first has no answer and the second has an expensive one, the decision is usually clear.

We sell in this category, so weigh that. The part we would argue regardless: the prototype is not the project, and estimating from the prototype is the most common reason these builds run over.