Ask ten things: where answers come from, what happens on unfamiliar phrasing, what happens when it does not know, how escalation works, how a resolution is defined, whether abandonment and handoffs are billed, whether there is a spend cap, whether you can audit billing, how your data is handled, and how long until you are live.
Key takeaways
- Get every billing answer in writing, not in a deck.
- A vendor that always answers is not grounded, it is confident.
- Escalation design is what customers actually judge AI support on.
- Time to first resolved question matters more than demo quality.
Vendor evaluation in this category is unusually hard, because every product describes itself identically and every demo goes well. These ten questions separate them faster than any feature comparison. Ask for written answers.
1. Where do the answers come from?
The options are a script, an intent library, retrieved passages from your documentation, or general model knowledge. Three of those are fine and one is not.
Follow up: can the system answer from general model knowledge when retrieval fails? That is where invented policies come from, and the answer should be no.
2. What happens with a phrasing nobody anticipated?
This separates scripted from generative systems in one question. Ask them to demonstrate live with a question you write during the call, not from their prepared set.
Scripted systems will repeat a menu. Generative ones will attempt an answer. Whether that answer is good depends on question one.
3. What happens when it does not know?
A system that always produces an answer is not grounded, it is confident. There should be an explicit path where it says it cannot confirm something and routes to a person.
This lowers the resolution rate, which is why some vendors resist it. It is also the single feature that makes customer-facing AI safe to deploy, so treat reluctance here as disqualifying.
4. How does escalation actually work?
Customers judge AI support almost entirely on what happens when it cannot help. Ask specifically:
- Can a customer reach a human on request, immediately, without negotiating?
- Does the handoff carry the full transcript, not just a summary?
- Does the receiving agent see what the AI already tried?
- How is the escalation trigger configured, and by whom?
A handoff that makes the customer start again converts a minor failure into a resented one.
5. How is a resolution defined?
If they price on resolutions, they wrote the definition. Get it in writing and read it carefully.
A vendor quoting seventy percent and one quoting thirty-five may be describing identical performance, if the first counts any conversation the customer did not escalate and the second counts only confirmed answers with no repeat contact.
6. Is abandonment billed? Are handoffs billed?
These are two separate questions and both matter.
If an abandoned chat is billable, you are paying for customer frustration. If a handoff is billable, you are paying the vendor for the product declining to do its job. Neither should be true, and both frequently are.
7. Is there a hard spend cap?
Usage pricing without a ceiling is an unbounded liability. An incident or a viral moment can multiply volume within hours and the invoice follows.
Ask what the mechanism is, not whether costs can spike. A hard monthly cap, alerts before you reach it, and a defined behaviour at the cap. The best behaviour is routing everything to your team so customers are still served while spend stops.
8. Can I audit billed usage?
If you are billed on a unit the vendor defines and counts, you should be able to inspect a sample. A vendor who will not allow this is asking for a level of trust that the arrangement does not earn.
9. How is my data handled?
Four sub-questions:
- Are our conversations used to train third-party models?
- Where is data stored, and can it stay in a specific region?
- What are the retention and deletion terms?
- Who are the sub-processors?
Get these in writing regardless of how the sales conversation goes. They are also the questions your security reviewer will ask later, so obtaining them early saves a round trip.
10. How long until we are answering real questions?
Not the demo. The deployment.
Ask for a reference timeline from a customer of your size. If the honest answer is six weeks with an implementation partner and you are a five-person team, that is a mismatch regardless of product quality, because the evaluation alone will outlast your patience.
The evaluation that actually decides it
After the questions, run the test:
- Take fifty real questions from your last month, including the badly phrased ones.
- Write your own resolution definition before any vendor sees the set.
- Run each candidate on the same questions.
- Score against your definition.
- Read twenty transcripts yourself.
- Track repeat contacts for two weeks after any live pilot.
Aggregate scores hide the failure modes that matter. An hour spent reading transcripts will teach you more than any report.
Fix your content first
Whatever you deploy reproduces your documentation faithfully, including its contradictions. Auditing for conflicting answers on refunds, pricing, cancellation, and security before you evaluate anything makes every vendor look better and makes the comparison meaningful.
You are trying to test the system, not your content. Get the content out of the way first.
Our own answers
For transparency, since we sell in this category. Fidiora answers from your own content and declines when it cannot ground an answer. Escalation is free, immediate on request, and carries the full transcript. A resolution is a genuinely closed issue: abandonment, timeouts, and handoffs are never billed, and the definition is published in full. There is a hard monthly spend cap you set. Customer conversations are not used to train third-party models. Setup is designed to take under an hour.
Ask every vendor the same ten questions and compare the answers side by side. That comparison is more informative than any of our marketing pages.
Frequently asked questions
What is the most important question to ask an AI support vendor?
How do I test AI support quality before buying?
Should I worry about AI hallucinations in support?
What data questions should I ask?
Resolve, don't deflect.
See Fidiora resolve a ticket, capture a lead, and keep the bill predictable.