Seven Questions to Ask Before You Buy an “AI Agent”

Product documentation checked November 4, 2026. Examples are illustrative; features and availability can change.

Short answer: Before you buy an AI agent, ask the vendor to show it finishing your real task end to end, name exactly what it can touch, explain how it hands a customer to a person, agree on one number you’ll measure, and spell out what you keep if you leave. A vague answer is an answer.

Every software demo this fall seems to have the word “agent” in it. Some of those products really are agents: software that takes a goal, makes decisions and acts inside your tools. Plenty of them are a chatbot with a new label and a higher price.

Gartner estimates that only about 130 of the thousands of vendors selling “agentic AI” are real, a gap it blames on “agent washing,” rebranding existing chatbots and assistants as agents (Gartner, June 25, 2025). The same release predicts that over 40% of agentic AI projects will be canceled by the end of 2027, due to escalating costs, unclear business value or inadequate risk controls.

That doesn’t mean agents are a bad buy. It means the demo isn’t enough. A demo is the vendor’s best day. You’re buying your worst Tuesday: the day a customer asks something odd, the CRM is half updated and nobody on your team has time to babysit software.

If you run a six-person home service company in Plano and a vendor just pitched you an agent to answer web inquiries and book estimates, keep that pitch in front of you for every question below.

Here are the seven questions I’d ask before signing anything. Ask them on the call, and write the answers down.

1. Can you show it doing my task, start to finish?

Not a sample company. Not a slide. Your task, with your kind of input. If the pitch on your desk is an agent that answers website inquiries and books estimates, hand the vendor three real (scrubbed) inquiries from last month and ask them to run all three live.

A good answer sounds like: “Send us the examples and we’ll walk through them on a second call, including the one we can’t handle.”

Red flag: the demo only ever runs on the vendor’s own sample data.

2. What can it touch, and what can it change?

This is the question most buyers skip. Reading your calendar is one thing. Sending email as you, editing deal records or issuing refunds is another. Ask for a plain list: every system it connects to, and for each one, whether it can only read or can also write. An agent that books estimates is writing to your calendar, so that pitch already needs a write answer.

Then ask how you’d turn a permission off. If the answer involves a support ticket and a week of waiting, you don’t control it.

3. What happens when it doesn’t know the answer?

Every agent hits a question it can’t answer. What matters is what it does next. Does it guess? Does it say “let me get someone” and actually route the conversation, with the history attached, to a named person?

Your customers care about this more than vendors admit. Ask the vendor to show you the handoff, not describe it.

4. Who checks its work, and where would I see a mistake?

Ask for the log. A real agent leaves a record of what it read, what it decided and what it did, with a timestamp. You should be able to open any conversation from last week and see why it said what it said.

A good answer sounds like: “Every action is logged, and you can set it to require approval before it sends anything for the first 30 days.”

Red flag: “It’s AI, it learns as it goes.” That’s not an audit trail.

5. How will we know in 60 days whether it’s working?

Pick one number before you buy, and only one. For most service businesses the number is estimates booked per 100 inquiries. Measure it now, while the old way is still running, so you have a baseline to compare against.

If the vendor can’t help you choose the number, that’s the “unclear business value” problem from the Gartner prediction, showing up before you’ve paid a dollar.

6. What does it cost when we get busier?

Agent pricing is often per conversation, per task or per “credit,” and the bill grows with your volume. Ask for the price at your current volume, at double it and at your busiest month last year. For a home service company, that busy month is usually the first heat wave or the first hard freeze. Ask what happens when you go over: does it stop working, or does it keep running and send you an overage bill?

Here’s the math that makes this worth an awkward question. Say the agent runs $400 a month plus a $2,500 setup fee, and you cancel after six months because nobody could prove it helped. That’s $4,900 out the door, before you count the hours your team spent correcting it. (Illustrative numbers. Plug in the quote on your desk.)

7. If we leave, what do we take with us?

You’ll spend weeks teaching an agent your services, your rules and your exceptions. Ask whether you can export that: the instructions, the conversation history, the workflows. Ask how much notice the contract needs and whether your data is deleted when you go.

Red flag: a 12-month term with no exit for performance, on a product you’ve never run.

How do you score the answers?

Give each question a 2 (clear, shown to you), a 1 (described but not shown) or a 0 (dodged). Fourteen is the max.

1. Under 8: walk. Another vendor will be on your calendar next week.
2. 8 to 11: go back with the questions they missed, in writing, and wait for written answers.
3. 12 to 14: run a paid pilot on one task with a written exit date.

Weight question 2 more than the rest. An agent that can write to your systems and scores a 0 on permissions is a hard no, whatever the total says.

Is this checklist for you?

Skip it if you’re trying a $20 a month AI assistant on your own laptop. Kick the tires and cancel if you don’t like it. These questions are for software that will act inside your business: answering customers, updating records or sending messages with your name on them. That’s where your worst Tuesday shows up.

If there’s a proposal on your desk right now and you want a second read before you sign, that’s the kind of thing I help with. See how I approach AI assistants.

Frequently asked questions

What’s the difference between an AI agent and a chatbot?

A chatbot answers questions. An agent takes actions toward a goal: it might read an inquiry, check your calendar, book the slot and update the CRM. That’s why question 2 matters so much more for an agent.

How long should a pilot run?

Long enough to cover a normal month of your work, usually 30 to 60 days, with the success number from question 5 agreed in writing before it starts.

Should I ask for a reference customer?

Yes, and ask for one your size, in your industry, who has used the product for at least six months. A logo on a slide isn’t a reference.