Reading an AI vendor deck: a teardown.

title: "Reading an AI vendor deck: a teardown." dek: "Three claims that show up in almost every AI vendor pitch, and the questions that reveal which version of those claims actually ships." date: "2026-04-08" pillar: "teardowns" author: "aaron" tags: ["vendor-evaluation", "teardowns", "buyer-questions"]

A specific kind of meeting happens about three times a quarter. A vendor has been pitching one of our clients for two months. The client wants a second opinion before signing. We get the deck, the recorded demo, the references, the SOW draft. The ask is some version of tell us what's real in this pitch and what isn't.

The pitches are not identical, but they rhyme. Three claims show up in nearly every one. Each can be true or false depending on what's underneath. The right questions distinguish the two.

This post walks the three.

Claim 1: "Reduces [X] by [N]%."

The most common version is reduces ticket response time by 60% or reduces processing time by 80% or cuts onboarding from two days to thirty minutes. The number is specific. The metric is operational. The slide is confident.

The question is not whether the number is true. The number is almost always true for the customer it was measured on. The question is what version of X and what version of baseline produced the number.

The questions that pry it open:

• What was the baseline you measured against? If the answer is the customer's previous state, fine. If the answer is industry average, the number is marketing, not measurement.
• What process were they running before, and what did they change at the same time? Most AI deployments coincide with a workflow redesign. If the redesign would have produced 40% of the gain even without the AI, the 60% number is shared credit.
• Over what time window? Six weeks after deployment, the team is paying attention. Six months in, attention drift erodes the gain. Ask for the eighteen-month number.
• What metric were you measuring on the same dimension before the deployment? If they don't have a baseline measurement of the same metric, the percent reduction is computed against an estimate. Estimates flatter.

What credible looks like, on this claim: the vendor produces a baseline measurement methodology, a definition of the metric, an explanation of what else changed in the workflow, and a same-metric measurement at 6, 12, and 18 months. Reduces X by N% becomes a paragraph of context, not a slide.

Claim 2: "Out of the box. No integration required."

This appears on the slide titled Get started in 10 minutes or No engineering team needed. The pitch is that the AI sits above your existing systems and just works.

If the AI is doing anything other than reading and writing through a browser, this is rarely true. The product touches your CRM. Or your ticketing system. Or your payroll. Or your file storage. Each touch is an integration. Each integration has authentication, data shape, rate limits, error handling, and edge cases. Out of the box is the demo, not the deployment.

The questions:

• Which of my systems does this need to read from? Make them list the systems. Then ask: Show me the connector for each one.
• Where does the data live during processing? If the answer is in our cloud, you have a data-residency conversation, a security-review conversation, and a contract conversation, all of which take more than ten minutes.
• How does the AI handle a system being down? If the answer is it queues and retries, ask for the queue depth, the retry policy, and the failure escalation.
• What does the deployment look like at 100 employees, at 500, at 2,000? If the same architecture serves all three, ask for a Tier 4 reference.

What credible looks like: the vendor produces an integration list with linked connectors, a deployment guide that runs more than five pages, and a customer success story at your scale with the integration scope documented. Out of the box doesn't appear in the operational documents. It appears only on the marketing slide.

Claim 3: "[Big customer] reduced [thing] by [amount]."

This is the social-proof slide. It usually includes a logo, a name, a quote, and a number. The number is the same shape as Claim 1, but the social proof shifts the conversation from will this work to if it worked for them, it will work for me.

The substitution is the problem. It worked for them is a claim about a customer who is not you. The right questions surface the differences:

• What was their headcount, their stack, and their team composition? If their stack is Salesforce + Snowflake + a dedicated MLOps team, and yours is HubSpot + spreadsheets + one ops manager, the deployment is a different deployment.
• Who at that customer ran the project? If the answer is their VP of AI Strategy, you need someone in your org capable of running the project the same way. If the answer is their COO with a six-week mandate, the project shape is closer to yours.
• How long did the deployment actually take? Six weeks on the slide is often six months in the customer's calendar. Ask for the actual go-live date and the actual signoff date.
• Can I talk to that customer directly, without your CSM in the room? The willingness to facilitate that conversation is itself data. The unwillingness, much more so.

What credible looks like: a reference customer at your scale, in your sector, with your stack, willing to take a 30-minute call without a vendor chaperone. If the vendor produces this, the social-proof claim is real. If they can't, it isn't.

What this is really for

These questions aren't traps. They are the questions any operator would ask if they had time to ask them. Most operators don't, because the diligence pace expected of an SMB or mid-market buyer is a fraction of the diligence pace expected of an enterprise buyer, and the budget for outside help is a fraction of what makes outside help pencil.

The point of running the questions is not to embarrass the vendor or to win a debate. The point is to find out which version of the product will ship into your operation. The vendor's responses, asked in the spirit of buying rather than rejecting, are usually honest. When they aren't honest, you find out before signing.

If you'd like a second opinion on a deck you're sitting with, the discovery call is the right starting point. Send the deck ahead of time. We'll walk it with you.