How to Evaluate a Web or AI Vendor Before You Sign Anything

The portfolio is the weakest signal you have

Every vendor has a highlight reel. Case studies get written by the winning team, for the client who was happiest, about the project that went closest to plan. None of that tells you how the vendor behaves on your project, when your requirements turn out to be messier than the slide deck assumed. If you're choosing between agencies, freelancers, or in-house hires for a dashboard, a website rebuild, or an AI feature, the portfolio review should take an afternoon. The rest of the evaluation is about process, pricing, and how they handle the parts of the project that haven't happened yet.

This matters more for AI work than for a standard web build, because AI projects fail in ways a portfolio can't show you. A vendor can ship ten successful chatbots and still be the wrong choice for your use case if none of those ten needed the reliability yours does. Ask what happened when one of their AI projects didn't work, not just which ones did.

How they price tells you how they think about risk

There are two pricing models you'll see, and the choice isn't neutral — it tells you who's holding the risk.

Fixed price means the vendor commits to a scope and a number. This works when the scope is genuinely well understood — usually after a discovery phase has pinned down the requirements — and it puts the risk of underestimating the work on the vendor, not you. Fixed-price quotes offered before any discovery, on a project with open questions, are a red flag: either the vendor padded the number heavily to cover the unknowns (you're overpaying for certainty), or they didn't think about the unknowns at all (you'll be renegotiating scope in month two).

Time and materials means you pay for hours worked, and the risk of scope uncertainty sits with you. This is the honest model when requirements are still being validated — a new internal tool, an AI feature where you don't yet know what "good enough" looks like — but it only works if the vendor gives you visibility into the hours as they're spent, not a lump invoice at the end of the month.

The arrangement to be wary of is a fixed-price quote for a project that obviously still has open requirements. That's not a vendor who de-risked the build for you — it's a vendor who's going to either cut corners to hit the number or come back asking for more money once "out of scope" starts appearing in every email.

A reasonable middle ground, and what you should ask for if a vendor doesn't offer it: a paid discovery phase (one to three weeks, fixed price, small) that ends in a scoped, fixed-price proposal for the build. You're paying for the vendor's attention before asking them to commit to a number, which is the only way a fixed-price number means anything.

Questions that separate vendors faster than a portfolio review

"What's a project where it didn't work, and why?" A vendor with no answer either hasn't done enough work to have failures, or won't tell you about them. Either way, that's information. The honest version of this answer usually involves a specific technical misjudgment or a scope that got away from them — not "the client just wasn't a good fit."

"Who's actually doing the work?" Agencies sell you the partner on the call and staff the project with whoever's free. Ask for the names and backgrounds of the people who will write the code or configure the model, not the people who will be in the kickoff meeting.

"What does maintenance look like after launch?" A web app or an AI feature is not done at launch — it needs monitoring, bug fixes, and in the AI case, ongoing evaluation as usage patterns shift. If the vendor's answer is vague, or if their business model only makes sense if they disappear after delivery, budget for a second vendor to pick up the maintenance, or push for a maintenance retainer in the contract now.

"What would make this project fail?" This is the question most vendors aren't asked and most don't want. A vendor who can name specific risks — a third-party API that's flaky, a data source that's messier than it looks, a timeline that assumes no scope changes — has actually thought about your project. A vendor who says "we don't anticipate any issues" either hasn't looked closely or is managing your expectations instead of informing them.

AI-specific diligence that a standard web vendor won't need

If the project involves an LLM, a few extra questions matter:

• How do they measure whether it's working? "The demo looked good" is not an evaluation method. Ask whether they built a test set of real queries and expected answers, and whether they re-run it when they change the prompt or the model. If the answer is no, you're the one who'll discover the failure modes, in production, from your customers.
• What happens when the model is wrong? Every LLM-based feature has a failure rate above zero. A vendor who's thought about this has an answer involving confidence thresholds, human review queues, or graceful fallbacks — not just "it's pretty accurate."
• What's the actual cost per use, at your volume? API costs for AI features scale with usage in a way flat-fee web hosting doesn't. A vendor should be able to model your cost at 10x current volume, not just quote a build price and leave the running cost as a surprise.

The takeaway

Don't evaluate vendors on their best work — evaluate them on how they talk about risk, pricing, and failure. A vendor who offers a scoped discovery phase before quoting a fixed price, names a real project that went wrong, and can describe how they'd catch an AI feature failing before your customers do, is telling you more than any case study will.