Why a Vendor Cannot Give You an Honest Second Opinion

This is not a cynicism argument. It is a structural one. A vendor who sells you an implementation, a platform license, or a managed service has a direct financial conflict on every dimension that matters in a real review:

• Feasibility: Admitting your use case is a poor fit costs them the deal. The honest answer is sometimes 'this problem does not need an LLM.'
• Scope of data work: Underestimating the data pipeline effort is how proposals stay affordable on paper. The real number often triples the headline estimate once you factor in cleaning, labeling, retrieval tuning, and ongoing refresh.
• Eval plan: A vendor who built the thing cannot define the evaluation criteria objectively. They will optimize the demo for the metrics they chose, not the metrics your business actually needs.
• Lock-in: Every vendor has proprietary connectors, fine-tuned model weights, or a custom vector store schema that makes migration painful. They are not incentivized to quantify that exit cost for you.
• Alternatives: No vendor will tell you that a lighter-weight open-source solution covers 80 percent of your requirement at 10 percent of the cost.

The conflict is not malicious. It is structural. The only fix is independence.

The Five Things an Independent Review Actually Checks

1. Feasibility Against the Real Problem

The first question is whether the stated problem is actually an AI problem. I have reviewed proposals where the underlying issue was bad data governance, a missing API integration, or an understaffed ops team. Adding an LLM layer on top of a broken process produces an expensive broken process. A real feasibility check traces from the business outcome back to the technical approach and asks: is this the minimum viable intervention, or is this the most expensive one that happens to be fashionable right now?

2. Hidden Data Work

This is where most plans collapse in production. Vendors quote an implementation timeline that assumes clean, labeled, consistently formatted data already exists in one place. It almost never does. An independent reviewer asks to see the actual data sources: schemas, update frequency, access controls, quality samples. I estimate data prep work separately from model work, and I flag when the data work is larger than the model work, because that is the norm, not the exception. A realistic timeline for a mid-size RAG deployment over internal enterprise documents is 6 to 10 weeks of data work before a single meaningful eval can be run. Most vendor proposals allocate 2 weeks.

3. The Eval Plan

If a proposal does not include a written eval plan with specific metrics, thresholds, and a process for handling failures, it is not an engineering plan. It is a demo roadmap. I look for: named ground-truth datasets (not synthetic ones the vendor generated), task-specific metrics (RAGAS scores for retrieval, F1 for classification, human preference rates for generation), latency and cost budgets per query, and a defined failure mode for each agent step in multi-step pipelines. An eval plan is not a QA checkbox. It is the thing that tells you whether the system is actually working in production six months after launch.

4. Lock-in and Exit Cost

I quantify lock-in across three dimensions. First, model lock-in: are you fine-tuning on a proprietary model whose weights you cannot export? Second, data lock-in: is your retrieval index, embedding schema, or conversation history stored in a vendor-proprietary format? Third, operational lock-in: does your team have the skills to run this without the vendor on retainer? I ask for the data export format, the migration path, and the expected re-implementation cost if you switch providers in 18 months. If the vendor has not thought about this, that is itself a signal.

5. Security and Compliance Surface

AI systems introduce attack surfaces that standard security reviews miss: prompt injection via user input or retrieved documents, training data extraction, model inversion, and indirect tool-calling exploits if the system uses MCP or function-calling. I check whether the proposal addresses input sanitization, output filtering, tool permission scoping, audit logging for every LLM call, and PII handling in the retrieval layer. Most proposals do not mention any of these. That is not the vendor being negligent. It is the field moving faster than procurement checklists.

What Teams Get Wrong Before They Call for a Review

The most common mistake is waiting until after the contract is signed. By then, the architecture is locked, the vendor relationship is established, and pushback reads as obstruction rather than diligence. The right moment for an independent review is before the statement of work is finalized, when your leverage is highest.

The second most common mistake is framing the review as a vendor audit rather than a strategy audit. The question is not 'is this vendor good?' The question is 'is this the right approach for this problem, with this data, at this cost, for this team?' A good vendor executing the wrong approach is still the wrong outcome.

Third: teams underweight the cost of the eval gap. I have seen systems that demoed beautifully, hit production, and produced answers that were factually wrong 20 percent of the time, with no mechanism to detect or measure that. No one had defined what 'working' meant before the build started. The eval gap is where AI projects go to fail quietly.

A short internal checklist before you bring in a reviewer:

• Do you have a written definition of success with a measurable threshold?
• Do you know who owns the ground-truth dataset and how it stays current?
• Do you know the monthly inference cost at your expected query volume?
• Do you know what happens when the model returns a wrong answer?
• Do you know how you would migrate if the vendor raises prices 3x in year 2?

If you cannot answer all five, you are not ready to sign. You are ready for a review.

A Worked Example: RAG Over Internal Knowledge Base

A mid-size SaaS company came to me with a vendor proposal to build a RAG system over their internal support documentation. The proposal was 180k euros, 14 weeks, and used a proprietary vector database with a custom embedding pipeline tied to the vendor's hosted infrastructure.

The independent review found five issues:

1. The data was not ready. The documentation lived in four systems (Confluence, Notion, a legacy CMS, and a shared drive) with no consistent update process. Real data prep estimate: 8 weeks, not the 1.5 the proposal assumed.
2. The retrieval metric was undefined. The proposal measured success by 'user satisfaction in UAT.' That is not a metric. I proposed RAGAS faithfulness and answer relevancy scores against a 200-question ground-truth set drawn from real support tickets.
3. The embedding model was proprietary. Switching providers would require re-embedding the entire corpus. At their document volume, that was a 3,000-euro re-indexing cost plus 4 weeks of work per migration event.
4. No prompt injection defense. Support agents would paste customer emails directly into the query interface. The proposal had no input sanitization layer and no output confidence gating.
5. A lighter alternative existed. OpenAI's file search API with their existing document set would have covered 70 percent of the use case at roughly 12k euros to implement and 400 euros per month to run. The team decided to start there and revisit the custom build if they hit the ceiling.

The review cost them 3,000 euros and two weeks. It saved them from a 180k commitment that would have required another 40k in remediation before it could go to production safely.

How to Structure the Review Engagement

A second opinion engagement is not an open-ended consulting retainer. It has a defined scope, a defined deliverable, and a defined timeline. Here is the structure I use:

Total: 6 to 8 working days. The output is a written report you can share with your board, your procurement team, or your vendor as a negotiation document. It is not a slide deck full of frameworks. It is a decision document.

When You Need More Than a Review

A second opinion is the right tool when you have a concrete plan in front of you and need an independent judgment on it. It is not the right tool when you do not yet have a plan and need help building one. Those are different engagements.

If you are at the 'we know AI is important but we do not know where to start' stage, what you need is a strategy engagement, not a review. That involves mapping your highest-leverage use cases, ranking them by feasibility and ROI, and building a phased roadmap with resource requirements. I cover that in more depth in the AI consultancy services page.

If you are past review and into build, the ongoing questions shift to: who owns the eval harness, who monitors production drift, who handles model version upgrades, and how does the system degrade gracefully when the LLM returns low-confidence output. Those are architecture and engineering questions that a one-time review does not answer. But a review is always the right first step when real money is on the table.

What the Review Costs and What It Returns

An independent AI strategy review with the scope above typically runs between 2,500 and 6,000 euros depending on proposal complexity and the number of systems involved. That range covers a single vendor proposal review at the low end and a multi-vendor, multi-use-case strategy review at the high end.

The ROI math is straightforward. The median AI implementation project I have reviewed has had at least one critical finding that, left unaddressed, would have cost more than 30,000 euros to remediate post-launch or would have required a scope reduction that invalidated the business case. The review does not guarantee a good outcome. It guarantees that the decision-maker has the information they need before committing.

A note on timing: a review done before contract signing is worth 10x a review done after. After signing, the findings become a remediation list. Before signing, they become a negotiation instrument or a reason to walk away. Both outcomes are valuable. Only one is cheap.

Frequently Asked Questions

How do I get a second opinion on an AI vendor proposal?

Hire an independent reviewer with no vendor relationships before you sign the statement of work. Give them the full proposal, your data inventory, and access to your technical lead and business sponsor. A structured review takes 6 to 8 working days and produces a written go / no-go recommendation with specific findings. The key word is independent: a competing vendor, a large consulting firm with vendor partnerships, or an internal champion of the original proposal cannot play this role credibly.

What should an AI strategy review actually include?

At minimum: a feasibility assessment against the real business problem, a realistic data work estimate from someone who has seen your actual data, a written eval plan with measurable thresholds, a lock-in and exit cost analysis, and a security surface review covering prompt injection, PII handling, and audit logging. If the review does not produce a written report with specific risk ratings, it is not a review. It is a conversation.

Can I ask my current vendor to review their own proposal?

You can ask, but the structural conflict makes the answer unreliable on exactly the dimensions that matter most: scope inflation, lock-in risk, and the existence of cheaper alternatives. A vendor reviewing their own proposal is like asking a contractor to inspect the house they just built. They may be honest. The incentives do not support it.

How much does an independent AI review cost?

For a single vendor proposal review, expect 2,500 to 6,000 euros from a senior independent practitioner. Larger firms charge more and add overhead that does not improve the quality of the technical judgment. The number to compare it against is the contract value you are reviewing, not a line-item consulting budget. A 3,000-euro review on a 150,000-euro commitment is 2 percent insurance on a decision with multi-year operational consequences.

What is the difference between an AI audit and an AI strategy review?

An audit is retrospective: it reviews a system already in production for performance, security, and compliance. A strategy review is prospective: it reviews a plan before it is built. Both are valuable. If you are pre-build with budget in hand and a vendor proposal on the table, you need a strategy review. If you have a running system and you are not sure it is working the way you think it is, you need an audit.

How do I know if my AI plan is realistic?

Four signals that a plan is not realistic: the timeline allocates less than 40 percent of total effort to data work, there is no written eval plan with specific metrics and thresholds, the proposal does not mention failure modes or graceful degradation, and success is defined by demo quality rather than production performance. If any three of those are true, the plan needs independent review before you proceed.

Ready to Review Your Plan Before You Sign?

If you have a vendor proposal, an internal AI roadmap, or a platform recommendation in front of you and you want an independent judgment before you commit the budget, I can help. I work directly with the technical lead and the decision-maker, I have no vendor relationships, and I deliver a written report with specific findings, risk ratings, and a clear recommendation.

The review covers feasibility, hidden data work, eval plan, lock-in, exit cost, and security surface. It takes 6 to 8 working days. You leave with a document you can act on.

Start on the AI consultancy page to see the full scope of what I cover, or go directly to the contact page to send me the proposal and the timeline. I will tell you within one business day whether the engagement is a fit.