Is Your Data Ready for AI?
Your data is ready for AI when it is accessible, labeled well enough to answer a specific question, and representative of the conditions your model will face in production. Most companies fail at one or more of those three things, and the failure usually lives in the last two years of operational data sitting in disconnected systems, inconsistently formatted, never labeled for machine learning.
I am Mahmoud Zalt, an independent AI systems architect with 16 years building production software since 2010. Through Sista AI, the company I founded, I have spent the past year keeping a fleet of autonomous agents running against live production data. I work directly with engineering and product teams as an AI strategy and architecture consultant to run exactly this kind of readiness assessment before any model gets selected or budget gets committed. What I describe below is the framework I use on real engagements.
Why Data Readiness Beats Model Selection Every Time
The most common pattern I see: a team gets excited about a capability, picks a foundation model or a vendor platform, then discovers three months in that their data cannot support the use case. The model was never the bottleneck.
Foundation models are commodities. GPT-4o, Claude, Gemini, Llama 3, Mistral: they are all capable enough for most enterprise tasks. What differentiates your AI product is the proprietary signal in your own operational data. That data is your moat, but only if you can get it into a shape the model can use reliably.
The honest reframe: 'we need AI' almost always means 'we need our last two years of messy operational data accessible and labeled'. Fix that, and model selection becomes a commodity decision. Skip that, and no model saves you.
The Three Blockers I See Repeatedly
- Inaccessibility: data lives in SaaS tools with no export API, in PDFs on shared drives, in a data warehouse that no one on the ML team has credentials for.
- Missing labels: you have events (orders, tickets, clicks) but no outcome labels (resolved, churned, high-value) attached to those events in a joinable way.
- Distribution mismatch: you have data, but it covers only your happy path. Edge cases, recent product changes, and seasonal shifts are underrepresented or absent.
The 5-Dimension Data Readiness Scorecard
Score each dimension 1 to 3. A total of 12 or above means you can likely start a focused pilot. Below 10, fix data before writing a single line of model code.
| Dimension | Score 1 (not ready) | Score 2 (partial) | Score 3 (ready) |
|---|---|---|---|
| Accessibility | Data locked in SaaS UIs, PDFs, or systems with no programmatic access | Accessible but requires manual exports or one-off scripts per request | Queryable via API or data warehouse; ML team can self-serve |
| Labeling | Raw events only, no outcome labels or ground truth | Some labels exist but inconsistently defined or partially joined | Clear outcome labels, consistently applied, joinable to features |
| Volume and recency | Under 1,000 labeled examples or data older than 18 months dominates | 1,000 to 10,000 examples, mixed recency | 10,000+ labeled examples covering the last 12 to 24 months |
| Quality and consistency | High null rates, inconsistent schema across time, no validation | Schema documented but drift visible; occasional nulls in key fields | Validated pipelines, known null rates below 5% on key features, schema versioned |
| Representativeness | Only happy-path or one customer segment; edge cases absent | Reasonable coverage but known gaps in edge cases or recent changes | Covers realistic production distribution including edge cases and recent behavior |
Run this scorecard in a 90-minute working session with your data engineering lead, your domain expert (who knows what the labels mean), and one engineer who has tried to use the data before. The conversation itself surfaces blockers faster than any audit tool.
Worked Example: A Support Ticket Classifier
A B2B SaaS company wants to auto-route support tickets to the correct team using an LLM classifier. Here is what the readiness check actually found.
What They Thought They Had
Five years of Zendesk tickets, fully exported. 200,000 tickets. Engineers were confident.
What the Scorecard Found
- Accessibility: 3. Zendesk bulk export worked. Good.
- Labeling: 1. The 'group' field (the label) had been reorganized twice in three years. Tickets before 2022 used a taxonomy that no longer matched current teams. Nobody had remapped them. Effective labeled set: 18,000 tickets from the last 14 months.
- Volume and recency: 2. 18,000 is workable but thin for 12 output classes. Some classes had under 300 examples.
- Quality: 2. Subject lines were frequently copy-pasted error codes. Body text was present but 20% of tickets had been submitted via a broken mobile form that stripped formatting.
- Representativeness: 1. A major product launch 8 months ago created a new ticket category that was ad-hoc labeled by one support manager with no written criteria.
Total: 9 out of 15. Not ready for production. The recommendation was a 6-week data remediation sprint before any model work: remap old labels, write a labeling guide for the new category, get a second human to relabel 500 examples for inter-annotator agreement, and flag the broken mobile form tickets for exclusion. After remediation, the same dataset scored 13 and the pilot launched on schedule.
The Labeling Problem Is a Business Problem, Not a Technical One
Teams reach for automated labeling tools immediately. Sometimes that is right. More often, the real issue is that no one in the business has written down what the label actually means.
For a churn prediction model, 'churned' sounds obvious until you ask: does a customer who downgraded count? What about a customer who stopped paying but is in a payment dispute? What about a customer acquired through a reseller whose contract renews differently? These are not edge cases; they are 30 to 40 percent of your rows once you look.
The Label Definition Protocol I Use
- Write a one-paragraph definition of the positive class in plain language, with three concrete examples of what qualifies and two that look similar but do not.
- Have two domain experts independently label 200 examples using only that definition.
- Compute inter-annotator agreement (Cohen's kappa). Target above 0.7. Below 0.6, the definition is ambiguous and the model will learn noise.
- Resolve disagreements by editing the definition, not by majority vote. The goal is a definition that produces consistent labels, not a label set produced by committee.
- Apply the final definition to your full historical dataset, documenting assumptions.
This step takes one to two weeks on a real engagement. It feels slow. It prevents months of wasted model training cycles.
A Special Case: Retrieval and RAG Readiness
If the use case involves retrieval-augmented generation (RAG), the readiness criteria shift. You are less dependent on labeled training data and more dependent on document quality and chunking strategy.
The RAG readiness check I run:
- Document freshness: are the source documents current? A knowledge base last updated 18 months ago will produce confidently wrong answers.
- Structure: are documents well-structured with clear headings and self-contained sections? Wall-of-text PDFs and poorly formatted HTML produce retrieval misses even with good embedding models.
- Deduplication: duplicated or near-duplicate content causes the retriever to over-retrieve a single source and miss coverage elsewhere. Run a similarity pass before indexing.
- Coverage: does the corpus actually contain answers to the questions users will ask? Run 50 representative queries against the raw corpus before building any pipeline. If the answer isn't in the documents, retrieval won't find it.
- Access control: if documents have different permission levels, your retrieval pipeline must respect those at query time. This is an architecture requirement, not an afterthought.
The single most underestimated RAG failure mode is a corpus that is too broad. More documents do not help when the retriever cannot distinguish a relevant chunk from an adjacent but misleading one. Narrow, high-quality corpora beat large noisy ones consistently.
The Honest Cost of Getting Your Data Ready
I tell clients this upfront: data readiness work is usually 30 to 60 percent of the total cost of an AI project. Teams budget it at 10 percent and are surprised when model work starts late. Here is a realistic breakdown for a mid-size company running a first serious AI initiative.
| Activity | Typical duration | Who does it |
|---|---|---|
| Data audit and scorecard | 1 to 2 weeks | AI architect + data lead |
| Label definition and inter-annotator agreement | 1 to 3 weeks | Domain experts + data lead |
| Historical data remediation (schema, joins, nulls) | 2 to 6 weeks | Data engineering |
| Human labeling of new examples | 2 to 8 weeks depending on volume | Labelers (internal or vendor) + QA |
| Retrieval corpus preparation (if RAG) | 1 to 4 weeks | Data engineering + domain review |
| Eval dataset creation (held-out, adversarial) | 1 to 2 weeks | ML engineer + domain expert |
A company with genuinely well-governed data can compress this significantly. But most companies I work with have not governed their data well, because there was no reason to until now. The cost is real and the only way to reduce it is to start earlier, not to skip it.
One cost-saving lever that works: do not label everything. Label the data for the specific question the model must answer, for the specific time window it will operate in, for the specific input distribution it will see in production. Scope cuts labeling cost by 50 to 80 percent on most projects I have run.
What Teams Get Wrong (and What to Do Instead)
Mistake 1: Starting with the model, not the question
Teams evaluate LLMs and embedding models before defining the exact task the model will perform. The task definition determines the data requirements. Define the task first in one sentence: 'Given a support ticket body (text), predict which of 12 routing teams should handle it, measured by agreement with senior support staff.' Then assess what data that task requires.
Mistake 2: Treating all historical data as useful
Old data reflects old conditions. Product behavior changes. User expectations change. Team structures change. For most tasks, data older than 24 months actively hurts model performance unless you have verified that conditions were stable. Default to the last 12 to 18 months; expand only if volume is insufficient and you can confirm stability.
Mistake 3: No eval set before model selection
You cannot compare models without a held-out eval set that reflects production conditions. Build the eval set before you run a single model comparison. It takes two weeks, costs little, and prevents expensive vendor decisions made on the wrong evidence.
Mistake 4: Confusing data volume with data quality
One million poorly labeled rows is worse than 10,000 carefully labeled rows for most fine-tuning and few-shot tasks. I have seen teams celebrate having 'enough data' while running models trained on labels that three domain experts would disagree on 40 percent of the time. Measure inter-annotator agreement. It is the single number most predictive of whether your model will work.
Mistake 5: Skipping observability planning
Data readiness includes planning how you will detect when production data drifts away from your training distribution. Build data drift monitoring into the readiness plan, not as an afterthought post-launch. The tools are straightforward (distribution shift tests on key feature statistics); the failure to plan for it is an organizational habit, not a technical constraint.
Frequently Asked Questions
How much data do we need to build an AI model for our company?
For a fine-tuned classifier or ranker, 5,000 to 50,000 labeled examples covering your production distribution is a workable starting range. For RAG, volume matters less than document quality and corpus coverage. For prompt-based workflows using foundation models with no fine-tuning, the answer is zero training examples, but you still need a high-quality eval set of 200 to 500 examples to validate and iterate. The number you need is always a function of task complexity and the number of output classes, not a universal threshold.
Can we use synthetic data to fill gaps in our training set?
Yes, with discipline. Synthetic data works well for expanding coverage of rare classes, for augmenting text with paraphrase variation, and for stress-testing edge cases. It does not replace real production data for capturing the actual distribution your model will face. The failure mode is training on synthetic data that is too clean or too consistent, which causes the model to underperform on the messy real-world inputs it will actually receive. Use synthetic data as a supplement, not a replacement.
What is the fastest way to assess if our data is ready for AI?
Run the 5-dimension scorecard above in a 90-minute working session with three people: your data engineering lead, a domain expert who understands what the labels mean, and one engineer who has previously tried to use the data for something. The conversation surfaces the real blockers faster than any automated tool. Follow up by sampling 100 rows and having two domain experts independently label them; compute agreement. That two-hour exercise tells you more than weeks of automated profiling.
Do we need a data warehouse before we can do AI?
No, but you need some programmatic path to the data. A data warehouse is the right long-term foundation, but for an initial pilot, a well-structured set of CSV exports, a readable Postgres database, or clean API access to your SaaS tools is sufficient. The blocker is not the warehouse; it is whether your ML team can access and query the data without filing tickets and waiting days for manual exports.
How do we know if our AI project failed because of bad data versus a bad model?
Compute your model's performance on a clean, carefully labeled eval set versus its performance on randomly sampled production data. A large gap (more than 10 percentage points on your key metric) almost always indicates a training-to-production distribution mismatch, which is a data problem. Uniform poor performance across both sets usually indicates a task definition problem or insufficient training volume. Isolating the failure mode is the first step; treating 'bad model' as the default diagnosis is almost always wrong.
Should we clean all our data before starting an AI project?
No. Cleaning all your data before starting is a way to never start. Clean the data for the specific task, for the specific time window, for the specific input distribution the model will see. That scoped remediation is 10 to 20 percent the effort of a full data cleanup and delivers 80 to 90 percent of the benefit for your pilot. Expand scope after the pilot validates the use case and the business case for deeper investment is confirmed.
Ready to Find Out Where You Actually Stand?
The readiness work described here is not glamorous, but it is the difference between an AI project that works in production and one that runs for six months and gets quietly cancelled. Most of the teams I work with arrive with optimism about their data and leave the first session with a clearer, more honest picture and a concrete remediation plan they can actually execute.
If you want to run this scorecard against your actual systems, understand what it will cost to get ready, and make a defensible decision about where to start, that is exactly what I do as an independent AI consultant. No vendor agenda, no platform to sell, just a concrete answer to whether your data can support what you are trying to build.
Read more about my background on the about page, see past projects at /projects, or get in touch directly at /contact.
Book a data readiness assessment and get a scorecard for your actual systems.
