Traditional software can be “shipped and maintained” with periodic updates. AI products require continuous operational investment because they degrade without it:

• Data staleness: Knowledge bases become outdated. Training data stops reflecting current reality.
• Model drift: Provider model updates change behavior. User patterns evolve. The world changes.
• Cost creep: Usage patterns shift. Token consumption grows. New features add cost layers.
• Quality erosion: Edge cases accumulate. New failure modes emerge. User expectations increase.

An AI product that isn’t actively operated is an AI product that’s actively degrading. The operational investment is not a cost center — it’s the cost of maintaining the product’s value.

Plan for ongoing operations from the start. A common mistake is budgeting for development but not for operations:

Typical allocation (post-launch):
• 40% — Improvement: Fixing failure modes, expanding capabilities, improving quality
• 25% — Data operations: Knowledge base updates, data pipeline maintenance, quality audits
• 20% — Infrastructure: Monitoring, cost optimization, scaling, reliability
• 15% — Incident response: On-call, bug fixes, safety issues, provider outages

This means roughly 60% of post-launch effort goes to operations, not new features. Teams that don’t plan for this end up choosing between quality degradation and feature velocity.

Ingestion:
New documents, updated policies, fresh content must flow into the knowledge base continuously. Automated pipelines detect changes in source systems (CMS, wikis, databases) and trigger re-indexing.

Processing:
Raw documents are cleaned, chunked, embedded, and stored. Schema validation ensures data quality. Duplicate detection prevents contradictory entries.

Deprecation:
Outdated documents must be removed or marked as superseded. A stale document that contradicts current policy is worse than no document at all.

Monitoring:
Track pipeline health: ingestion latency (how quickly new content becomes searchable), processing errors, index freshness, and coverage gaps.

Continuous (automated):
Incremental ingestion of new/updated documents. Schema validation. Duplicate detection. Pipeline health monitoring.

Weekly (PM + data team):
Review coverage gaps (queries with no relevant documents). Check for stale content. Review data quality metrics. Prioritize content creation for uncovered topics.

Monthly (broader team):
Full knowledge base audit. Verify accuracy of high-traffic content. Update taxonomy and metadata. Review data source health.

Quarterly (strategic):
Evaluate new data sources. Assess whether the knowledge base scope matches product scope. Plan major content initiatives.

When you use third-party models (OpenAI, Anthropic, Google), the provider periodically updates the model. These updates can change behavior in ways that affect your product:

• Prompts that worked perfectly may produce different outputs
• Safety boundaries may shift (more or less restrictive)
• Performance characteristics (latency, cost) may change
• New capabilities may become available

The response protocol:
1. Pin to a specific model version when possible
2. When a new version is released, run your full evaluation suite against it
3. Compare quality, cost, and latency to the current version
4. If the new version is better or equivalent, migrate through the standard canary process
5. If it’s worse, stay on the current version and file issues with the provider

The weekly improvement sprint:

1. Review failures (Monday): Analyze the top 20 worst-performing queries from the past week. Categorize by root cause.

2. Prioritize fixes (Monday): Rank by frequency × severity. Pick the top 3–5 for this sprint.

3. Implement (Tue–Thu): Prompt adjustments, knowledge base updates, retrieval tuning, or guardrail additions.

4. Evaluate (Thursday): Run changes through the eval suite. Verify improvements without regressions.

5. Deploy (Friday): Canary deployment of improvements. Monitor over the weekend.

This cadence produces measurable quality improvement every week. Over 3–6 months, the cumulative effect is dramatic — the product gets noticeably better every month.

1. Model routing:
Not every query needs the most expensive model. Route simple queries (FAQs, greetings, basic lookups) to a smaller, cheaper model (GPT-4o-mini, Claude Haiku). Reserve the premium model for complex queries. This alone can reduce costs 30–40%.

2. Semantic caching:
If the same question (or a semantically similar one) was asked recently, serve the cached response instead of calling the model again. Effective for products with repetitive query patterns (customer support, FAQ bots).

3. Prompt optimization:
Shorter prompts cost less. Remove redundant instructions. Use concise language. Compress few-shot examples. A 30% reduction in prompt length = 30% reduction in input token cost.

4. Batch processing:
For non-real-time workloads (document analysis, report generation), batch requests for lower per-token pricing. Many providers offer 50% discounts for batch API calls.

5. Context window management:
Don’t stuff the full context window. Retrieve fewer, more relevant chunks. Trim conversation history to the most recent N turns. Every unnecessary token costs money.

6. Output length control:
Set maximum output lengths appropriate to the task. A billing question doesn’t need a 500-word essay. Shorter outputs = lower output token costs (which are typically 3–4x more expensive than input tokens).

7. Provider negotiation:
At scale (>$10K/month), negotiate volume discounts with providers. Consider committed-use pricing. Evaluate open-source alternatives for high-volume, lower-complexity tasks.

Weekly: Review cost per query trends. Identify anomalies.
Monthly: Full cost analysis by feature, model, and user segment. Identify top optimization opportunities.
Quarterly: Evaluate model pricing changes. Reassess build vs. buy. Consider model migration for cost savings.

Quality incidents:
The AI starts producing wrong, harmful, or low-quality outputs. May be caused by model updates, data issues, prompt regressions, or adversarial attacks.

Provider outages:
The model provider (OpenAI, Anthropic, Google) experiences downtime or degraded performance. Your product is down through no fault of your own.

Cost incidents:
Unexpected cost spikes from traffic surges, prompt bugs (infinite loops), or provider pricing changes.

Safety incidents:
The AI produces harmful, biased, or legally risky content that reaches users. The highest-severity incident type.

Data incidents:
Knowledge base corruption, stale data serving, or data pipeline failures causing incorrect or missing information.

1. Detect (automated):
Monitoring alerts trigger based on predefined thresholds. Target: <5 minutes from incident start to alert.

2. Triage (on-call):
Assess severity. Is this a P0 (safety, system down) or P2 (quality degradation)? Severity determines response speed and escalation.

3. Mitigate (immediate):
Stop the bleeding. Activate kill switch, roll back to previous version, enable fallback mode, or rate-limit traffic. Don’t try to fix the root cause yet — just stop the damage.

4. Investigate (hours):
Find the root cause. Use traces, logs, and metrics to identify what changed. Was it a model update? Data issue? Prompt regression?

5. Fix (hours–days):
Implement the permanent fix. Test through the standard evaluation and deployment process.

6. Retrospective (within 1 week):
What happened? Why? How do we prevent it? What monitoring would have caught it earlier? Add new regression tests and monitoring.

1K queries/day (early stage):
Manual review is feasible. One person can read a meaningful sample of outputs daily. Cost is negligible. Incidents are rare. Operations are lightweight.

10K queries/day (growth stage):
Manual review becomes sampling. Automated quality monitoring is essential. Cost becomes a line item. You need dedicated on-call. Data pipeline reliability matters.

100K queries/day (scale stage):
Full automation required. Sophisticated model routing for cost management. Multiple model providers for reliability. Dedicated operations team (or significant allocation). Cost optimization is a strategic priority.

1M+ queries/day (enterprise stage):
Custom infrastructure. Negotiated provider contracts. Multi-region deployment. Compliance and audit requirements. AI operations is a full team, not a part-time responsibility.

Early (manual is fine):
• Quality review (read outputs yourself)
• Knowledge base updates (manual edits)
• Cost tracking (check the dashboard)

Growth (automate the basics):
• Quality monitoring (automated sampling and scoring)
• Data pipeline (automated ingestion and indexing)
• Alerting (automated detection and notification)
• Regression testing (automated on every change)

Scale (automate everything):
• Model routing (automated based on query complexity)
• Cost optimization (automated caching, batching)
• Incident detection and initial mitigation (automated rollback)
• Capacity planning (automated scaling based on demand)

Enterprise (optimize and govern):
• Compliance reporting (automated audit trails)
• Multi-model orchestration (automated provider selection)
• Feedback-driven improvement (automated retraining triggers)

As AI products handle more sensitive decisions and data, governance becomes non-negotiable:

• Regulatory requirements: EU AI Act, GDPR, HIPAA, SOC 2, and industry-specific regulations increasingly require documentation of AI decision-making processes
• Audit trails: For regulated industries, every AI decision must be traceable: what model, what prompt, what data, what output, and why
• Access control: Who can modify prompts? Who can update the knowledge base? Who can deploy model changes? Unauthorized changes are a top risk
• Data handling: User inputs may contain PII. Model outputs may inadvertently include sensitive information. Data retention and deletion policies must be enforced

Model cards:
Document each model’s capabilities, limitations, training data characteristics, known biases, and intended use cases. Update when models change.

Change approval workflows:
Prompt changes, model updates, and data modifications require review and approval. For high-risk products, require sign-off from technical, product, and legal stakeholders.

Audit logging:
Log every model interaction with sufficient detail for retrospective analysis. Include: timestamp, user ID (anonymized), input, output, model version, prompt version, retrieval results.

Data retention policies:
Define how long interaction data is retained. Implement automated deletion. Ensure compliance with privacy regulations. Allow users to request data deletion.

Model decommissioning:
When retiring a model or AI feature, follow a structured process: notify users, migrate to alternatives, archive data, and document the decision.

Level 1: Reactive
No automated monitoring. Quality issues discovered by users. Manual data updates. No incident process. Ad-hoc cost tracking. Most teams start here.

Level 2: Monitored
Basic monitoring and alerting. Automated evaluation on changes. Weekly quality reviews. Defined incident response. Cost dashboards. Target: reach within 30 days of launch.

Level 3: Proactive
Drift detection catches issues before users notice. Automated improvement pipelines. Model routing for cost optimization. Structured governance. Continuous testing. Target: reach within 6 months.

Level 4: Optimized
Feedback-driven automated improvement. Multi-provider orchestration. Predictive scaling. Full compliance automation. AI operations is a competitive advantage, not just a cost center. Target: 12+ months for mature products.

Ask these questions:

• How quickly do you detect quality degradation? (Hours = L2, minutes = L3, predicted = L4)
• How do you handle model provider updates? (Manually = L1, evaluated = L2, auto-migrated = L3+)
• What’s your cost optimization strategy? (None = L1, dashboards = L2, automated routing = L3+)
• How do you improve quality? (Ad-hoc = L1, weekly sprints = L2, automated pipelines = L3+)
• What’s your governance posture? (None = L1, audit logs = L2, full compliance = L3+)