Answer

How do you build AI that doesn't need the Cloud?

Most people think using AI means sending data to cloud providers.

But what if your client is a bank, government agency, or healthcare organization where data can never leave their private network?

The solution is on-premises AI.

Instead of calling external APIs, you deploy an open-source LLM (such as Llama, Qwen, Mistral, or Gemma) directly on the client's GPU servers. The entire AI stack runs within their infrastructure—no internet access required.

Typical Architecture

Users
  │
  ▼
AI Application
  │
  ├── Authentication
  ├── AI Agent
  ├── RAG Pipeline
  ├── Vector Database
  ├── Document Store
  └── Local LLM Server

When a user asks a question:

"What is our KYC policy for high-risk customers?"

The system: 1. Retrieves relevant documents from the local vector database. 2. Sends only the relevant context to the local LLM. 3. Generates the response—all without any internet connectivity.

Why enterprises choose this approach

• 🔒 Data never leaves the organization
• 🛡️ Easier regulatory compliance
• 📋 Complete auditability
• 🚫 No dependency on external AI providers
• ⚡ Full control over models and infrastructure

Building AI is no longer just about prompting an API. Increasingly, it's about designing secure, scalable AI platforms that can run entirely inside an organization's own data center.

The future of enterprise AI isn't necessarily cloud-first—it can be completely cloud-independent.

Answer

Unnecessary tool calls usually mean the agent lacks clear stopping criteria, tool selection guidance, or feedback on wasted actions.

Step 1: Instrument and classify waste

Log every turn: user goal, tools considered, tools called, args, result size, and final answer. Tag calls as:

• Redundant — same tool + same args repeated
• Premature — called before gathering enough context from conversation
• Over-broad — fetched far more data than needed
• Wrong tool — task solvable without tools or with a cheaper one

Fix the highest-frequency category first.

Step 2: Tighten the decision policy in prompts

• Add explicit rules: "Call a tool only when you lack information required to answer"
• Require a brief plan before first tool call ("I need X; tool Y provides it")
• List when not to call each tool (e.g., don't search docs for greetings)
• Cap tools per turn and per session

Step 3: Improve tool design

Bad tools invite overuse:

• Return summaries, not raw dumps
• Narrow APIs (search by ID vs full-table scan)
• Idempotent, fast tools reduce the model's urge to "double-check"
• Tool descriptions must state inputs, outputs, and failure modes — vague descriptions cause exploratory calls

Step 4: Add a routing layer

Use a lightweight intent classifier or rules engine upstream:

• FAQ / chit-chat → no tools
• Known workflows → fixed tool sequence (deterministic graph)
• Open-ended research → full agent

Not every message needs the agent loop.

Step 5: Reward correct restraint in eval

Build test cases where the correct action is no tool call. Penalize extra calls in offline eval. Models optimize what you measure.

Step 6: Reflect on failures

After N tool calls without progress, inject a system nudge: "Summarize what you know; call at most one more tool or answer with uncertainty."

Interview line: "Measure which calls are wasted, tighten prompts and tool APIs, route simple intents out of the agent loop, and eval for restraint — not just task success."

Answer

How do you prevent duplicate retrievals in a RAG system?

A common problem in RAG systems is this:

A query retrieves 10 chunks, but 5 of them contain nearly identical information. As a result, another relevant but unique chunk gets pushed out of the LLM's context window.

More chunks don't always mean better answers.

A better retrieval pipeline

Instead of sending the top-k results directly to the LLM, add a diversification layer:

Query
  │
  ▼
Vector Search (Top 50)
  │
  ▼
Deduplicate Similar Chunks
  │
  ▼
Diversity Re-ranking
  │
  ▼
Top 10 Context
  │
  ▼
LLM

Techniques that work

• Maximal Marginal Relevance (MMR): Balances relevance with diversity, reducing redundant results: Don't rank by relevance alone. MMR balances relevance vs. novelty — each new chunk must be relevant to the query AND different from chunks already selected.
• Semantic deduplication: Remove chunks whose embeddings are highly similar above a threshold.
• Document-aware retrieval: Limit the number of chunks returned from the same document or section.
• Hybrid retrieval: Combine keyword (BM25) and vector search to surface unique but relevant information.
• Context packing: Merge adjacent or overlapping chunks before sending them to the LLM.

The outcome

Instead of:

• Chunk A
• Chunk A (duplicate)
• Chunk A (duplicate)
• Chunk A (duplicate)
• Chunk A (duplicate)

You get:

• Chunk A
• Related policy
• Exception case
• Supporting procedure
• Reference document

The LLM receives broader, richer context, leading to more complete and accurate answers.

In RAG, retrieval quality isn't just about relevance—it's about relevance and diversity.

Answer

Start with one agent unless separation buys clear wins in reliability, security, or maintainability.

Prefer a single agent when:

• Workflow is linear with a small, stable tool set
• Steps share heavy context (one thread is cheaper than handoffs)
• Team is early-stage — operational complexity of multi-agent hurts more than model confusion
• Latency budget is tight — each agent hop adds another LLM round

Prefer multiple specialized agents when:

• Domains differ sharply — e.g., legal review vs code execution vs customer tone — each needs different prompts, models, or safety rules
• Least privilege matters — the research agent must not access write tools; the executor agent has narrow permissions
• Parallelism helps — researchers gather sources while a planner drafts structure
• Failure isolation — one flaky sub-skill (browser automation) shouldn't poison the whole run
• Different models per role — small fast classifier + large reasoner + cheap summarizer

Decision framework

Orchestration patterns

• Supervisor routes to specialists (good for clear intents)
• Pipeline — extract → validate → act (good for compliance)
• Peer debate — only when quality gain justifies 3× cost (rare in production)

Anti-pattern

Multiple agents that duplicate the same system prompt and tools — you added latency and debugging surface without separation of concerns.

Interview close: "One agent until context overlap breaks down or security/parallelism force a split; specialize by permission boundary and failure domain, not by job title alone."

Answer

Agents loop when they lack hard stop conditions and progress detection. Combine budget limits with semantic checks.

Hard budgets (always enforce server-side)

• Max steps / tool calls per session (e.g., 10)
• Max wall-clock time and token spend per request
• Max identical actions — same tool + same args twice → block and escalate
• Max LLM turns in the ReAct loop regardless of model intent

Never rely on the model to stop itself.

Progress detection

Track a state fingerprint each turn: tools used, args hash, new facts extracted. If two consecutive turns produce no new information, break the loop and:

• Return partial answer with uncertainty, or
• Ask the user one clarifying question, or
• Hand off to human

Loop-specific patterns

Architecture

• Model plan once, executor runs deterministic steps where possible
• Human approval gate before irreversible or expensive loops (payments, mass emails)
• Checkpoint and resume — persist state so loops can be killed without orphan side effects

Observability

Alert on sessions hitting 80% of step budget. Post-mortem loops: was it bad tools, ambiguous goal, or missing stop instruction?

Eval

Include adversarial loop traps in tests: contradictory docs, tools that always return empty, tasks with no solution. Pass = graceful exit within budget.

Interview summary: Server-enforced budgets, duplicate-action detection, progress fingerprints, and graceful degradation — not "please don't loop" in the prompt alone.

Answer

A shared AI platform is guardrails + self-service APIs + centralized policy — teams ship apps, not bespoke LLM plumbing.

Core platform services

• LLM gateway — unified auth, routing, rate limits, logging
• RAG-as-a-service — ingest, chunk, embed, retrieve with tenant isolation
• Eval harness — golden sets, regression gates, human review UI
• Secrets and key management — no team-owned production API keys in repos
• Observability — traces with model version, cost, latency per app

Safety and governance

• Central content safety (input/output moderation)
• Approved model allowlist per data classification level
• PII redaction in logs; retention policies
• Mandatory risk tier per app (read-only Q&A vs write tools vs customer-facing)

Self-service with guardrails

• Teams register apps with: owner, data scope, models allowed, budget cap
• Sandbox environment with synthetic data
• CI template: eval must pass threshold before prod promotion
• Shared prompt patterns and tool SDK — reduce one-off insecure agents

Multi-tenancy

• Namespace isolation: indexes, configs, quotas per team/tenant
• No cross-tenant retrieval — enforce in API layer, not prompts

Platform team KPIs

• Time-to-first-production-app
• Incidents caused by platform gaps
• % traffic on blessed paths vs rogue direct API calls (drive to zero)

Interview close: Gateway, shared RAG, eval gates, tiered risk policies, and tenant isolation — teams innovate on workflows, platform owns models, safety, and spend.

Answer

Support chatbot eval must mirror real tickets: intent, policy, tone, resolution, and escalation — not generic trivia accuracy.

Build the dataset

• Sample historical tickets (scrub PII): user message, ideal resolution, policy refs
• Label categories: billing, technical, account, angry user, out-of-scope
• Include adversarial cases: prompt injection, requests for refunds outside policy, empty messages
• Add unanswerable cases — correct behavior is escalate or refuse

Offline metrics per category

Pipeline stages

1. Unit: retrieval recalls correct policy doc (Recall@k)
2. End-to-end: full bot answer vs gold on held-out set
3. Regression gate in CI: block deploy if any critical category drops >2%
4. Canary: 5% prod traffic, compare escalation and CSAT vs control
5. Weekly human review of 100 live chats

Tooling

• Version everything: prompt_v, index_v, model_v on each eval run
• Dashboard: pass rate by category, failure clusters, citation errors
• Export failures to labeling queue for dataset growth

Tie to business outcomes

Track containment rate (no human needed), CSAT, average handle time, and repeat contact rate — offline eval must correlate with these over time.

Interview summary: Gold set from real tickets, category-stratified metrics, CI regression gate, canary in prod, weekly human review — optimize resolution and policy compliance, not fluency alone.

Answer

Banking assistants require defense in depth: identity, data classification, auditability, and human gates on anything that moves money or changes accounts.

High-level flow

Employee/Customer → WAF → API Gateway (mTLS, OAuth)
    → Policy engine (role, data class, allowed intents)
    → Orchestrator
        → Retrieval (segmented indexes: public FAQ vs internal policy vs customer-specific)
        → LLM (private endpoint or approved cloud with BAA/DPA)
        → Output guardrails (PII, regulatory language, citation check)
    → Action executor (only for approved tools, server-side validation)
    → Audit log (immutable, SIEM-integrated)

Key components

Identity & authorization - SSO with role-based access — tellers vs advisors vs ops see different knowledge and tools - Customer-facing bot scoped to their accounts only via token-bound queries

Knowledge layers - Public: product FAQs, branch hours - Internal: compliance procedures, underwriting guides - Never mix customer PII into shared embedding indexes

Model tier - Often VPC-hosted or dedicated cloud tenancy - No training on customer data; zero-retention inference contracts

Tools (highly restricted) - Read-only: balance lookup, transaction search (parameterized, authZ checked server-side) - Write actions: human approval or not exposed to LLM at all

Compliance - Every response logged with inputs, retrieved doc IDs, model version - Explainability: citations to internal policy docs - Model risk management documentation (SR 11-7 style governance)

Human oversight - Low-confidence or high-risk intents → live agent queue with full context packet

DR and resilience - Multi-region failover, offline FAQ mode, circuit breakers on model provider

Interview close: RBAC at the gateway, segmented indexes, private inference, citation-backed answers, no autonomous money movement, immutable audit — compliance is architecture, not a disclaimer.