Question 1

How do you choose a model for a coding agent?

Accepted Answer

List the criteria: task fit, tool-calling reliability, context window, latency, cost per completed task, licence and data rules. Build a small eval set from real tasks, run candidates on it, then route: a cheap model for simple steps, a stronger one for hard steps.

Question 2

Open-weight or hosted API?

Accepted Answer

Hosted wins on speed to ship and top quality. Open-weight (Llama, Mistral, DeepSeek) wins when you need data control, fixed cost at volume, or fine-tuning, but you own serving, GPUs and upgrades. Many teams start hosted and move a stable, high-volume step to open-weight once evals prove parity.

Question 3

A provider deprecates your model next month. What happens?

Accepted Answer

Calls go through one gateway layer so the model is config, not code. Pin versions, keep an eval suite, run the replacement against it, shadow-test a slice of traffic, then switch with a rollback path.

Question 4

How do you keep AI-generated code trustworthy?

Accepted Answer

Small diffs, tests written or reviewed first, type-check and lint in CI, human review of every change, and a rules file that tells the assistant your conventions. Treat output as a junior contribution: verify, never assume.

Question 5

Compare an inline assistant, an IDE agent and a terminal agent.

Accepted Answer

Inline completion suits boilerplate while you type. An IDE agent suits multi-file edits you want to watch. A terminal agent suits larger tasks that run commands and tests. Pick by task size and how much oversight you need.

Question 6

Tell me about a time an AI assistant was wrong.

Accepted Answer

Use STAR. Say what it got wrong, how you caught it (test, review, reading the diff), the fix, and the guardrail you added afterwards.

Question 7

When would you use an agent instead of a fixed workflow?

Accepted Answer

If the steps are known, use a deterministic workflow with LLM calls inside. Use an agent only when the path depends on intermediate results and cannot be listed upfront. Start simple, add autonomy where an eval shows it helps.

Question 8

How would you choose between LangGraph, CrewAI and AutoGen?

Accepted Answer

Explicit state, branching, checkpoints and human approval point to a graph model like LangGraph. Role-based task delegation fits crew-style frameworks. Conversational multi-agent fits AutoGen-style setups. Decide on determinism, observability and team familiarity, and prototype both on one real task.

Question 9

How do you stop an agent looping or burning budget?

Accepted Answer

Max steps, per-run token and cost budgets, timeouts, tool allow-lists, loop detection on repeated calls, and human approval for destructive actions. Log every step so you can replay a failure.

Question 10

What is MCP and why does it matter?

Accepted Answer

An open protocol where a client (the AI app) talks to servers that expose tools, resources and prompts. It replaces one-off integrations per app and per model with one standard interface, so a tool is written once and reused.

Question 11

What are the security risks of connecting MCP servers?

Accepted Answer

Untrusted servers, over-broad permissions, and prompt injection through tool output. Mitigate with least privilege, read-only by default, authentication, confirmation before writes or deletes, auditing every call, and treating tool results as data, never as instructions.

Question 12

MCP server, plain REST API, or in-app function calling?

Accepted Answer

Function calling inside one app is simplest. MCP pays off when several clients or models need the same tools. A REST API stays right for non-AI consumers. Often the MCP server is a thin wrapper over an existing API.

Question 13

How would you pick a vector store?

Accepted Answer

Prototype with something light (Chroma or FAISS). Managed (Pinecone) when you want no ops. Weaviate or OpenSearch when you need keyword plus vector hybrid search or already run the stack. Milvus when you self-host at large scale. Note FAISS is a library: no persistence, auth or filtering out of the box.

Question 14

Your RAG answers are poor. What do you check?

Accepted Answer

Measure retrieval separately from generation. Build a golden question set and track hit@k and MRR. Then work through chunking, metadata filters, hybrid search, a reranker, and query rewriting, changing one thing at a time. Only then tune the prompt.

Question 15

When is a graph better than vectors?

Accepted Answer

When questions need multi-hop relationships across entities, which plain similarity search handles badly. Prove it on a golden set of multi-hop questions first; otherwise vectors plus a reranker are simpler and cheaper.

Question 16

Serverless functions or containers for an LLM backend?

Accepted Answer

Functions are cheap and scale to zero but have request time limits and cold starts, which hurts long generations and large model loads. Containers (Cloud Run or Kubernetes) allow streaming, longer jobs and warm models. Use queues and workers for long-running agent tasks.

Question 17

How do you run long agent tasks reliably?

Accepted Answer

Put the task on a queue, run it in a worker, make steps idempotent, persist state between steps, retry with backoff, and report status to the user by polling or events.

Question 18

How do you test a non-deterministic system?

Accepted Answer

A golden set of representative cases, including ones that must be refused. Deterministic checks where possible (format, citations, tool calls), LLM-as-judge for fuzzy quality after calibrating it against human labels, thresholds in CI, and periodic human spot checks.

Question 19

What do you measure for a RAG system?

Accepted Answer

Retrieval: hit@k and MRR. Answers: groundedness (is each claim supported by the sources), relevance, and correct refusal when the corpus has no answer. Track cost and latency alongside.

Question 20

How would you gate a release on quality?

Accepted Answer

Run the eval suite on every prompt, model or retrieval change. Fail the build below a recorded floor, and never lower the floor without writing down why.

Question 21

What do you log for an LLM application?

Accepted Answer

One trace per request: model, latency, token counts and cost, tool calls, retrieved chunks, errors and user feedback. Redact or avoid storing sensitive prompt text, and keep keys out of logs entirely.

Question 22

How do you detect quality regressions in production?

Accepted Answer

Sample production traces into the eval pipeline, track feedback signals (thumbs, corrections), alert on cost, latency and error spikes, and compare quality by model or prompt version.

Question 23

Why use OpenTelemetry instead of a single vendor SDK?

Accepted Answer

A standard lets you change or add backends (Langfuse, Grafana, others) without re-instrumenting the code, and it correlates AI spans with the rest of your services.

Question 24

How do you deploy and roll back an AI feature?

Accepted Answer

Version prompts and models as config, gate releases on the eval suite, roll out behind a feature flag or canary, watch cost and quality metrics, and keep a one-step rollback to the previous prompt or model.

Question 25

How do you keep AI cost predictable?

Accepted Answer

Per-user and per-tenant limits, output caps, caching, routing simple steps to cheaper models, and budget alerts. Measure cost per completed task, not just token price.

Question 26

Which cloud, and why?

Accepted Answer

Start from constraints: where your data and team already are, compliance needs, and which managed model services you need. Name the lock-in you accept and the abstraction that limits it.

AI Coding Agent Ecosystem

How to answer anything here

1. Foundation Model

2. AI Coding Assistant

3. Agent Frameworks

4. MCP Layer

5. Context & Knowledge

6. Runtime & APIs

7. Evaluation & Testing

8. Observability

9. Deployment