Blog

There is a lot of noise around AI agents. Every month brings a new framework, a new benchmark leader and another promise of full autonomy.Some of these tools are useful. Most will change. The important question for a CTO is not which framework will win. The important question is which engineering capabilities will continue to create value when the models and frameworks are replaced.When I work with engineering organizations, I begin with one principle: harness first.Let us start with a simple mental model. An LLM can reason and generate an answer, but the model alone is not an agent. The agent is the complete system around it:<ul><li>what information the model receives;</li><li>what tools it can use;</li><li>which actions are allowed;</li><li>how work is planned and coordinated;</li><li>how results are checked;</li><li>when a human must take control. This system is the harness. In production, the harness is often more important than the difference between two strong models.</li></ul><h2>Harness first</h2>A model can produce an impressive demo with one prompt. A business process is different. It needs permissions, current data, clear rules, retries, audit logs, cost limits and a safe failure path.Imagine an agent that reviews the risk of a software release. Without a harness, it may produce a confident report from incomplete information. With a harness, it receives the current changes, test results, service ownership, recent incidents and SLO data. It follows the company’s risk policy, links every conclusion to evidence and requests approval before a high-risk release.The model writes the report. The harness makes the report useful and safe.I build this harness around four pillars.<h3>Pillar 1: Context engineering: Context engineering means giving the model the right information, in the right structure, at the right time.</h3>Andrej Karpathy’s analogy of the LLM as an operating system is helpful here. The model is similar to a processor; its context window is working memory. If we accept this model, context should be treated with the same discipline as a database workload.We need to know:<ul><li>which source is authoritative;</li><li>which records are relevant to the current task;</li><li>how fresh the information is;</li><li>what should be excluded;</li><li>where each fact came from.</li></ul>More context is not always better. A large amount of irrelevant information can make an agent slower, more expensive and less reliable.Example: During an incident, an operations agent does not need every document in the company. It needs the affected service map, recent deployments, current SLOs, relevant dashboards, the correct runbook and a few similar incident reviews. Each item should include its source and timestamp.This is why context engineering is also a data-management problem. Organizations need clear ownership, good metadata, reliable search and proper access control before agents can work reliably across the business.<h3>Pillar 2: Specification-driven engineering Agents are much more effective when intent is explicit.</h3>A specification turns a broad request into a working contract. It defines the objective, scope, constraints, interfaces, acceptance criteria, non-functional requirements and rollback conditions.The flow is straightforward:Specify → clarify → plan → create tasks → implement → validateThe specification should stay connected to implementation and testing. It is not a document that we write once and forget. It is the source of truth for the work.Example: “Move this service to Kubernetes” is not a useful instruction for an agent. A useful specification defines the availability target, traffic profile, resource limits, security controls, observability requirements, migration stages, acceptable downtime and rollback procedure. The agent can then create a plan that engineers can review against real business and technical outcomes.This becomes more important as agents generate code faster. Faster implementation without clear intent simply produces the wrong system more quickly.<h3>Pillar 3: Tool design and orchestrated flows Tool design is not plumbing. It is part of the product mission.</h3>An agent can act only through the tools we give it. Those tools define its practical abilities and its safety boundaries. A vague, powerful tool creates vague, powerful risk. A narrow, well-described tool makes the correct action easier and the dangerous action harder.Consider the difference between these two tools:<ul><li>run_command(command) gives the agent broad access and leaves policy decisions inside a text prompt;</li><li>get_deployment_status(service, environment) exposes one clear business capability with typed inputs, controlled permissions and predictable output. </li></ul>Good tools should have one clear purpose, structured inputs and outputs, useful error messages, minimum required permissions and safe defaults. Destructive operations should support dry runs, idempotency and explicit approval.Example: A release agent should not receive unrestricted shell access. It can receive separate tools to read deployment status, compare versions, create a rollback plan and request an approved deployment. The tool boundaries express the organization’s operating model.Orchestrate the full flow I prefer bounded, end-to-end flows over open-ended autonomy:Intake → retrieve context → plan → execute bounded actions → verify → approve → recordAn orchestrator can delegate narrow tasks to specialized sub-agents and then assemble the result. Each sub-agent receives only the context and permissions required for its task.Example: For a dependency vulnerability, one sub-agent assesses exposure, another proposes the smallest safe upgrade, and a third runs focused tests. The orchestrator compares the results, checks policy and prepares a pull request. Deployment still passes through the required human approval.This is autonomous where autonomy is useful and controlled where mistakes are expensive.MCP versus CLI Agents need a safe way to connect with the organization’s infrastructure. MCP and command-line tools solve different parts of this problem.MCP (Model Context Protocol) provides a standard way to expose data, tools and workflows to AI systems. It is valuable when the same capability must work across teams, agents or model providers. MCP can make tools discoverable and give them consistent descriptions and schemas.Example: An organization can expose approved Jira, Confluence, Slack and observability capabilities through MCP. Different agents can then use the same governed interfaces.CLI tools remain excellent when a mature, scriptable command already exists close to the infrastructure. They are transparent, reproducible and easy to run in CI/CD. They should still be wrapped with allow-lists, short-lived credentials, timeouts, dry-run modes and structured output.Example: An infrastructure agent can use a restricted wrapper around terraform plan or read-only kubectl. The harness records the exact command, result and exit code. A destructive change requires a separate approval.The decision is not MCP or CLI. MCP is a useful protocol layer for portable capabilities. CLI is often the direct execution layer. A good architecture may expose a controlled CLI operation through an MCP tool.<h3>Pillar 4: Evaluation from the beginning Evaluation is not the final project phase. It must begin with the first prototype.</h3>Without repeatable evaluations, a team cannot know whether a new model, prompt, retrieval strategy or tool description improved the system. A successful demonstration is not evidence of reliable production performance.Start with a small set of representative business tasks and known difficult cases. Measure:<ul><li>successful task completion;</li><li>factual accuracy and evidence quality;</li><li>tool-selection and execution errors;</li><li>policy or permission violations;</li><li>human correction and escalation rates;</li><li>latency and cost per successful result;</li><li>stability after model, prompt or tool changes.</li></ul>Example: An incident-triage evaluation should include normal incidents, ambiguous symptoms, missing telemetry and dangerous false positives. The agent should identify the likely owner, show its evidence, recommend safe next steps and escalate when confidence is low. Every system change should run against this set, just as software changes run regression tests.Evaluation changes the management conversation. “The agent feels better” becomes “Successful triage increased from 71% to 84%, cost stayed within target and unsafe actions remained at zero.”That is a result a CTO can manage.<blockquote>The CTO challenge: managing uncertainty Technology leaders are working through an unusual period. Model capabilities, vendor products and economics are all changing at the same time. We cannot remove that uncertainty, but we can build organizations that learn safely within it.</blockquote>The practical response is to:<ul><li>Standardize the harness, not one framework. Create common patterns for context, tools, identity, approvals, observability and evaluation. Keep models replaceable.</li><li>Separate reversible and irreversible decisions. Experiment quickly with models and frameworks. Apply stronger controls to data contracts, identity boundaries and production actions.</li><li>Build shared evaluation infrastructure. Turn real failures into evaluation cases. Every incident should make the harness stronger.</li><li>Define risk levels for agent actions. Reading a document is not the same as changing production. Permissions and approvals should reflect the possible impact.</li><li>Measure business outcomes. Track cases resolved, engineering time saved, lead time, error reduction and cost per successful task—not token volume or demo quality.</li><li>Invest in engineering capability. Context design, specification, tool contracts, evaluation, security and orchestration are transferable skills. They remain valuable when the stack changes.</li><li>Frameworks are tools, not the strategy AutoGen, CrewAI, Semantic Kernel, DSPy and other agent platforms can be useful. They can accelerate learning and reduce initial implementation effort. But they do not remove the hard production questions: data quality, permissions, failure handling, auditability, cost and operational responsibility.</li></ul>The durable capabilities are simpler:<ul><li>engineer the context;</li><li>make the specification explicit;</li><li>design tools around real business capabilities;</li><li>orchestrate bounded flows;</li><li>connect infrastructure through governed MCP and CLI interfaces;</li><li>evaluate from day one;</li><li>keep humans accountable for high-impact decisions.</li></ul>My conclusion is straightforward: build the harness before optimizing the model.Start with one bounded business process. Define the expected outcome. Connect only the context and tools that are required. Make the workflow visible. Test it on real cases. Expand autonomy only when the evidence supports it.This is less dramatic than the promise of a fully autonomous agent. It is also how reliable engineering systems are built.

Agentic Engineering Principles that Matter

Serhii Volokh Individual-entrepreneur OSVČ