AI agents for software engineering are AI models that take a goal, use tools like your test runner, and work in a loop until a check passes. For a Chief Technology Officer (CTO), they pay back first in coding, QA and DevOps, where 70% of developers using agents told Stack Overflow in 2025 they save time on specific tasks.
The hard part is control: Gartner predicts over 40% of agentic AI projects will be canceled by the end of 2027, citing cost, unclear value and weak risk controls.
This page covers agents your engineering team runs and agents inside your product. Agents across internal systems like the ERP and CRM are covered in AI Agents for the CIO.
An AI agent in software engineering takes a goal, plans steps, calls tools (edit files, run tests, open a pull request) and checks its own results, while a coding assistant suggests code a developer accepts line by line. METR found the task length agents finish at a 50% success rate has doubled about every 7 months.
Every coding agent has the same 4 parts: a model, a set of tools, a loop, and a stop condition. The stop condition is usually a check, like the test suite passing, or a limit on steps or spend.
That loop is what makes an agent useful and what makes it risky. A coding assistant can only suggest. An agent can run a command.
How you use AI coding assistants day to day, and whether they make developers faster, is covered in AI for the CTO. This page starts where the tool starts acting on its own.
Gartner estimates only about 130 of the thousands of vendors selling “agentic AI” have real agentic capabilities. It calls the rest “agent washing”: chatbots, assistants and RPA tools with a new label.
Ask a vendor 3 questions:
If the answers are vague, you’re buying an assistant at agent prices.
METR’s March 2025 study measured task length in human time. Agents succeeded almost 100% of the time on tasks that take a person under 4 minutes, and under 10% of the time on tasks over about 4 hours.
The trend moves fast, so I re-check those limits every quarter. The planning rule holds either way: give an agent a ticket a developer could finish in an hour or two, with a test that proves it’s done.
Hand agents work that comes with its own check: dependency upgrades, test writing, flaky-test triage, small bug fixes with a reproduction, and first-pass incident triage. In my planning model for an 8-developer team, those 5 jobs give back about 17.5 hours a week, worth roughly $71,400 a year at an $85 loaded hourly rate.
A lot of agent advice says to start with new features, because there’s no legacy code to break. I disagree, and I start teams on maintenance and QA work.
The reason is the verifier. A dependency upgrade either passes the test suite or fails it, and a flaky test either reproduces or it doesn’t. A new feature has no test yet, so a person has to judge every line, and the agent saves far less time.
Google’s 2025 DORA report, drawn from nearly 5,000 technology professionals, describes AI as an amplifier of the engineering system it lands in. Strong tests make an agent useful, and weak tests let it ship mistakes faster.
Verified work also builds the team’s trust faster. Each merged agent pull request with a green test run is evidence the team can see.
Here’s a planning model for an 8-developer team at a $20M software company. The hours are assumptions from the kind of week I see at this size, so replace them with your own baseline before you budget against them.
| Engineering task | Team hours/week now | With agents | Hours given back | Mechanism |
|---|---|---|---|---|
| Dependency upgrades and security patches | 6 | 2 | 4 | Agent opens a draft pull request from each scanner alert, runs the full test suite in a sandbox, and a developer reviews the diff |
| Writing tests for uncovered code | 10 | 5 | 5 | QA agent reads the coverage report, writes tests for untested paths, and runs them until they pass; the code owner reviews each test |
| Flaky-test triage | 4 | 1.5 | 2.5 | Agent reruns failing tests, compares logs across runs, and labels each one as a flake or a real failure with the evidence attached |
| Small bug tickets with a reproduction | 12 | 8 | 4 | Agent turns the reproduction into a failing test, fixes the code until it passes, and opens a pull request for human review |
| First-pass incident triage | 5 | 3 | 2 | DevOps agent with read-only access pulls logs, recent deploys and error rates into a summary; the on-call engineer decides what to roll back |
| Total | 37 | 19.5 | 17.5 |
Every row clears my 20% rule: if an agent would give back less than 20% of a task’s time, cut or redesign the task before you build an agent for it.
The incident row stays read-only on purpose. The agent drafts the rollback command, and a person runs it.
Here’s what 17.5 hours a week is worth, with every assumption stated.
The math on engineering agents
Those hours go back to the roadmap and the tech debt list. Every role stays in this model; the team ships more of what it already planned.
Here’s how I’d model a first coding agent for a 6-person software team. It’s a planning example built from typical numbers, so use it as a template for your own baseline.
The agent gets one bounded job: add unit tests to an established internal service. Engineers name the behaviors that need coverage, and the agent reads the existing code and tests, drafts new cases, and runs them in an isolated environment. Then it opens a proposed change for review.
| Engineering effort | Before | With agent |
|---|---|---|
| Selecting cases and defining expected behavior | 4 hours | 4 hours |
| Writing tests and preparing fixtures | 12 hours | 6 hours |
| Reviewing, correcting and validating tests | 4 hours | 6 hours |
| Total per week | 20 hours | 16 hours |
That’s 4 engineering hours a week, a 20% reduction, after counting the extra review time. The team puts those hours into reproducing production bugs and adding regression tests.
Look at the review row. It goes up by 2 hours, and the model only holds because the team budgets for that.
A 20% gain sits right on my 20% rule, so this job clears the bar with no room to spare. If review creeps up another hour, cut the job or redesign it.
The agent can’t merge its own changes. Engineers check that each assertion tests the behavior the team intended, because a test written by reading the current code can lock in a current bug.
Passing tests and higher coverage are the minimum bar. Judging test quality still takes a person reading the assertions.
A coding agent needs 4 guardrails: its own scoped credential, a sandbox that can’t push to protected branches, the same security scans and review as human code, and human approval before anything reaches production. AWS’s July 2026 control framework for coding agents calls for 2 human approval gates: one after security scans and one before deployment.
The credential matters most. An agent running on a developer’s personal token can do anything that developer can do, including changes no one asked for.
The AWS framework puts it plainly: give each MCP server the minimum tools and a dedicated, scoped-down credential, and keep an allowlist of reviewed servers. It also warns against wildcard auto-approval, which removes the human prompt on every tool call.
Prompt injection is text that hijacks an agent: an instruction hidden in an issue, a README, a web page or a log line. NIST’s Center for AI Standards and Innovation (CAISI) ran a red-team competition with more than 400 participants and 250,000 attack attempts, and found successful attacks against all 13 frontier models tested.
So assume any agent that reads outside text can be steered. The fix is permissions: an agent that reads untrusted content gets read-only access, and anything with write access reads only trusted input.
OWASP published its Top 10 for Agentic Applications in December 2025. I use it as the checklist for a security review of any agent before it moves up a rung.
I put every engineering agent on a 4-rung ladder. Every agent starts on rung 1, and it climbs only when it clears the gate.
| Rung | The agent can | Gate to climb to this rung | Typical work |
|---|---|---|---|
| 1. Read | Read code, logs and tickets; write nothing | None; every agent starts here | Incident triage, explaining unfamiliar code |
| 2. Propose | Open a draft pull request from a sandbox branch | Passes its 20 to 50 eval tasks; 2 weeks of reviewed output on rung 1 | Tests, dependency upgrades, small bug fixes |
| 3. Act in staging | Rerun CI, deploy to staging, run migrations on test data | 4 weeks on rung 2 with change failure rate flat | Flaky-test triage, release prep |
| 4. Act in production with approval | Run a prepared change after a named person approves it | Written rollback plan, full audit log, security review against the OWASP agentic list | Config changes, approved rollbacks |
There’s no fifth rung. Unattended writes to production stay off the table at a $5M to $50M company, because there’s no separate platform team to clean up after one.
Build a product agent from 4 parts: a model, a short list of tools with tight inputs, a loop with a step limit, and a log of every tool call. The Model Context Protocol (MCP) is the open standard for connecting those tools, and its November 2025 specification says hosts must get explicit user consent before invoking any tool.
MCP defines 3 things a server can expose: tools (functions the model can run), resources (data for the model or user), and prompts (templated workflows). It runs on JSON-RPC 2.0 messages between a host, its clients and the servers.
The specification is blunt about risk. It says tools represent arbitrary code execution, and that tool descriptions should be treated as untrusted unless they come from a trusted server.
Every tool you add is another decision the model can get wrong. I start a product agent with 3 to 5 tools, each with a narrow input schema, and add more only when the eval suite shows a gap.
Chain length drives failure rates. If each step in an agent’s task succeeds 95% of the time, a 5-step chain succeeds 0.95⁵ = 77% of the time, and a 10-step chain succeeds 0.95¹⁰ = 60% of the time.
So design for short chains. Add a check between steps (validate the tool output against a schema before the next call), cap the loop at a fixed number of steps, and set a spend limit per run.
Put tool servers behind the same service layer that owns your other model calls. Python handles tool logic and error handling well, and Cloudflare Workers can host lightweight tool endpoints close to users.
Log the prompt, each tool call with its arguments, each result, the model, and the token cost per run. That log is your audit trail, your debugging record, and your eval data source in one place.
Model selection, retrieval and the service layer itself are covered in AI for the CTO.
Test an agent with an eval suite: 20 to 50 tasks drawn from real failures, each with a grader that checks the outcome, run on every change to the prompt, tools or model. Anthropic’s January 2026 guidance recommends that starting size and code-based graders wherever a deterministic check exists, because they’re fast, cheap and reproducible.
Anthropic’s guidance describes 3 grader types. Code-based graders check outcomes exactly. Model-based graders handle judgment calls but need calibration against people, and human graders set the gold standard at the highest cost.
For coding agents, you already own the best grader: the test suite. For product agents, write a check per task, like “the refund tool was called once, with the right order ID.”
Capability evals start with a low pass rate and give the team something to climb. Regression evals should sit near 100%, and a drop blocks the merge the same way a failing unit test does.
When a capability eval passes consistently, move it into the regression suite. Over a quarter, the regression suite becomes a record of every failure you’ve fixed.
A single passing run can hide an inconsistent agent. Anthropic’s guidance gives the math: an agent with a 75% per-trial success rate passes 3 trials in a row only about 42% of the time (0.75³).
For anything customer-facing, run each eval task 3 times and count it as passed only when all 3 pass. Then read the transcripts: Anthropic reported a model scoring 42% on one benchmark that rose to 95% once grading bugs were fixed.
Measure agent payback as hours given back minus agent cost, against a 4-week baseline taken before rollout. At published Claude Sonnet 5.5 prices, a 2-million-token coding agent run costs about $1.56, so 150 runs a month costs about $234 against roughly $5,950 a month for 17.5 weekly hours of developer time.
Here’s the cost of 1 run, using Anthropic’s published API pricing (accessed October 2026). Agents reread their context on every step, so most input tokens are cache reads.
| Model | Fresh input | Cache reads | Output | Per run | 150 runs/month |
|---|---|---|---|---|---|
| Claude Sonnet 5.5 ($2 in / $0.10 cached / $10 out per million) | $0.80 | $0.16 | $0.60 | $1.56 | $234 |
| Claude Opus 5.5 ($4 in / $0.20 cached / $20 out per million) | $1.60 | $0.32 | $1.20 | $3.12 | $468 |
Cache writes add a little on top. Your real token counts will differ, so log them from week 1 and replace my assumptions.
Developer review time costs far more than tokens, which is why review minutes belong in the payback math.
Run your own hours and rates through the AI ROI Calculator before you expand past the first 2 agents.
Gartner’s cancellation reasons map to these numbers. Rising cost shows in cost per merged change, unclear value shows in merge rate, and weak risk control shows in change failure rate.
Ordered people first, then process, then tech. Each step names who does it and what it produces.
Days 1 to 30: people
Days 31 to 60: process
Days 61 to 90: tech
I’m Lee Pomerantz, Certified Chief AI Officer, founder of eMediaAI, Fort Wayne, Indiana. Every engagement runs in one order: people first, then process, then tech.
The AI Opportunity Blueprint™ is where it starts. It’s a 10-day, $5,000 forensic audit: your team records the work, I map where the hours and data go, and you get a prioritized workflow map, an ROI projection, an implementation roadmap and governance recommendations.
For a CTO, the Blueprint scores candidate agent jobs against the 20% rule and places each one on the permission ladder. If the numbers don’t hold, the report says don’t build it.
When an agent clears the Blueprint, I build it with error handling, logging and a handoff your team can run: an SOP, a Loom walkthrough, a governance doc and an ROI report. That’s the AI agents work.
AI coding agents are safe on a production codebase when they work in a sandbox branch, use a dedicated scoped credential, and every change passes security scans and human review before merge. NIST’s CAISI reported in March 2026 that red-teamers hijacked all 13 frontier models it tested, so agents that read untrusted text like issues or web pages should get read-only access.
The Model Context Protocol (MCP) is an open standard for connecting AI applications to tools and data. An MCP server exposes tools, resources and prompts over JSON-RPC 2.0 messages. The November 2025 specification says hosts must get explicit user consent before invoking any tool, and treats tool descriptions as untrusted unless the server is trusted.
AI coding agents cost a seat fee or per-token usage, and usage dominates for heavy work. At Anthropic’s published October 2026 prices, a run of 2 million input tokens (1.6 million cached) and 60,000 output tokens costs about $1.56 on Claude Sonnet 5.5 and $3.12 on Claude Opus 5.5. An 8-developer team running 150 Sonnet runs a month spends about $234 in tokens.
Rolling out coding agents takes about 2 hours a week from 1 named agent owner for 90 days, plus 2 weeks of each developer logging hours on chores. Building an eval suite takes a few days per agent job, because each needs 20 to 50 tasks drawn from real tickets. Review time on agent pull requests is the ongoing cost to track.
Calculate AI agent ROI as hours given back × loaded hourly rate, minus token, seat and review costs. With a 4-week baseline, 17.5 hours a week × $85 × 48 weeks = $71,400 a year in engineering capacity. Subtract agent spend and the review minutes each agent pull request takes, then confirm change failure rate held flat.
Agentic AI projects fail mostly on cost, unclear value and weak risk controls, the 3 reasons Gartner gives for predicting over 40% will be canceled by the end of 2027. Teams also give agents tasks that are too long: METR found agents succeed under 10% of the time on tasks that take a person over 4 hours.
Prompt injection is hidden text that hijacks an AI agent, such as an instruction planted in an issue, a document or a log line the agent reads. The OWASP Top 10 for Agentic Applications, published December 2025, lists agent goal hijack first among its 10 risks. The main defense is permissions: an agent that reads untrusted text gets read-only access.
An AI agent eval suite can start with 20 to 50 tasks drawn from real failures, according to Anthropic’s January 2026 guidance on agent evals. Each task needs a grader that checks the outcome, ideally with code. For customer-facing agents, run each task 3 times, since a 75% per-trial success rate passes 3 runs in a row only 42% of the time.
AI agents are changing what software engineers spend their hours on, while the employment data still points to growth. The Bureau of Labor Statistics projects software developer jobs will grow 10% from 2025 to 2035. At eMediaAI, agent work is scoped to give engineering hours back to the roadmap, and engagements aimed at cutting headcount are declined.
A CTO’s first step with AI agents is a one-page note telling the engineering team what agents are for and that every agent change gets human review, followed by naming 1 agent owner. Next, take a 4-week baseline of review time and change failure rate. Then start 2 agents on read-only work, such as incident triage or flaky-test analysis.
Book a free 20-minute call to see if the AI Opportunity Blueprint™ is right for you.
Book Your Call
© 2026 eMediaAI.com. All rights reserved. Terms and Conditions | Privacy Policy
