What coding agents can and can't do

Agents measure: they create impact checks, read verdicts and record decisions. They never publish to your site, and safety rules keep their results honest.

A coding agent connected to Needlr can measure what it ships. It can't change your website. Here is exactly what it can do, what it can't, and the safety rules that keep its results honest.

Think of a lab assistant

A lab assistant runs experiments, takes measurements and writes down the results. But only the head of the lab decides what goes into the product. And every step goes into the lab notebook.

In Needlr, the agent is the assistant. You are the head of the lab. The notebook is Needlr's audit log.

What an agent can do

• Ask what's measurable on its projects: the KPIs, the weekly visitors, and how long a verdict would take.
• Create an impact check for what it shipped: Before vs after for a release that's already live, Adoption for a new feature, or a split test in its own code.
• Read the verdicts of its impact checks.
• Record what it did next: keep, roll back, iterate, give up, or fix the measuring. Rolling back or giving up on its own running split test stops it, and everyone gets the current version again.
• Look up past impact checks on its projects, so it doesn't repeat an idea that already lost.
• Propose a KPI. You see it under Waiting for you on Agents, and press Create KPI or Dismiss.

What an agent can't do

• Publish to your site. It never touches Tag Manager. Publishing and shipping a winner stay with a person, with a code from their phone. This is built in. No setting changes it.
• Create a KPI itself. It can only propose one.
• Peek. Until the verdict, it only sees progress and an expected date, never which way the result leans (why).
• Work on other projects. Only on the projects you picked for it.
• See single visitors. It can't read anything about one visitor.
• Ask without limit. Each agent can create up to 50 impact checks a day.

You choose how its split tests start

When you connect an agent, you pick one of three options. You can change it later on the agent's card, under Its split tests start. Letting split tests start on their own is the account Owner's call.

• Always ask me: its split tests wait until someone starts them in Needlr.
• On its own, when the code came through a pull request: someone can review the code before it's live. Without a pull request, it asks.
• Always on its own: its split tests start as soon as it asks. You're responsible for what it ships.

Giving an agent more needs a code from your authenticator app: connecting it, changing its projects, letting its split tests start on its own, a new key, or turning it back on after a pause. Pausing it, or going back to Always ask me, doesn't. Owners and Editors manage agents.

Safety rules for its split tests

• 95% confidence, always. Whatever the agent asks for.
• At least 7 full days, and the planned number of visitors, like any split test.
• At most 2 at once on the same KPI of a project.
• It stops by itself on harm. Needlr checks at every update. When version B is clearly hurting your KPI, or one of up to two guardrail KPIs the agent named, the test stops and everyone gets the current version. It shows as Stopped · it hurt. It only stops for harm: a win still waits for the full verdict.
• No repeats. A change that already didn't win isn't measured again, unless the agent sends it as a follow-up.
• Releases close together are flagged. When other releases on the same KPI are within 14 days, the agent is told the evidence is weak.

Why “not confirmed”?

Think of flipping a coin 100 times. A few lucky streaks will show up, even with a fair coin. An agent that runs lots of split tests is like that: some “wins” are just luck.

So Needlr looks at all the agents' split tests in a project together. Each new verdict gets its own bar, based on the tests before it. That keeps the share of false wins at 5% or less, on average. A win that doesn't clear its bar is not confirmed, and the agent is told to try again rather than keep it.

What's in the audit log

Needlr writes down every step an agent takes, under the agent's name and the key it used. Open Activity in the menu on the left to see it, newest first:

Each impact check an agent asked for also shows Requested by and the agent's name on its check page.

Pause or disconnect

• Pause on the agent's card stops it at once: its key stops working. Resume turns it back on.
• Disconnect stops its keys at once. Its impact checks and their history stay.
• Keys run out after 90 days. Press New key before then. The old key keeps working until you press Revoke next to it, so nothing breaks while you switch.

Common mistakes

• Choosing Always on its own on day one. Start with Always ask me, see what your agent asks for, then decide.
• Leaving verdicts undecided. An agent learns from what was decided. Look at Waiting for you once a week.
• Giving it every project. Start with one.