Evaluating an AI SEO agent takes two weeks, not one demo call, because the failures that matter show up on two different clocks. Week one tells you whether a single run publishes cleanly and survives a real render check on the live page, not just a passing API response. Week two tells you whether that same run holds up once it meets the site’s own weekly reconciliation – sitemap and indexing status, drift watch, log completeness – because some failures only become visible on a second look, days after the first one looked fine.
This site runs an agent called organic-os against its own WordPress install, publishing roughly one post a day from a queue of approved briefs. The two-week window below is not a theoretical framework. It is the actual cadence this pipeline’s own weekly checks run on, and both weeks below are illustrated with something that actually went wrong here, not a hypothetical.
Week one: does a single run publish clean
The first thing to check is not whether an AI SEO agent can produce a plausible-looking post. Almost any current model can do that. The first real test is narrower: does the exact HTML it writes to WordPress render on the live page the way it looks in the editor, with no leftover formatting artifacts, and does the agent’s own verification step actually check the rendered page rather than trusting a successful API call.
That distinction is not academic on this site. A prior publish run once wrote eighteen backslash-escaped Gutenberg block comments into a live post, the result of shell escaping mishandling a literal exclamation mark during that run. WordPress could not parse the escaped comments, so on the live page they rendered as visible garbage text sitting in the middle of the article. The API response for that same post’s content had already passed a content scan before anyone looked at the page itself – the check that was supposed to catch the problem was looking in the wrong place. Since that incident, every publish on this site has to fetch the actual rendered HTML of the live URL, not just the REST API payload, and confirm three things: no literal block-comment text sitting outside a real HTML comment, no backslash-escaped characters anywhere in the body, and an opening paragraph that reads as ordinary prose rather than a fragment of markup. An HTTP 200 status code and a matching title are treated as necessary but explicitly not sufficient.
If you are evaluating an agent yourself, this is the cheapest week-one test available: ask to see the actual live URL it produced, view the page source, and look for exactly this class of problem. An agent that can show you a clean rendered page, and can describe what its own verification step checks for, has already cleared a bar that a demo transcript alone cannot prove.
What this site does, and what it refuses to do
Inside that first week, the boundaries matter as much as the output. This pipeline logs every publish with the brief it came from and the resulting WordPress post ID, and every approval with a named actor, a channel, and a timestamp attached to the decision – the same approval structure described in the site’s own comparison of an SEO agent against a human SEO consultant. It does not treat an incomplete or malformed approval record as an implicit yes; a missing decision field stops the publish gate outright rather than letting the run guess what was probably meant.
It also does not have write access to anything that would let it hide its own mistakes. The application-password credential this pipeline publishes with is scoped to editor-level actions – creating and updating posts and media – and does not carry the higher `manage_options` capability needed to change site-wide settings. That distinction turns out to matter directly in the second week’s failure mode below, because it means the agent can observe a problem without being able to fix it by itself.
Failure modes observed in production
Two specific incidents from this site’s own operating history are worth naming here, because each one only became visible on a different timescale than the one before it.
The first showed up on the week-two clock. This site’s Rank Math XML sitemap cache froze at a fixed timestamp and stayed frozen for twenty-eight straight days across eight consecutive weekly checks, even as new posts kept publishing successfully every day. Over one of those weeks alone, the count of published posts missing from the sitemap grew from eighteen to twenty-three, with the single largest jump – two posts in one day – coming from two articles that happened to publish on the same date. A direct index-inspection check on two of the missing URLs confirmed Google considered them entirely unknown, not merely slow to crawl. That is a measurable indexing cost sitting quietly behind a pipeline that, on any single day, looked like it was working. The editor-level credential that publishes the posts gets refused with a permissions error when it tries to clear that sitemap cache itself, because purging it requires the higher capability the publishing credential deliberately does not have. Fixing it requires a person with admin access, which is exactly the kind of gap a one-day evaluation would never surface, and only a second week of checking caught.
The second is the render-verification incident already described above, which showed up immediately, on the week-one clock, the moment someone looked at the actual page instead of the API response. Both incidents are logged in the same place: the site’s own append-only decision record, which is examined in more depth in this site’s piece on why an SEO automation needs an audit trail. Neither incident was caught by the agent congratulating itself on a successful publish. Both were caught by a specific, repeatable check looking for a specific class of problem – which is the actual pattern worth evaluating for, more than any general claim about accuracy.
Week two: does it hold up under reconciliation
The second week is where slower failures surface. This site runs a weekly reconciliation pass that checks the sitemap and indexing status, watches for content or ranking drift on already-published posts, and cross-checks its own decision log against the site’s actual publish history to make sure nothing was written and never recorded, or recorded and never actually written. The sitemap freeze above is the clearest example of why this matters: nothing about it was visible from a single day’s publish log, and it took a full week of the same check returning the same frozen timestamp before the pattern was unambiguous. The same reconciliation habit is what a broader comparison of different AI SEO agent approaches should be checking for, not just publish volume.
If you are running this evaluation on a vendor or a self-hosted setup, ask specifically whether there is a recurring check like this, what it looks for, and whether it has ever caught something the daily output did not. An agent with no answer to that question has not been tested past day one, however many posts it has shipped. The same logic extends to ongoing content health after publish, which is why a related check worth asking about is how the system watches for drift in already-published content over time, not just at the moment of publishing.
When a human must take over
None of the above removes the person from the loop, and the two incidents here show exactly where the line sits. The render-verification failure was something the pipeline could catch and, with a repaired script, fix itself on the next run. The sitemap freeze was something the pipeline could only observe and report – a person with admin-level access had to be the one to actually clear it, because the publishing credential is deliberately not allowed to touch that setting.
That is the practical dividing line to evaluate for: an agent should be able to tell you clearly which category a given problem falls into, rather than treating every failure as either fully automatic to fix or a reason to distrust the entire system. A two-week evaluation, run against a real publish history rather than a demo, is what actually surfaces which category most failures fall into for a given setup – and whether the system tells you the difference honestly when they do.
Frequently asked questions
How long does it really take to evaluate an AI SEO agent?
Two weeks is the practical minimum on this site’s own experience, because it covers two different failure clocks: a first week that tests whether a single publish renders cleanly and survives a live-page check, and a second week that tests whether the same output holds up under a full weekly reconciliation cycle, such as a sitemap or indexing check. A shorter test can miss failures that only appear on the second clock.
What is the single cheapest check to run in week one?
Fetch the actual live URL the agent published and view its page source, rather than trusting a successful API response. On this site, a real incident once passed an API-level content scan while the live page rendered visible, broken markup – an error that was only caught by checking the rendered page directly.
What kind of failure only shows up in week two?
Slow-moving infrastructure problems, such as a sitemap cache that silently stops updating. On this site, that exact failure ran for twenty-eight consecutive days across eight weekly checks before being fully confirmed, growing the count of unindexed published posts over that time, even though each individual day’s publish looked successful.
Does a clean two-week evaluation mean a human is no longer needed?
No. On this site, the publishing credential is deliberately scoped so it cannot fix every problem it can detect – a sitemap-cache purge, for example, requires higher-level access than the agent is given. A well-run evaluation should tell you which failures the system can repair itself and which ones still require a person with broader access, not eliminate that person from the loop.
Leave a Reply