A WordPress robots.txt That Does Not Cost You Traffic

A WordPress robots.txt only supports four directives that Google actually reads: user-agent, disallow, allow, and sitemap. Everything else, including a noindex line, is silently ignored. Most robots.txt problems on WordPress sites come from either a stray physical file overriding WordPress’s own output, or a disallow rule that blocks something Google needs to render or index the page correctly. Check which file is actually live, fix the specific rule causing the problem, and confirm the change with Google’s own live-URL test rather than assuming it worked.

Why it happens

WordPress does not ship with a physical robots.txt file. Instead, core’s do_robots() function generates one on the fly whenever a request hits the robots.txt URL and no real file exists in the site’s root directory. The default virtual output is User-agent: *, Disallow: /wp-admin/, and Allow: /wp-admin/admin-ajax.php. That admin-ajax.php exception exists because a blanket /wp-admin/ block was interfering with front-end AJAX requests that themes and plugins depend on; it was added to core through a dedicated ticket after developers ran into exactly that problem.

The whole virtual output also passes through a robots_txt filter before WordPress echoes it, so an SEO plugin like Rank Math can rewrite it without ever touching a file on disk. That is also where the first common failure comes from: if a physical robots.txt file already exists in the root directory, WordPress’s virtual generator never runs at all. The web server just serves the static file. Rank Math’s own robots.txt editor documents this directly, because it is a support question they get often: their editor writes to the virtual output, so a leftover physical file silently overrides every change made inside WordPress until that file is edited over FTP or deleted.

The second common failure is what gets blocked. A rule that was reasonable advice a decade ago, like disallowing /wp-content/ or /wp-includes/ to save “crawl budget,” now actively hurts a site, because Google renders pages using their CSS and JavaScript to judge layout and content. Blocking those directories can make Googlebot see a broken, unstyled version of the page. A narrower but just as damaging version of the same mistake is disallowing /wp-content/uploads/ specifically, which removes every image in the media library from Google Image Search even though the post pages around them stay indexable. A third pattern shows up on sites that recently changed how they handle crawler traffic: someone edits robots.txt to block an AI crawler (a legitimate and increasingly common reason to touch the file, and one covered in more depth in this site’s guide to blocking AI crawlers), and a typo in the user-agent line or path ends up blocking Googlebot along with it.

Step-by-step fix

1. Confirm what is actually live. Visit https://yoursite.com/robots.txt directly in a browser. Do not trust a cached copy or a plugin’s settings screen; the browser request shows exactly what a crawler receives right now.

2. Check for a physical file first. Connect over FTP or your host’s file manager and look in the site’s root directory for a file literally named robots.txt. If one exists, everything you change inside WordPress or an SEO plugin is being ignored in favor of that file, because do_robots() never runs while a physical file is present.

3. Edit the file that is actually in control. If a physical file exists and you want WordPress or Rank Math to manage it going forward, delete or rename that file so WordPress’s virtual generator (and any plugin filter attached to it) takes over. If no physical file exists, edit the content through Rank Math SEO → General Settings → Edit robots.txt, which requires Advanced Mode to be turned on since the editor is hidden in Easy Mode by default.

4. Remove any rule broader than it needs to be. Replace a blanket Disallow: /wp-content/ or Disallow: /wp-includes/ with nothing at all unless there is a specific subfolder you actually need hidden, and never disallow /wp-content/uploads/ if you want those images to show up in Google Image Search.

5. Delete any noindex line inside robots.txt. Google stopped supporting an unsupported noindex: directive inside robots.txt years ago, and it never worked reliably before that either. If the goal is to keep a specific page out of Google’s index, that has to be a noindex meta tag or an X-Robots-Tag HTTP header on the page itself, not a line in this file.

How to verify it worked

Reload https://yoursite.com/robots.txt in an incognito window after making changes and read the raw text, not a rendered preview. Confirm the disallow rules you removed are actually gone and that a Sitemap: line still points at your real sitemap, since a robots.txt block is one of the more common reasons a submitted sitemap stops getting crawled the way it should (a failure mode this site has covered separately in its guide to a WordPress sitemap that stops updating).

Then run the file through Google Search Console’s robots.txt testing tool, or use the URL Inspection tool’s “Test Live URL” feature on a page you expect to be crawlable, and confirm Google’s own reading of the rule matches what you intended. Do not rely on assumption or on how the rule reads to you; robots.txt matching is case-sensitive and wildcard behavior is easy to get subtly wrong.

What to do when the standard fix fails

If the live file still shows the old rules after you have edited it, look for caching in front of the request. Some hosts and CDNs cache the robots.txt response separately from the rest of the site, so a plugin-level or file-level change does not show up until that specific cache entry is purged.

If Search Console still reports pages blocked that you believe should be crawlable, re-check the exact path and user-agent match rather than assuming the rule is wrong. A rule scoped to User-agent: * applies to every crawler that does not have a more specific block further down the file, and a trailing or missing slash on a disallowed path changes what it matches.

If the site is also managing an llms.txt file for AI crawlers alongside robots.txt, keep the two consistent rather than treating them as unrelated; a mismatch between what robots.txt allows and what an llms.txt file on WordPress promises to AI crawlers is a newer version of the same “which file is actually in control” confusion that causes most robots.txt problems in the first place.

Frequently asked questions

Can I use robots.txt to remove a page from Google’s index?

No. Robots.txt controls crawling, not indexing. If a page is blocked by robots.txt, Google may still index the URL and show it in search results without a description, because it was never able to crawl the page to see any noindex signal. Removing a page from the index is a job for a noindex meta tag or header, which only works on pages Google can actually reach; this site’s separate guide to common WordPress noindex mistakes covers that side of the problem in detail.

Does WordPress create a real robots.txt file by default?

No. WordPress generates a virtual one on request through the do_robots() function whenever no physical file exists in the root directory. If you never created one yourself and no plugin created one for you, there is no actual file on the server; the URL just resolves to generated output.

Will an SEO plugin’s robots.txt editor work if a physical file already exists?

No. A physical file in the root directory is served directly by the web server and takes priority over anything WordPress or a plugin would otherwise generate. The physical file has to be edited directly or removed before a plugin’s built-in editor has any effect.

Is it safe to block /wp-admin/ entirely?

Not entirely, and WordPress does not do this by default. The default rule blocks /wp-admin/ but explicitly allows /wp-admin/admin-ajax.php, because front-end features on many themes and plugins depend on that file being reachable by crawlers as well as visitors.

---