Category: Guides

A WordPress category page can lose its index slot in two different ways, and the fix depends on which one you have. If Rank Math has quietly noindexed an empty or thin category, the fix is a single toggle: open Titles & Meta, find the Categories tab (or the global “Noindex Empty Category and Tag Archives” setting), and switch it off for any category that actually has posts worth showing. If the problem is duplication instead – your category’s intro paragraph repeating on every paginated page, or a stale canonical tag – the fix is to trim that repeated text down to page one and confirm each paginated page canonicalizes to itself, not back to page one. Check which failure mode you have before you touch a setting; the two look similar in Search Console but need opposite responses.

Why it happens

WordPress builds a category archive automatically for every category you create, whether or not you meant it to become a real destination. That archive lists the posts assigned to the category, paginates once the list grows past your posts-per-page setting, and gets its own URL and its own spot in the index – or it would, if nothing else intervened.

Rank Math intervenes by default. Its Titles & Meta settings include a global toggle called “Noindex Empty Category and Tag Archives,” and it ships turned on. The logic behind it is reasonable on its own: an empty category page is thin content with nothing to show a visitor, so there is no reason to ask Google to index it. The problem shows up when a category that used to be empty gets posts added later, or when a category was never empty but still gets swept into the same noindex treatment through a separate per-taxonomy Robots Meta setting under the Categories tab. Either way, you end up with a category page that has real posts on it and still carries a noindex tag, because nothing told Rank Math the category stopped being empty.

The second failure mode is different and has nothing to do with the noindex toggle. It is duplication, and it comes from how category archives handle their own description text. Many themes print the category’s description paragraph at the top of every paginated page in that archive – page 1, page 2, page 3, all showing the identical intro copy above a different set of post excerpts. That repeated paragraph is not the whole page, but it is enough overlapping content across multiple URLs to register as duplication, and it is a separate issue from whatever your noindex settings say about the archive as a whole.

Canonical tags compound this if they are set up wrong. Older SEO guidance had sites canonicalize every paginated category page back to page 1, on the theory that only page 1 deserved to rank. Current practice has moved to self-referencing canonicals instead – each paginated page points to itself, not to page 1 – because collapsing every page’s canonical onto page 1 tells Google to treat pages 2, 3, and beyond as non-canonical duplicates of page 1, which can pull posts that only appear on later pages out of the index entirely.

Step-by-step fix

Work through these in order. The first two apply if your category page is noindexed and should not be; the second two apply if it is indexed but competing with itself across pagination.

1. Confirm whether the category is actually empty. Open the category in your WordPress admin and check its post count. If it has posts and still is not showing up in search, the empty-category noindex rule is not the real cause – skip to the pagination checks below.

2. Check Rank Math’s Categories tab under Titles & Meta. Look for a per-category or per-taxonomy Robots Meta override set to No Index, separate from the global empty-archive toggle. If a specific category has this switched on and you want it indexed, turn Robots Meta off for that taxonomy, or clear the override on the individual category’s own edit screen. This is the same Titles & Meta interface most sites configure once for post types and then forget has a separate tab for taxonomies.

3. Set the Advanced Robots Meta values for categories you want indexed. Once a category is not noindexed, set Max Snippet to -1 (unlimited), Max Video Preview to -1, and Max Image Preview to Large, so Google has no reason to render a truncated snippet for a page you want performing well in search.

4. Trim the repeated intro text to page one only. If your theme or a category-description widget prints the same paragraph on every paginated page, change that template logic (or the relevant plugin setting, if one exists) so the description shows only when the visitor is on page 1 of the archive. Pages 2 and beyond should show just the post listing, with no repeated intro copy above it.

5. Confirm each paginated page canonicalizes to itself. Do not force every paginated URL’s canonical tag back to page 1. WordPress core’s own canonical-resolution functions – wp_get_canonical_url() and the redirect_canonical() filter – handle this correctly by default; the failure case is usually a theme or plugin overriding that default with custom canonical logic left over from older guidance. This is the same category of canonical-tag mismatch that causes problems elsewhere on a site, just applied to paginated archives instead of single posts.

How to verify it worked

Do not trust the settings screen. Visit the live category URL in a fresh, logged-out browser tab and view the page source. Look for <meta name="robots" content="noindex... in the <head> – if you turned noindex off and that tag is still there, either the setting did not save or a cache is serving the old version. Clear any full-page cache and check again before assuming the fix failed.

For the pagination and duplication side, load page 2 and page 3 of the same category directly and confirm two things in the rendered source: the repeated intro paragraph is gone, and the canonical tag on each page points to that page’s own URL, not back to page 1. If you are unsure what your permalink structure does to these paginated URLs in the first place, it is worth a quick check against how your permalink settings shape category and archive URLs generally – a nonstandard structure can make it harder to spot which paginated URL you are actually looking at.

Search Console is the other half of verification, on a longer timeline. Open the Page Indexing report and check for “Duplicate without user-selected canonical” or “Duplicate, Google chose different canonical” against the category’s URLs. Neither status is exclusive to category pages, but seeing either one attached to a category archive you just fixed is a sign the duplication issue predates your change and Google has not recrawled it yet – not necessarily a sign the fix did not work.

What to do when the standard fix fails

If you removed the noindex tag and the category still is not indexed weeks later, that gap is usually a crawl-and-recrawl delay, not a broken setting. A noindex tag tells Google to drop a page from the index the next time it is crawled; removing that tag does not force an immediate recrawl on its own. Give it time, and use Search Console’s URL Inspection tool to request indexing directly if the category has not reappeared after a few weeks.

If the repeated intro text keeps showing up on paginated pages even after you changed the relevant setting, check whether your theme hardcodes the category description into its archive template file rather than pulling it conditionally. Some themes print category_description() unconditionally near the top of every archive page, with no built-in check for which page number is currently loading – in that case, the fix has to happen in the template file itself (or a child-theme override), not in a plugin setting, since there is no toggle to change if the theme never added one.

One more edge case: if canonical tags still point back to page 1 after you have confirmed no plugin or theme setting is doing it deliberately, check for a caching layer serving an old, pre-fix version of the page’s <head> markup. Canonical tags are easy to overlook in a cache-invalidation routine that mostly focuses on visible content, so a stale cached copy can keep showing the old canonical value long after the underlying template has been corrected.

---

Most WordPress sites do not need author archive pages indexed. WordPress creates a page at /author/{username}/ automatically for every user who has published content, whether you intended search engines to see it or not, and on a single-author site that page just repeats what your blog index and category pages already show. The short fix is one toggle: open Rank Math’s Titles & Meta settings, go to the Authors tab, and turn off Author Archives. If you might add more writers later and want the option open, set the archive to noindex instead of disabling it outright – same result for search engines, less to undo if your site grows.

Why it happens

You did not build an author page. WordPress built it for you. The moment a user account publishes its first post, WordPress generates a pretty-link archive at /author/{username}/ based on that user’s login name or nicename, and it stays reachable for as long as the account has published content. There is no setting that turns this off at the WordPress core level – the page exists by default, on every install, for every author.

That default causes two separate problems. The first is content duplication. An author archive is just a list of that person’s posts, reverse-chronological, with excerpts. On a single-author blog, that list is functionally identical to your homepage or your main blog index – same posts, same order, different URL. You end up with two (or more, once you count category and tag archives) indexed pages competing to rank for the same content instead of one page doing the job.

The second problem is smaller but easy to overlook: the default author slug is built from the username, and a predictable username is one less thing an attacker has to guess during a brute-force login attempt. It is not a severe vulnerability on its own, but it is an unnecessary piece of information to leave exposed on a public URL, especially on a site with only one or two accounts where the author slug and the login name are easy to connect.

If you are running Rank Math, you already have the control for this without installing anything new. The same Titles & Meta setup most sites configure once and forget includes an Authors tab with an Author Archives toggle, and that single setting is doing double duty here: it decides both whether the archive page exists in a crawlable form and what happens when someone lands on that URL anyway.

Step-by-step fix

The fix does not require a plugin install or a code change. It is a settings change in the SEO plugin you are already running, plus one verification pass afterward.

1. Decide if you actually need author archives. Single-author sites almost never do – your homepage and category pages already cover the same ground. Multi-author sites are the exception: if each writer has a distinct beat and you want their archive to work as a mini-portfolio page in search results, keep it. Otherwise, turn it off.

2. Open Rank Math’s Titles & Meta settings and go to the Authors tab. This is a separate tab from the general post-type settings, easy to miss if you only configured posts and pages when you first set up the plugin.

3. Turn off the Author Archives toggle if you do not need the pages at all. With this off, visiting an author archive URL redirects to the homepage instead of returning the page or a 404 – the archive effectively stops existing as a separate crawlable destination.

4. Use noindex instead of disabling, if you want the pages to stay reachable. Some sites still want authors to be able to view their own archive page even if it should not compete in search. For that case, leave Author Archives enabled but turn on Robots Meta for that archive type and check No Index – the same general process Rank Math uses for noindexing any other post type or archive. The page stays live; it just stops asking to be ranked.

5. Save, then clear any full-page cache. If your host or a caching plugin is serving a cached copy of the author archive, the old version can keep showing up in a crawl or a manual check until the cache clears, even though the setting change is already live on the server.

How to verify it worked

Do not take the settings screen’s word for it – confirm what the live page actually does. Visit /author/{yourusername}/ directly in a fresh browser tab, logged out, with no cache-busting query string. If you disabled Author Archives entirely, you should land on your homepage, not a 404 and not the old archive listing. A 404 usually means the redirect has not taken effect yet, often a caching issue; the archive page still rendering means the setting was not saved or a caching layer is serving a stale copy.

If you chose the noindex route instead, view the page source of the live author archive and look for <meta name="robots" content="noindex... in the <head>. That tag needs to say noindex, not just be absent – an empty or missing robots meta tag means the page is still indexable by default. This is the same source-level check worth running any time you rely on a robots-level rule to keep a URL out of search results: the setting only counts once it shows up in what the browser actually receives.

Worth a separate look, since it is easy to conflate the two: a noindex meta tag on the author page is not the same as blocking the URL in robots.txt. Blocking crawl access in robots.txt can actually prevent Google from seeing the noindex tag at all, which sometimes leaves the URL indexed with no description. For author archives, the meta-tag route above is the one that reliably removes the page from search results without that side effect.

What to do when the standard fix fails

If the Author Archives toggle is off but the page is still showing up in search results weeks later, the setting change worked – search engines are just slow to drop a URL they have already crawled and indexed. A noindex tag or a redirect stops future indexing; it does not retroactively pull an already-indexed page out of the index on its own. Give it time, and if the URL still has not dropped after a few weeks, request removal or a recrawl through Search Console rather than assuming the WordPress-side fix did not take.

If the redirect is not happening at all – the archive URL still renders the old post listing after you have confirmed the toggle is off and the cache is cleared – the most common remaining cause is a theme or page builder that has its own author-template file overriding the default behavior. Some themes ship a custom author.php template that renders regardless of what your SEO plugin’s settings say, because the plugin’s toggle controls the meta output, not necessarily whether the page renders at all on every theme. Check your active theme’s files (or ask your theme’s support) for a custom author template before assuming the SEO plugin is misconfigured.

One more edge case: if your canonical tag on the author archive still points at the archive URL itself rather than somewhere more useful, that is a separate, smaller signal working against you even after noindex is in place – it tells any crawler that does see the page that this URL is the authoritative version of its own content, which is exactly what you are trying to avoid. Most SEO plugins handle this automatically once Author Archives is configured correctly, but it is worth a quick source check alongside the robots meta tag, especially if you made the change some other way.

---

Two active SEO plugins on the same WordPress install almost always fight over the same handful of jobs: the title tag, the meta description, the canonical URL, and the JSON-LD schema. Each plugin hooks into wp_head independently and writes its own version of these tags, so the page ends up with duplicates instead of one clean set. The fix is not a setting to toggle – it is to pick one SEO plugin, let it import the other’s data, and fully deactivate the loser. Rank Math’s own documentation names a second active SEO plugin as one of the most common causes of conflicts it sees, and Yoast publishes the same warning from its side.

Why it happens

SEO plugins do not coordinate with each other. Rank Math, Yoast, All in One SEO, and SEOPress all attach to the same WordPress hooks – wp_head chief among them – and each one assumes it is the only plugin managing meta output for that page. When two are active, both fire. The result is not one plugin overriding the other cleanly; it is two competing tags for the same purpose landing in the same <head>.

The most visible symptom is duplicate meta tags: two <title> declarations, two meta descriptions, two sets of Open Graph and Twitter Card tags. Search engines and social platforms generally pick one and ignore the other, but which one gets picked is not something you control, and the tag you did not intend to ship is still sitting in your page source.

Canonical URLs have a second wrinkle on top of that. WordPress core outputs its own canonical link via rel_canonical() in wp-includes/link-template.php, whether or not an SEO plugin is installed. If your SEO plugin sets a custom canonical but does not also suppress that core output, you can end up with two different canonical tags pointing at two different URLs on the same page – one from core, one from the plugin, and now a third if a second plugin joins in.

Structured data has the same problem from a different angle. JSON-LD schema for a single page can come from more than one source at once – an SEO plugin, the active theme, a page builder, and on a store, WooCommerce itself. When two of those sources emit the same schema type, breadcrumb schema being the most common collision, you get duplicate or contradictory structured data blocks rather than one coherent one. If you have not audited what is actually being output, the JSON-LD your site sends to search engines may not match what any single plugin’s settings screen shows you, because no single plugin is the only source.

Step-by-step fix

The fix is the same regardless of which two plugins are involved: keep exactly one SEO plugin active and remove the rest. Here is the sequence that avoids losing data in the process.

1. Decide which plugin stays. If you are standardizing on Rank Math, its Setup Wizard includes a step that detects other active SEO plugins and offers to import their titles, meta descriptions, and redirects before you deactivate anything – so you are not starting from a blank slate.

2. Run the import before you deactivate anything else. Let the surviving plugin pull in the settings from the one you are retiring. This is the point where most of the manual re-entry work gets skipped.

3. Deactivate the losing plugin, do not just disable individual modules. A plugin that is still active but has its title/description module switched off can still be hooking into canonical or schema output elsewhere. Full deactivation is what actually stops the duplicate hooks from firing.

4. Check the canonical output specifically. Confirm your surviving plugin’s canonical setting matches what you want, and that WordPress core is not also emitting a competing one. This is the step people skip because it is not visible in the plugin’s own settings screen – it only shows up in the rendered page source.

5. Re-check your sitemap. Two SEO plugins can also mean two sitemap generators pointed at the same site, and that is one of the more common ways an XML sitemap ends up stale or corrupted – one plugin’s generator stops updating while the other keeps running, or both run and search engines get pointed at two different sitemap files with different contents.

If you are setting Rank Math up from scratch rather than migrating off a second plugin, the step-by-step Rank Math configuration process covers the same setup wizard and module choices in more detail, including which modules to enable first.

How to verify it worked

Do not trust the plugin’s settings screen alone – it only shows you what that one plugin intends to output, not what the page actually renders. View the page source directly (not the block editor preview) and confirm there is exactly one <title> tag, one meta description, one canonical link, and one JSON-LD script block per schema type.

Rank Math also gives you a built-in way to check for conflicts systematically rather than guessing: deactivate every other plugin, then reactivate them one at a time (or in small batches), checking the site after each reactivation to isolate exactly which plugin reintroduces the problem. This is slower than eyeballing the source, but it is the documented method for confirming a fix actually holds rather than just looking clean on the one page you happened to check.

Worth checking separately: Yoast actively scans for other active plugins and will surface an admin warning naming a specific plugin if that plugin is known to cause Open Graph conflicts. If you are running Yoast and that warning disappears after your cleanup, that is a reasonable secondary confirmation – though the page-source check above is still the one that matters for what search engines and social platforms actually see.

What to do when the standard fix fails

Deactivating the second plugin resolves most conflicts, but not all of them. If duplicate tags or schema persist after you are down to one active SEO plugin, the remaining source is usually the theme or a page builder, not a plugin at all – both are capable of injecting their own meta output or JSON-LD independently of any SEO plugin. Check your theme’s SEO-related settings and any page-builder-level meta fields before assuming the SEO plugin itself is misconfigured.

Yoast’s own troubleshooting documentation lays out a systematic version of this for exactly this scenario: first test Yoast’s own add-ons against each other for conflicts, then test every other active plugin one by one, using either the Health Check & Troubleshooting plugin or a staging copy of the site so you are not experimenting on the live page. The same process works if you have standardized on Rank Math instead – the tool changes, the deactivate-and-isolate method does not.

If the sitemap specifically is still wrong after the plugin cleanup – missing recent posts, still pointing at the old plugin’s file, or returning a 404 – that is very rarely the SEO plugin’s fault at that point. It is almost always a caching layer serving a stale copy of the sitemap file, or a permalink/rewrite rule that needs to be flushed. Confirm what the sitemap URL actually returns right now, in a fresh request, before spending more time in either plugin’s settings.

---

IndexNow is a small protocol that lets your WordPress site tell a search engine directly when a URL changes, instead of waiting for that engine to crawl and notice on its own. On WordPress the fastest path is Rank Math’s built-in Instant Indexing module: turn it on under Rank Math SEO, and it generates the API key, hosts the key file, and submits new and updated URLs automatically. What it buys you is faster discovery on Bing, Yandex, Naver, Seznam.cz, and Yep. It does not reach Google, which has never adopted IndexNow and still relies on its own crawler, sitemaps, and Search Console.

How IndexNow actually works

Search engines normally find your new or changed pages by crawling: a bot revisits your site on its own schedule, checks your sitemap, and eventually notices what’s different. IndexNow flips that around. Instead of waiting to be discovered, your site pushes a notification the moment something changes.

The mechanism is a key-based handshake, not an account or an API subscription. You generate a key, host a plain text file containing that key at your domain root (`https://yoursite.com/{key}.txt`, or a subdirectory if you set a matching keyLocation), and every submission includes the key so the receiving engine can confirm the request actually comes from someone who controls the domain. A single-URL submission is a GET request to the engine’s /indexnow endpoint with url and key parameters; a batch submission is a POST with a JSON body carrying host, key, keyLocation, and a urlList of up to 10,000 URLs. The engine replies with a status code: 200 for a clean submission, 202 if the URL was received but key validation is still pending, 400 for a malformed request, 403 for a bad or missing key, 422 if the URLs don’t match the host or key you submitted, and 429 if you’re sending too many requests too fast.

Whichever engine you submit to, the protocol shares the notification with every other participating engine, so one submission covers the whole group. That group is currently Microsoft Bing, Yandex, Naver, Seznam.cz, and Yep. Getting that key file hosted correctly assumes your site actually serves plain-text files to crawlers without interference, which is the same access a correctly configured robots.txt is meant to guarantee – if something is blocking bots at the root level, the key file check will fail the same way any other crawl would.

Step-by-step setup

You don’t need a Bing Webmaster Tools account or a manually generated key for the common case. If you’re running Rank Math, which is what this site uses, the module handles the entire handshake:

1. Enable the module. Go to Rank Math SEO in your WordPress dashboard and turn on the Instant Indexing module.

2. Open its settings. Navigate to Rank Math SEO -> Instant Indexing -> Settings. Confirm the API key field shows a generated key – Rank Math creates and hosts this automatically, so there’s no file to upload by hand.

3. Choose which post types submit automatically. The settings screen lets you pick which content types trigger a submission on publish or update. Rank Math excludes anything marked noindex by default, so you don’t have to separately filter those out.

4. Save. From this point forward, publishing or editing a covered post type submits the URL automatically in the background. No extra step per post.

If you’re not on Rank Math, the official IndexNow Plugin (built by the Bing Webmaster team, free, and open source) does the same job standalone: install it, activate it, and it generates and hosts its own key. Deactivating and reactivating the plugin rotates the key, which is also the documented way to reset it if you ever need to.

For content you’re publishing through an automated pipeline rather than the block editor – which is how this site ships most of its posts – the same submission can happen over HTTP right alongside whatever else your automation is already doing through the WordPress REST API. There’s no need to build a separate integration; it’s one more POST request in the same publish step.

How to verify it worked

Rank Math gives you two direct ways to confirm a submission actually happened, beyond just trusting the toggle. First, the manual path: from the Posts screen, hover over any post and you’ll see an “Instant Indexing: Submit Page” row action – use it on a post you just published and watch for a success confirmation instead of an error. Second, select several posts and use the “Instant Indexing: Submit Pages” bulk action the same way, useful right after a setup change when you want to push your existing catalog rather than waiting for the next edit to trigger it.

Beyond the plugin’s own UI, check the response codes if you’re submitting manually or through a script: 200 or 202 means the engine accepted the request, 403 means your key file isn’t being found or doesn’t match, and 422 means the URL you submitted doesn’t belong to the host tied to that key. A 403 almost always traces back to the key file – fetch https://yoursite.com/{key}.txt directly in a browser and confirm it returns the exact key as plain text, not a 404 page or a redirect.

What you won’t see is any signal in Google Search Console, because Google isn’t part of this handshake at all. If your mental model of “did indexing work” is built around GSC’s coverage report, IndexNow submissions won’t show up there – that’s expected, not a failure.

What it doesn’t fix, and what to use instead

This is the part that’s easy to miss: IndexNow does not reach Google. Google evaluated the protocol after its 2021 launch and never adopted it for web search. It still relies entirely on its own crawler, your XML sitemap, and the tools built into Search Console. If your traffic – and most sites’ organic traffic – comes overwhelmingly from Google, IndexNow setup buys you faster indexing on a set of engines that aren’t where most of your readers are searching, and it buys you nothing on the one that is.

That’s not a reason to skip it. Bing, Yandex, and the smaller engines in the group still send real traffic, and instant submission is close to free once it’s configured. But treat it as an addition to your Google indexing workflow, not a replacement for it. For Google specifically, the correct tools are still your sitemap, Search Console’s URL Inspection “Request indexing” action, and – if you’re automating at any scale – the Search Console API for checking indexing status programmatically rather than clicking through the UI one URL at a time. If a page still isn’t indexed in Google days after publishing, the fix lives in that toolkit, not in IndexNow’s dashboard, no matter how clean your IndexNow submissions look.

If IndexNow submissions themselves start failing after previously working, check for the same root causes any indexing signal problem has: a plugin change that regenerated the key without updating the hosted file, a caching layer serving a stale version of the key file, or a robots rule that started blocking the file path. It’s the same category of problem as watching a sitemap stop updating even though the plugin generating it looks configured correctly – the fix is confirming the file the engine actually fetches, not just trusting the setting that’s supposed to produce it.

Frequently asked questions

Do I need a Bing Webmaster Tools account to use IndexNow?

No. The IndexNow Plugin and Rank Math’s Instant Indexing module both generate and host the required key automatically, so submissions work without registering or verifying your site with Bing or any other engine first.

Will IndexNow help my Google rankings?

No. Google does not participate in IndexNow. Faster indexing on Bing, Yandex, Naver, Seznam.cz, or Yep has no bearing on how quickly Google crawls or ranks the same page.

Can I use both Rank Math’s Instant Indexing and the official IndexNow Plugin at the same time?

Running two IndexNow integrations side by side is redundant rather than additive – each generates and hosts its own key file, and only one is needed for the protocol to work. Pick one, most commonly whichever SEO plugin you already run, and leave the other deactivated.

What happens if I regenerate my IndexNow key?

Any previously issued key stops validating once a new one replaces it at the hosted key file location. That’s expected and not something to do casually – regenerate only if you suspect the key has been exposed or reused somewhere it shouldn’t be.

---

WP-Cron is not a real cron job. It is a pseudo-scheduler that only checks for due tasks when someone loads a page on your site, which means it can silently stop firing on a low-traffic site or a fully cached one and nothing in the admin will tell you. The fix is to add define('DISABLE_WP_CRON', true); to wp-config.php and point a real system cron job or scheduled task at wp-cron.php on a fixed interval instead. Verify it worked with wp cron test and wp cron event list, not by assuming the fix took effect.

Why it happens

WordPress core does not run a background daemon. Instead, every page load calls wp_cron(), which checks the scheduled-events table stored in the cron option and, if anything is due, calls spawn_cron() to kick off a run. That call is guarded by a transient lock – the doing_cron option, held for WP_CRON_LOCK_TIMEOUT seconds (60 by default) – so two visitors hitting the site at the same moment don’t trigger two overlapping cron runs.

The part that actually causes the failures we care about is how that trigger fires. spawn_cron() sends a non-blocking HTTP request, via wp_remote_post(), to wp-cron.php with a very short timeout and blocking set to false. The page that triggered it doesn’t wait around to find out whether the cron run succeeded, or whether it ran at all. That’s a deliberate design choice – it keeps a visitor’s page load fast – but it also means a broken cron trigger fails silently. No error in the admin, no email, nothing. It just stops running, and unless someone visits infrequently enough or checks directly, you won’t notice.

This matters more than it sounds like on a site that leans on scheduled background work for SEO tasks. We use WP-Cron indirectly every time we automate Rank Math field updates with an agent on a schedule, and the same dependency shows up anywhere a plugin queues sitemap regeneration, rank tracking, or scheduled publishing behind a cron hook. If the page-load trigger stalls – because a page cache is serving every request from disk without touching PHP, or because traffic on a given route just dries up – every one of those scheduled jobs stalls with it, quietly.

Step-by-step fix

The fix has two parts: stop relying on page loads to trigger cron, and start triggering it from something that actually runs on a schedule regardless of traffic.

1. Disable the page-load trigger. Add this line to wp-config.php, above the line that says “That’s all, stop editing”:

define('DISABLE_WP_CRON', true);

This stops wp_cron() from firing its non-blocking request on every page view. It does not clear or disable the scheduled events themselves – they stay registered in the cron option and will run the moment something else calls wp-cron.php directly.

2. Point a real scheduler at wp-cron.php. On Linux or macOS, a crontab entry that hits the endpoint once a minute (or whatever interval matches your tightest scheduled task) looks like this:

0 0 * * * wget --delete-after http://YOUR_SITE_URL/wp-cron.php

On Windows, the equivalent is a scheduled task running a PowerShell command:

powershell "Invoke-WebRequest http://YOUR_SITE_URL/wp-cron.php"

Either way, you’re replacing an unreliable, traffic-dependent trigger with one that fires on a fixed interval no matter how many visitors the site gets. This pairs naturally with any other scheduled automation already hitting the site’s endpoints – if you’re already calling into the WordPress REST API for SEO automation, adding one more scheduled HTTP call to the same infrastructure is a small addition, not a new dependency.

3. Know that ALTERNATE_WP_CRON is not a substitute. Some hosts that block loopback HTTP requests suggest setting ALTERNATE_WP_CRON instead. That’s a redirect-based trigger that still depends on page loads – it works around a specific hosting restriction, but it does not give you a true, traffic-independent scheduler the way a real cron job does.

How to verify it worked

Don’t assume the fix took. WP-CLI has a built-in diagnostic for exactly this: wp cron test. It checks whether DISABLE_WP_CRON is set and errors if the page trigger is still disabled without a replacement in place, warns if ALTERNATE_WP_CRON is active, and attempts to spawn WP-Cron over HTTP itself, warning on anything other than a 200 response. On a healthy setup it reports “Success: WP-Cron spawning is working as expected.”

Follow that with wp cron event list, which shows every registered hook along with its next scheduled run time and recurrence. If a hook’s next-run time is in the past and staying there across repeated checks, the event is registered but not actually firing – which is the exact failure mode a broken trigger produces.

This is the same verification discipline we apply to drift monitoring on this site: drift checks are themselves a scheduled job, and if WP-Cron stalls, the monitoring meant to catch problems stalls right along with it. Checking wp cron event list after any server or caching change is a five-second habit that catches this before it becomes a multi-day blind spot.

What to do when the standard fix fails

If you’ve set DISABLE_WP_CRON, wired up a real cron job, and wp cron test still isn’t clean, the trigger mechanism usually isn’t the problem anymore – the scheduling code registering the events is. The most common cause is a plugin or custom code calling wp_schedule_event( $timestamp, $recurrence, $hook, $args ) with an $args array that changes slightly between calls. wp_next_scheduled() is supposed to prevent duplicate registrations, but it only matches on the exact same $hook and $args pair used originally – if $args drifts even slightly, the duplicate check fails silently, and you end up with the same event registered multiple times, bloating the cron option and multiplying background load.

wp cron event list will show you this directly: look for the same hook name appearing more than once with different next-run times. From there, wp cron event run --due-now lets you force-execute everything currently due, which is useful for confirming a specific hook actually works once triggered, separate from whether it’s being triggered on schedule at all.

This is the same category of failure as a stale, stuck background job anywhere else on the stack – it looks a lot like what happens when a sitemap stops updating even though the plugin generating it appears configured correctly. In both cases the fix isn’t re-checking the obvious setting again; it’s confirming the underlying scheduled job is actually running, not just registered.

Frequently asked questions

Does DISABLE_WP_CRON stop scheduled events from running entirely?

No. It only stops the automatic page-load trigger. Scheduled events stay registered in the cron option and will still run the moment something else – a real cron job, WP-CLI, or a direct request – hits wp-cron.php.

What if my host doesn’t allow real cron jobs?

ALTERNATE_WP_CRON is the documented fallback for hosts that block loopback HTTP requests, but it’s still page-load-dependent and carries the same silent-failure risk as the default trigger. If your host offers any scheduled-task feature at all – even one meant for other purposes – pointing it at wp-cron.php is more reliable than staying on the page-load trigger.

How often should the real cron job hit wp-cron.php?

Match it to your tightest scheduled interval. If nothing on the site runs more often than hourly, once every 15-30 minutes is more than enough; there’s no benefit to polling far more often than your most frequent scheduled task actually needs.

Can I use WP-CLI’s wp cron event run instead of a system cron job?

wp cron event run --due-now executes due events on demand, but it has to be triggered by something – it is not itself a scheduler. It’s the right tool for manually forcing a run or testing that a hook works, not a replacement for a real cron job or scheduled task calling wp-cron.php on an interval.

---

Multisite SEO problems are not bugs to patch. They come from the architecture: every subsite on a network shares one database, one set of plugins, and often one domain root, so anything that depends on a single, unique file or a single, unique canonical URL – robots.txt, XML sitemaps, boilerplate pages – has to be handled deliberately per subsite. Fix it by setting a self-referencing (or cross-domain) canonical on every subsite, confirming your SEO plugin’s robots.txt and sitemap behavior for your specific network type, and treating your subdomain-vs-subdirectory choice as permanent from day one.

Why it happens

A WordPress Multisite network is one WordPress install pretending to be several sites. Every subsite draws from the same database, the same core files, and the same wp-content directory. Themes and plugins are installed once and made available network-wide; a Super Admin can activate a plugin for every subsite at once, and individual site admins cannot always deactivate it. There is no code-level wall between subsites – a plugin conflict or a misconfigured setting on one subsite is a network-wide condition, not a contained one.

That shared architecture is exactly why SEO tooling behaves differently on multisite. Robots.txt is a good example: a domain can only ever serve one robots.txt file, at its own root. On a subdirectory-structured network, every subsite shares the main site’s domain root, so there is no way for a subdirectory subsite to have its own independent robots.txt no matter which SEO plugin you run. Rank Math, which this site uses, is explicit about this: robots.txt can only be edited on the network’s main site, and every other subsite simply inherits those settings, because subsites do not have an actual file structure to support a separate physical file. Rank Math manages this through a filter rather than writing a file to disk, and it will automatically disable its own robots.txt editor if it detects a real file already sitting on the server, so it does not silently overwrite something you or your host put there on purpose.

Sitemaps run into a related version of the same problem. Yoast SEO has a long-standing, still-open issue where XML sitemaps can mix up content between subsites on a subdirectory network. Yoast ships a filter to change that behavior, but the filter’s effect is itself gated by a check for whether the install is subdomain- or subdirectory-structured, and any attempt to aggregate sitemap data across a very large network is capped well below what a big network actually contains. None of this is a defect specific to one plugin. It is what happens when tooling built around “one site, one root” assumptions meets an architecture where several sites share a root.

Duplicate content is the third recurring issue, and it is the most avoidable. Boilerplate pages, syndicated posts, or a shared template repeated across subsites read to a search engine exactly like duplicate content on any single site, because from the search engine’s point of view, that is exactly what it is.

Step-by-step fix

1. Decide subdomain or subdirectory before you launch, not after. Subdomain subsites (site1.example.com) each get their own domain root and typically need a wildcard DNS record. Subdirectory subsites (example.com/site1) sit under one existing domain and need no extra DNS work, but they inherit the main site’s root-level files, including robots.txt. Neither structure is universally better; the right one depends on how closely related your subsites are. What matters for SEO specifically is that switching later means editing SUBDOMAIN_INSTALL in wp-config.php, updating every site URL, and running a database search-and-replace across tables including wp_options, wp_posts, wp_postmeta, wp_blogs, and wp_site. Treat the choice as permanent.

2. Confirm your SEO plugin’s robots.txt behavior for your structure. If you are subdirectory-structured, verify in your plugin’s own documentation that only the main site can edit robots.txt and that every subsite correctly inherits it – do not assume a per-subsite robots.txt editor screen actually does anything on a subdirectory network. Our guide to WordPress robots.txt covers the file’s rules in more depth outside the multisite case.

3. Check your sitemap settings per subsite, not just at the network level. If you run Yoast on a subdirectory network, look specifically at whether the subdirectory-sitemap filter needs to be set, and do not assume a single network-wide sitemap correctly represents every subsite once the network passes a few hundred sites. If a sitemap looks wrong or frozen and you cannot immediately tell why, our post on a sitemap that stops updating covers the general stale-cache failure mode, which can compound with the multisite-specific mixing issue above.

4. Put a canonical tag on every piece of shared or boilerplate content. If two subsites legitimately need the same page (a shared privacy policy, a syndicated post), pick one as the source and set a cross-domain canonical on the copy pointing back to it. If a page is unique to its own subsite, its canonical should simply point at itself. Our guide on WordPress canonical URLs walks through what a correct tag looks like in either case.

5. Audit which plugins are network-activated versus site-specific. A network-activated SEO or caching plugin runs identically on every subsite whether or not that makes sense for the subsite’s content. If one subsite has genuinely different SEO needs, that is a sign it may need site-specific plugin activation rather than relying on the network default.

6. If you’re running more than one SEO-managed site, keep their configurations from drifting apart silently. Multisite consolidates hosting, but the settings-drift problem it’s solving for is the same one you hit running several separate WordPress installs; see our notes on running an SEO agent across multiple sites for how to keep per-site configuration intentional instead of accidental.

How to verify it worked

Visit each subsite’s robots.txt URL directly (not the network admin’s) and confirm it matches what your plugin’s documentation says it should show for that subsite’s role in the network – main site or inherited. Then check each subsite’s XML sitemap and confirm it lists only that subsite’s own URLs, not a mix of URLs from other subsites on the network. Finally, spot-check the canonical tag on any shared or boilerplate page across subsites in view-source, confirming it points where you intend, not at whichever subsite happened to load the shared template first.

What to do when the standard fix fails

If a subdirectory subsite’s robots.txt still doesn’t match the main site after you’ve confirmed plugin settings, check for a physical robots.txt file sitting on the server outside the plugin’s control – some hosts or migration processes leave one behind, and most SEO plugins will defer to a real file over their own filter-based version without telling you.

If sitemap mixing persists after applying your plugin’s subdirectory-specific filter, check your network’s total site count. Aggregation caps on large networks mean the fix that works for a 20-site network can silently stop working once you cross into the hundreds.

If a canonical tag on shared content keeps reverting or pointing at the wrong subsite, look for a caching layer – object cache, page cache, or CDN – serving a stale version of the shared template to some subsites and not others. Purge every layer and re-check from a private browser window before concluding the canonical logic itself is wrong.

If none of the above resolves it and the symptom is specifically Google treating two subsites’ content as duplicates in Search Console, re-verify with a fresh crawl after fixing the canonical, since Search Console’s own duplicate-content reporting lags behind a live fix by design.

Frequently asked questions

Does every subsite need its own SEO plugin license?

That depends on the plugin’s own licensing model, not on WordPress. What matters for SEO mechanics is that a network-activated plugin’s behavior – including robots.txt and sitemap handling – applies the same way across every subsite regardless of licensing.

Can I just block duplicate subsites with robots.txt instead of setting canonicals?

Not on a subdirectory network – only the main site’s robots.txt can be edited at all, so you cannot selectively block individual subdirectory subsites that way. A canonical tag is the correct tool for duplicate or shared content regardless of network structure.

Is subdomain multisite easier for SEO than subdirectory?

Not inherently. Subdomain subsites can each carry their own robots.txt in principle, which removes one specific limitation, but they take on independent DNS management and, in some cases, are treated more like separate domains by search engines, which has its own tradeoffs depending on how related the subsites are.

Will migrating from separate WordPress installs to multisite hurt my rankings?

Multisite doesn’t change your existing URLs by itself if you migrate carefully, but a change to your domain structure – moving a separate site into a subdirectory of a new multisite network – is a URL change, and needs the same 301-redirect discipline as any other URL migration.

---

The short fix: open Settings > Permalinks and click Save Changes to flush WordPress’s rewrite rules, then set up 301 redirects from every old URL pattern to the new one before you tell anyone the URLs have changed. Skipping the redirect step is the most common way a permalink change quietly loses rankings, because an old link can keep loading without ever showing a clean error, which hides the problem from both visitors and Google.

Why it happens

WordPress gives you six permalink formats: Plain, Day and name, Month and name, Numeric, Post name, and a Custom Structure you build from structure tags like %year%, %monthnum%, %day%, %category%, %author%, %post_id%, and %postname%. The official documentation is specific about the one rule that matters: “End your structure with either %post_id% or %postname% so that WordPress can target an individual post.” Miss that, and WordPress has no reliable way to resolve the URL back to a single post.

Most sites settle on Post name, or a custom structure ending in %postname%, because it reads better than the default Plain structure’s ?p=123 query string. Changing that setting looks harmless. What actually happens underneath depends on your server. On Apache, saving the Permalinks screen automatically rewrites your .htaccess file with new rewrite rules, so the new structure works immediately. On Nginx, there is no .htaccess to rewrite at all; Nginx reads its own server block configuration, so a permalink change that works fine on Apache can 404 everywhere on Nginx until someone edits that server config by hand.

The part that actually costs you traffic is what happens to the URLs you already had. WordPress does not generate redirects for you when you change the structure. The reported failure mode is not even a clean 404: an old, now-invalid URL can silently render your homepage instead of an error page. A visitor does not notice anything obviously broken, and neither does Googlebot at first glance. Over weeks, search engines stop seeing a live version of that specific page and quietly drop it from the index, because nothing ever told them the content moved.

Step-by-step fix

1. Pick a structure that ends correctly. Use Post name, or a custom structure ending in %postname% or %post_id%. Avoid Plain unless you have a specific reason to keep it.

2. Save the Permalinks screen, even with no changes. Go to Settings > Permalinks and click Save Changes. This forces WordPress to flush and regenerate its rewrite rules. On the command line, the equivalent is wp rewrite flush. If you are calling flush_rewrite_rules() from your own plugin or theme code, only do it on an activation hook, never on every page load; it is a real performance cost when it runs on every request.

3. If you’re on Nginx, edit the server block by hand. The Permalinks screen save only touches .htaccess. Nginx needs its rewrite location block updated to match the new structure, then a config reload.

4. Check for a missing trailing slash. A custom structure that drops the trailing slash before %postname% parses differently than one that includes it, and can create its own redirect loops or 404s independent of the rest of this fix.

5. Set up 301 redirects from every old URL pattern to the new one. A 301 tells both browsers and search engines the move is permanent, and it is the step that actually prevents the ranking loss. You can do this with a redirect-manager plugin, such as the Redirection plugin’s source URL / target URL mapping, which also logs any 404s it sees afterward so you can catch redirects you missed. Or you can add manual Redirect 301 lines to .htaccess on Apache. Either way, map the pattern, not just a handful of individual URLs, if your old and new structures differ by a whole segment (for example, dropping a date prefix).

6. If you use custom post types, confirm they’re registered as public. A custom post type needs 'public' => true and 'publicly_queryable' => true in its registration for WordPress to generate rewrite rules for it at all. Without that, its URLs won’t resolve correctly no matter which permalink structure you pick.

7. Purge caches. A page cache or CDN edge cache can keep serving a stale 404, or stale old-URL content, even after the redirect is live on the server. Purge before you test.

How to verify it worked

Open a private or incognito browser window, not a logged-in session, since a cached login can mask an error that a real visitor would see. Visit a sample of your old URLs directly and confirm each one returns an HTTP 301 and lands on the correct new URL, not the homepage and not a 404.

While you’re checking URLs, it’s worth confirming the canonical tag on the new URL points to itself and not to the old pattern; see our guide on WordPress canonical URLs for what a correct one looks like. Then check that your XML sitemap has actually updated to the new URLs rather than still listing the old ones, which is a separate and surprisingly common failure covered in our post on a sitemap that stops updating. Once both check out, fold a permalink-structure spot-check into your standing routine; it’s one line item in our broader WordPress SEO checklist.

What to do when the standard fix fails

If old URLs are still broken after you’ve flushed rewrite rules and added redirects, work through these in order.

Redirects got overwritten. Some SEO or caching plugins store their own redirect rules and can silently override or duplicate what you set up in a different plugin. Check whether more than one plugin on the site manages redirects, and consolidate to one.

You’re on Nginx and only touched the WordPress side. The Permalinks screen save does nothing for Nginx. If your host manages the server config, ask them to confirm the rewrite location block was actually updated and reloaded.

A custom post type still 404s. Re-check its registration for 'public' => true and 'publicly_queryable' => true, then flush rewrite rules again after fixing the registration, not before.

Google Search Console keeps showing old 404s or “not indexed” for pages that should be live. That’s frequently a symptom of a page that was accidentally left set to noindex during the migration, not a permalink problem at all; our post on WordPress noindex mistakes walks through how to find those.

Everything above checks out but the old URL still loads something. Purge every layer of cache again, including any CDN, and re-test from a private window. Caching is the most common reason a fix that is genuinely live on the server doesn’t appear to work yet.

Frequently asked questions

Does changing permalinks hurt SEO?

Changing the structure itself does not. Leaving the old URLs unredirected does. The ranking loss comes from the missing 301s, not from picking a new structure.

Do I need a redirect for every single URL?

If your old and new structures differ by a predictable pattern, such as removing a date prefix, one pattern-based redirect rule can cover every affected URL. You only need one-off redirects for exceptions that don’t fit the pattern.

Is Post name really the best structure?

It’s the generally recommended one because it’s readable and ends in %postname% as WordPress’s own documentation asks. A custom structure ending the same way works just as well; there’s no single structure that’s SEO-superior beyond following that rule.

What about WooCommerce product URLs?

WooCommerce has its own product-permalink settings layered on top of the general ones, which is outside the scope of this fix. Check Settings > Permalinks for a separate product permalink section if you run a store.

---

Most block theme SEO complaints trace back to two specific mistakes. The first is a duplicate H1: the Site Title block and the Post Title block both rendering as h1 on the same page. The second is an archive template whose Query Loop runs a custom query instead of inheriting from the URL, which quietly serves the wrong set of posts under a category or tag link. Open the Site Editor’s document outline, fix whichever block is wrongly set to h1, and switch archive templates back to “Inherit query from URL.” Both fixes take a few minutes once you know where to look.

Why it happens

The Gutenberg Site Title block renders as an h1 unconditionally, on every page type, not only the homepage. That behavior was flagged directly against WordPress core as a defect: a 2019 Gutenberg issue documented the Site Title block outputting h1 everywhere and requested conditional rendering instead, an h1 on the front page and a plain element elsewhere, matching how classic themes like Twenty Twenty already behaved. The issue was closed once a fix merged, but the underlying template pattern still reappears whenever someone manually sets both the Site Title block and the Post Title block (or a stray Heading block) to h1 in the same template. Block themes make this easy to do by accident, because heading levels are a per-block setting with no template-wide validation stopping you.

The Heading block adds a second half to the trap. By default, Gutenberg’s core Heading block does not offer h1 as the first suggested level, on the assumption that the Post Title or Page Title block already supplies the page’s one h1. Remove or hide the Post Title block from a template, and a page can end up with zero h1 elements. Leave a manually set h1 Heading block in place next to an intact Post Title block, and the page gets two. Neither failure is visible in the editor unless you check the document outline, because both render as normal-looking bold text on screen.

A third source sits in the template files themselves. A block theme only requires one template, index.html, and WordPress resolves every other template by matching the current URL against the template hierarchy: a user-edited template stored in the database first, then a child theme’s /templates folder, then the parent theme’s /templates folder. Archive-type templates (category, tag, author, date, and the generic archive template) commonly ship with a Query Loop block set to inherit the page’s query automatically. Swap that setting for a custom query, even to “improve” the layout, and the template can start showing content that does not match what the URL and title imply, which is the same duplicate-content and canonical mismatch problem covered in more detail in our canonical URL guide.

Step-by-step fix

1. Open the template in the Site Editor. Go to Appearance → Editor → Templates, and open the specific template with the problem: the front page, a single post template, or an archive template.

2. Check the document outline. Click the list-view or outline icon in the top toolbar. It shows the current heading structure top to bottom and flags skipped levels, which is the fastest way to spot a missing or duplicate h1 without reading raw HTML.

3. Select the offending block. If two blocks show as h1, decide which one should keep that level. On a single post template, the Post Title block should almost always stay h1; the Site Title block should not be.

4. Change the heading level. With a Heading block selected, use the “HTML element” dropdown in the block’s settings sidebar (or the heading-level control in the block toolbar) to move it from h1 to h2, or the reverse if the page currently has zero h1 elements.

5. Fix archive templates separately. Open each archive-type template, select the Query Loop block, and confirm “Inherit query from URL” is enabled in the block’s settings sidebar rather than a custom query source.

6. Save and preview before publishing the template change. Site Editor changes to templates apply site-wide immediately on save, so preview a live example of each affected page type first.

How to verify it worked

Reopen the document outline for each template you changed and confirm exactly one h1 appears, in the position you intended. Then check a live, published page directly: open it in a browser, use “View Page Source,” and search for <h1. You should find exactly one match. This is a more reliable check than trusting the editor preview alone, since some SEO plugins run their own on-page analysis that can miss template-level duplication if the plugin’s checker only scans the content field, not the full rendered template. If your workflow already includes a broader pre-publish check, our WordPress SEO checklist is a reasonable place to fold this heading check in as a standing step.

For the archive-template fix, confirm the category or tag page now lists the posts you expect, in the order you expect, and that the page title and the visible content agree with each other. If you also run structured data on these templates, re-validate it after the change; heading and query fixes do not usually touch schema output, but it costs little to confirm, and our schema markup guide covers how to check that separately.

What to do when the standard fix fails

If the document outline still shows a duplicate after you changed the heading level and saved, check whether a child theme is overriding the template you just edited. Because WordPress resolves templates through a fixed precedence order (database-stored user edit, then child theme, then parent theme), an edit made in the Site Editor can be shadowed by a template file still present in an active child theme’s /templates folder. Compare the two and remove or update whichever one is stale.

If the fix looks correct in the editor but the live page still shows the old structure, a caching plugin or a CDN edge cache is very likely serving a stored copy of the old template output. Purge the page cache, then the site-wide cache, and re-check with a private browser window to rule out your own browser cache.

If an archive template still shows the wrong posts after switching the Query Loop back to “Inherit query from URL,” check for a second, nested Query Loop block lower in the same template, or a plugin that filters the main query via code. Both can override the block setting you just changed. If the archive itself should not be indexed at all for a given taxonomy, that is a separate decision from fixing the query, and it belongs with your broader indexing rules rather than a template patch; see our guide to noindex mistakes for how to make that call correctly.

Frequently asked questions

Does a duplicate H1 directly hurt search rankings?

There is no verified, direct ranking penalty tied to heading structure alone. The practical risk is indirect: a confused heading structure makes it harder for search engines and assistive technology to determine what the page is actually about, and it often signals a template-level bug that affects every page using that template, not just one post.

Should every page have exactly one H1?

That is the accessible, well-supported convention, and it is what block themes assume when they hide h1 as a default option on the Heading block. Multiple sections with their own document outlines, such as an embedded widget, are a narrow exception, not the norm for standard post and page templates.

Can I just delete the Post Title block instead of changing its heading level?

Deleting it removes the page’s intended h1 entirely, which trades a duplicate-heading problem for a missing-heading problem. Change the heading level on the block that should not be h1, rather than removing a title block outright.

Do these two traps apply to classic PHP themes too?

The Query Loop trap is specific to block themes, since classic themes do not use the Query Loop block. The heading-duplication trap can occur in classic themes as well, if a theme’s header template hard-codes the site name as h1 on every page instead of only the front page, but block themes make the mistake easier to introduce because heading levels are set per block with no template-wide check.

---

A WordPress robots.txt only supports four directives that Google actually reads: user-agent, disallow, allow, and sitemap. Everything else, including a noindex line, is silently ignored. Most robots.txt problems on WordPress sites come from either a stray physical file overriding WordPress’s own output, or a disallow rule that blocks something Google needs to render or index the page correctly. Check which file is actually live, fix the specific rule causing the problem, and confirm the change with Google’s own live-URL test rather than assuming it worked.

Why it happens

WordPress does not ship with a physical robots.txt file. Instead, core’s do_robots() function generates one on the fly whenever a request hits the robots.txt URL and no real file exists in the site’s root directory. The default virtual output is User-agent: *, Disallow: /wp-admin/, and Allow: /wp-admin/admin-ajax.php. That admin-ajax.php exception exists because a blanket /wp-admin/ block was interfering with front-end AJAX requests that themes and plugins depend on; it was added to core through a dedicated ticket after developers ran into exactly that problem.

The whole virtual output also passes through a robots_txt filter before WordPress echoes it, so an SEO plugin like Rank Math can rewrite it without ever touching a file on disk. That is also where the first common failure comes from: if a physical robots.txt file already exists in the root directory, WordPress’s virtual generator never runs at all. The web server just serves the static file. Rank Math’s own robots.txt editor documents this directly, because it is a support question they get often: their editor writes to the virtual output, so a leftover physical file silently overrides every change made inside WordPress until that file is edited over FTP or deleted.

The second common failure is what gets blocked. A rule that was reasonable advice a decade ago, like disallowing /wp-content/ or /wp-includes/ to save “crawl budget,” now actively hurts a site, because Google renders pages using their CSS and JavaScript to judge layout and content. Blocking those directories can make Googlebot see a broken, unstyled version of the page. A narrower but just as damaging version of the same mistake is disallowing /wp-content/uploads/ specifically, which removes every image in the media library from Google Image Search even though the post pages around them stay indexable. A third pattern shows up on sites that recently changed how they handle crawler traffic: someone edits robots.txt to block an AI crawler (a legitimate and increasingly common reason to touch the file, and one covered in more depth in this site’s guide to blocking AI crawlers), and a typo in the user-agent line or path ends up blocking Googlebot along with it.

Step-by-step fix

1. Confirm what is actually live. Visit https://yoursite.com/robots.txt directly in a browser. Do not trust a cached copy or a plugin’s settings screen; the browser request shows exactly what a crawler receives right now.

2. Check for a physical file first. Connect over FTP or your host’s file manager and look in the site’s root directory for a file literally named robots.txt. If one exists, everything you change inside WordPress or an SEO plugin is being ignored in favor of that file, because do_robots() never runs while a physical file is present.

3. Edit the file that is actually in control. If a physical file exists and you want WordPress or Rank Math to manage it going forward, delete or rename that file so WordPress’s virtual generator (and any plugin filter attached to it) takes over. If no physical file exists, edit the content through Rank Math SEO → General Settings → Edit robots.txt, which requires Advanced Mode to be turned on since the editor is hidden in Easy Mode by default.

4. Remove any rule broader than it needs to be. Replace a blanket Disallow: /wp-content/ or Disallow: /wp-includes/ with nothing at all unless there is a specific subfolder you actually need hidden, and never disallow /wp-content/uploads/ if you want those images to show up in Google Image Search.

5. Delete any noindex line inside robots.txt. Google stopped supporting an unsupported noindex: directive inside robots.txt years ago, and it never worked reliably before that either. If the goal is to keep a specific page out of Google’s index, that has to be a noindex meta tag or an X-Robots-Tag HTTP header on the page itself, not a line in this file.

How to verify it worked

Reload https://yoursite.com/robots.txt in an incognito window after making changes and read the raw text, not a rendered preview. Confirm the disallow rules you removed are actually gone and that a Sitemap: line still points at your real sitemap, since a robots.txt block is one of the more common reasons a submitted sitemap stops getting crawled the way it should (a failure mode this site has covered separately in its guide to a WordPress sitemap that stops updating).

Then run the file through Google Search Console’s robots.txt testing tool, or use the URL Inspection tool’s “Test Live URL” feature on a page you expect to be crawlable, and confirm Google’s own reading of the rule matches what you intended. Do not rely on assumption or on how the rule reads to you; robots.txt matching is case-sensitive and wildcard behavior is easy to get subtly wrong.

What to do when the standard fix fails

If the live file still shows the old rules after you have edited it, look for caching in front of the request. Some hosts and CDNs cache the robots.txt response separately from the rest of the site, so a plugin-level or file-level change does not show up until that specific cache entry is purged.

If Search Console still reports pages blocked that you believe should be crawlable, re-check the exact path and user-agent match rather than assuming the rule is wrong. A rule scoped to User-agent: * applies to every crawler that does not have a more specific block further down the file, and a trailing or missing slash on a disallowed path changes what it matches.

If the site is also managing an llms.txt file for AI crawlers alongside robots.txt, keep the two consistent rather than treating them as unrelated; a mismatch between what robots.txt allows and what an llms.txt file on WordPress promises to AI crawlers is a newer version of the same “which file is actually in control” confusion that causes most robots.txt problems in the first place.

Frequently asked questions

Can I use robots.txt to remove a page from Google’s index?

No. Robots.txt controls crawling, not indexing. If a page is blocked by robots.txt, Google may still index the URL and show it in search results without a description, because it was never able to crawl the page to see any noindex signal. Removing a page from the index is a job for a noindex meta tag or header, which only works on pages Google can actually reach; this site’s separate guide to common WordPress noindex mistakes covers that side of the problem in detail.

Does WordPress create a real robots.txt file by default?

No. WordPress generates a virtual one on request through the do_robots() function whenever no physical file exists in the root directory. If you never created one yourself and no plugin created one for you, there is no actual file on the server; the URL just resolves to generated output.

Will an SEO plugin’s robots.txt editor work if a physical file already exists?

No. A physical file in the root directory is served directly by the web server and takes priority over anything WordPress or a plugin would otherwise generate. The physical file has to be edited directly or removed before a plugin’s built-in editor has any effect.

Is it safe to block /wp-admin/ entirely?

Not entirely, and WordPress does not do this by default. The default rule blocks /wp-admin/ but explicitly allows /wp-admin/admin-ajax.php, because front-end features on many themes and plugins depend on that file being reachable by crawlers as well as visitors.

---

If a WordPress page or your whole site is missing from Google, the fastest fix is Settings → Reading and unchecking “Discourage search engines from indexing this site.” That single option, backed by the blog_public database value, forces a site-wide noindex tag and overrides any per-post SEO plugin setting. If that box is already unchecked, the next place to check is the individual post’s SEO plugin panel, not your robots.txt file, since a robots.txt block actually stops Google from ever seeing a noindex tag at all.

Why it happens: WordPress has two separate places to say “don’t index this”

WordPress controls indexing through a core function called wp_robots_noindex(). It checks one option, blog_public, and if that option is off, it adds noindex to the page’s robots meta tag through the wp_robots filter. That single option is the database value sitting behind the checkbox in Settings → Reading labeled “Discourage search engines from indexing this site.” Nothing exotic is happening under the hood; it’s one option controlling one filter.

The confusion comes from how that setting used to work. Before WordPress 5.3, checking that box made the virtual robots.txt file output Disallow: /, blocking crawlers outright. That approach had a real flaw: a robots.txt block stops a crawler from fetching the page at all, so it also stops the crawler from ever reading a noindex tag placed on that page. WordPress 5.3 switched to the current behavior, outputting a noindex,nofollow meta tag instead, specifically to close that gap.

That history matters because the same conflict still shows up constantly, just from different sources. Google’s own documentation is explicit about it: for a noindex rule to work, the page must not be blocked by robots.txt. If it is blocked, Googlebot never crawls it, never sees the noindex tag, and the page can still appear in search results. Combine a stray robots.txt disallow rule with a page you meant to noindex, and you get the opposite of what you wanted.

Layered on top of WordPress core’s setting is whatever your SEO plugin adds. Rank Math, for example, gives every individual post a Robots Meta control, plus a site-wide default per post type under Titles & Meta. That per-post control only wins if nothing upstream is overriding it. WordPress’s own “discourage search engines” checkbox sits above all of it: if that box is checked, it forces noindex site-wide regardless of what any individual Rank Math toggle says. Most “my noindex setting won’t stick” support threads trace back to exactly this: someone is fighting the wrong control.

Step-by-step fix

Work through these in order, checking each one against the live, rendered page rather than a settings screen, since that’s the only place you can see what’s actually being sent to Google.

1. Rule out the site-wide switch first. Go to Settings → Reading and confirm “Discourage search engines from indexing this site” is unchecked. This is the single most common cause of an entire site vanishing from search results, and it overrides everything else on this list. It’s also the setting most likely to get left on by accident, which the next section covers.

2. Check the post’s own SEO plugin setting. In Rank Math, open the post, switch to the Advanced tab in the metabox (turn on Advanced Mode from Rank Math SEO → Dashboard if you don’t see it), and look at Robots Meta. If it’s set to No Index, that’s a deliberate per-post override, either one you set intentionally or one left behind from an earlier draft state. The admin bar shortcut “Rank Math SEO → Mark this page → As NoIndex” sets the same field, so check there too if you’re not sure how it got set.

3. Check the post-type default. If individual posts look correct but an entire content type is missing, look at Titles & Meta → [Post Type] → Robots Meta in Rank Math. This sets the default for every post of that type and is easy to change once by accident while adjusting an unrelated setting.

4. Check robots.txt separately from noindex. These are two different mechanisms and fixing one doesn’t touch the other. If robots.txt disallows a path, no noindex tag on pages under that path will ever be read by Google, because the crawler stops at the robots.txt block. Load /robots.txt directly and look for a stray Disallow rule covering the page in question.

5. Save and move to verification. Once you’ve corrected whichever layer was wrong, save the change and don’t assume it worked from the dashboard alone.

How to verify it worked

Load the actual published URL in a browser, view source, and search for name="robots". That tag, as rendered on the live page, is the only reliable confirmation; a settings panel saying “Index” doesn’t guarantee the output matches, especially if a caching layer is serving a stale version of the page. This is the same discipline this site applies after any programmatic change made through the WordPress REST API: a successful API response confirms the database was updated, not what a crawler actually receives.

Separately, check the response headers for an X-Robots-Tag value, since some hosting configurations or CDN rules set this independently of the meta tag in the page’s HTML. Both the meta tag and the header are subject to the same robots.txt caveat: if the page is blocked, neither one gets read.

Confirm your XML sitemap doesn’t list any URL you intend to keep out of the index. A noindexed page sitting in your sitemap sends Google a contradictory signal and can slow down how quickly the removal actually takes effect once you fix it. After correcting a stuck noindex tag, expect the change to take days to weeks to show up in Google’s index; Google doesn’t publish a fixed recrawl schedule, so treat any faster claim with skepticism.

What to do when the standard fix fails

If you’ve cleared the site-wide setting, the per-post override, and the post-type default, and the page is still noindexed, the most common remaining cause is a staging-to-production migration that carried the “discourage search engines” setting along with it. Sites are routinely built on staging with that box checked on purpose, to keep the unfinished copy out of search results. When the staging database gets migrated to the live domain, that setting comes with it, and unless someone remembers to uncheck it after launch, the live site stays invisible to Google, sometimes for weeks before anyone notices the traffic never showed up. If your site recently moved from staging, or a migration or restore ran recently, this is the first thing to check, not the last.

Also worth remembering: “discourage search engines” is a request WordPress sends to crawlers, not an enforcement mechanism. Well-behaved crawlers like Googlebot respect it, but it doesn’t block access to the content itself, so it’s not a substitute for authentication or access control if the actual goal is keeping a page private rather than out of search results.

If none of the WordPress-side checks explain it, treat noindex verification as one line item in a broader routine rather than a one-off fix, alongside checks like confirming your canonical URLs point where you expect. Noindex and canonical tags solve different problems and are easy to confuse, but both are worth confirming together. Running through both as part of a standing WordPress SEO checklist, rather than only after Search Console flags a drop, catches this kind of drift before it costs weeks of missing impressions.

Frequently asked questions

Does unchecking “discourage search engines” index my site immediately?

No. Unchecking it removes the noindex signal, but Google still has to recrawl the page and decide to index it, which is not instant and has no published fixed timeframe. It generally takes days to weeks, not hours.

Can a page be noindexed and still show up in Google search results?

Yes, if robots.txt is blocking the page. Google’s documentation is explicit that a noindex tag only works if the crawler can actually reach the page; a robots.txt block prevents that, so the page can keep appearing in results even with a noindex tag sitting unread on it.

Is noindex the same as a canonical tag?

No. A canonical tag tells Google which URL is the authoritative version among duplicates; noindex tells Google not to show a page in results at all. Using one where you meant the other is a common source of confusion covered in more detail in the canonical URL guide.

Why does my SEO plugin’s noindex toggle not seem to work?

Check WordPress’s own Settings → Reading page first. The core “discourage search engines” setting overrides any per-post plugin toggle, so a plugin set to “Index” will still render as noindex if that core box is checked.

---