Robots.txt blocks your WordPress pages: find and fix the responsible rule

Recognize a crawl block

Search Console reports that an important page is blocked by robots.txt, even though it opens in your browser. This is possible: the file sends instructions to crawlers and does not close the page to ordinary visitors. The aim is to restore crawling only for the affected public paths, without removing restrictions that remain useful.

A robots.txt block is not a noindex directive. It prevents the crawler from reading the page; it does not guarantee that an address known from elsewhere will disappear from results. By contrast, noindex requires the crawler to retrieve the resource to read that instruction. Identify the precise reason before editing a file.

Start with one affected final URL and another URL whose current treatment must be preserved. Also keep a copy of the rules received and their date. Avoid templates presented as the best robots.txt for every WordPress site: a brochure site, a store, and an installation in a subdirectory do not necessarily have the same paths or crawling requirements.

Find the file for the correct origin

The file is requested at /robots.txt at the root of the relevant origin. The protocol, host, and port matter. A subdomain’s file does not automatically govern the main domain, and the HTTPS file does not represent every HTTP variant. Use the final version actually used by public pages.

If WordPress is installed in a subdirectory, do not assume that the file belongs in that subdirectory. For a public page under /blog/, the applicable file remains the one at the root of its host. Several applications may therefore share the rules of one origin, even when their administration interfaces differ.

Record the address of the robots.txt you consulted alongside the affected URL. If you need to edit it, identify the corresponding public directory on the hosting account; this is not necessarily the folder containing wp-admin. In multisite, have the network administrator check its scope. A correction at the root can affect several sites sharing the same host, while a separate domain has its own effective file.

Read the public response before making changes

Open the exact robots.txt address while logged out. Check that you receive rule text rather than a login page, a protection screen, or an HTML error. A reassuring browser title does not prove that the file is being read correctly. Save the content and response status in your record.

With an authorized terminal, the example below performs a bounded GET and displays the headers and text. Replace the fictional domain. On Windows, use curl.exe to call the intended program. The -q option must remain first; no cookie or password is supplied. This read does not change the site.

The command does not automatically follow redirects. If it receives one, identify the destination before continuing. Success from your computer does not guarantee that Google has already retrieved the same version. The file may also differ because of an intermediate protection layer; comparing it with Google’s report helps clarify that discrepancy without inventing the content of its last retrieval. The absence of a program error is not enough: without --fail, curl can finish with exit code 0 after an HTTP 403. Read the status and content received.

curl -q --silent --show-error --connect-timeout 5 --max-time 15 --include 'https://exemple.invalid/robots.txt'

Compare with the Search Console robots.txt report

In the appropriate property, open Settings, then the robots.txt report when available. This report applies to domain properties or URL-prefix properties at a host’s root; a property restricted to a subpath does not provide the same access. Its absence in the latter does not mean that no file exists. A domain property displays only its twenty main hosts in this report. If the relevant subdomain is missing, check a property suited to that subdomain rather than concluding that robots.txt is absent.

Select the address of the relevant file and record the retrieval date and any warnings or errors. View the retrieved version to compare it with the current text. When a Versions history is available, it can help connect the start of the block with an earlier change.

Look for an actionable difference: Google read an old block, cannot retrieve the file, or the current rule really does block the path. Do not rewrite the content to address a mere lack of data in the wrong property. Keep the dated result and precise URL; revisit these after the correction, along with the inspection of the affected page.

Identify the group that applies to the crawler

User-agent lines identify the crawlers targeted by a group of rules. For Google, a specific group matching the crawler is selected instead of the general * group. General rules are not automatically added to that specific group. If several groups match the same specific user-agent, their rules may be combined.

In this fictional example, the general permission does not unblock /guides/ for Googlebot, because its own group disallows it. This is a diagnostic fragment, not a replacement for your entire file. Find the applicable group before changing a line; adding an exception in another group can leave the problem unchanged.

The visual order of groups is not enough to determine which wins. Do not try moving a block to the bottom of the file on the assumption that it cancels everything above. Compare the crawler name, group structure, and paths actually targeted. Preserve groups dedicated to other purposes when their role remains justified.

User-agent: *
Allow: /guides/

User-agent: Googlebot
Disallow: /guides/

Read paths and rule precedence

Within the selected group, Allow permits crawling of a path and Disallow prohibits it. Matching starts at the beginning of the path and is case-sensitive. A restriction on /Guide/ does not necessarily cover /guide/. The most specific path determines the result; where an allowance and a prohibition match equally, the allowance takes precedence.

A Disallow: / rule targets the origin’s entire path, whereas an empty value after Disallow does not impose that blanket restriction. The trailing slash also matters: an exception for /dossiers/publics/ does not match the exact address without that final slash. Use the exact final URLs and any variants during checking.

Patterns containing * or $ can broaden or delimit a match. Do not add them to approximate a folder when its simple path is sufficient. A rule targeting parameters may also affect useful pages. If the file is complex, reduce the case to a few representative addresses and have a competent person examine how they match, rather than changing every line at once.

Back up and edit the responsible physical file

This approach applies when a real robots.txt file serves the response at the host’s public root. It requires SFTP access or access to that hosting account’s authorized file manager. First identify the domain’s directory and download a private copy of the file. The name visible in the URL cannot tell you a universal disk path.

If no physical file is found, do not create an empty file as an experiment: WordPress may produce a virtual version. Instead, go to its actual manager. A physical file added in its place could hide its rules and sitemap references.

1. Open the confirmed file in a plain-text editor, keeping the original copy outside the public directory.
2. Locate the applicable group and the sole restriction demonstrated to be responsible for blocking the public URL.
3. Prepare a limited correction, such as the path exception in the next section, keeping the other rules.
4. Save under the exact name robots.txt in UTF-8, then replace only this file using your usual access method.
5. Read its public response and the control paths again. If the result is unexpected, immediately restore the backed-up file and verify the restoration too.

Allow a public subpath without opening the entire folder

Suppose /dossiers/ must remain restricted, but /dossiers/publics/ must be crawled. The fictional example below retains the general prohibition and adds a more specific allowance. Adapt the paths to your site and place this exception in the group that actually applies; if a specific Googlebot group exists, the general block shown here is not enough to change it.

The expected result is that crawling is allowed for /dossiers/publics/guide/, while /dossiers/interne/ remains disallowed by this policy. Also test /dossiers/publics without the trailing slash: it is not covered by the exception shown. If it is a real variant, examine its redirect or prepare an appropriate rule instead of blindly broadening the entire set.

This fragment does not replace your complete file. Preserve the other groups and rules, along with sitemap references. To undo this test, remove only the exception you added and check the response, or restore the original file if this was the only change. The directory called interne in the example still needs access control if its content is confidential.

User-agent: *
Disallow: /dossiers/
Allow: /dossiers/publics/

Remove an inherited blanket block only after confirming it

An old Disallow: / may have been copied from staging. If the relevant group should now crawl public paths and this restriction is no longer justified, correct only this directive after confirming the scope. Do not delete the whole file, since it may contain other choices that are still necessary.

For a group that no longer needs any restrictions, a Disallow: line with no value expresses the absence of that prohibition. Keep the group structure clear: removing the last rule of a block without examining the next one can make the file misleading to read. If part of the site must remain restricted, use targeted rules instead, as in the previous example.

The native WordPress visibility checkbox no longer automatically produces this blanket block in the current core generation. Uncheck it if its noindex is unintended, but do not assume that this will erase a physical line or a custom filter. After the correction, check the effective file and a public page, then check staging separately. Rolling back means restoring the initial rule, without mistaking this instruction for data protection.

Use the Yoast SEO editor if it is already available

If Yoast SEO is already installed and its file editor is permitted, it can avoid an SFTP edit. As of November 3, 2026, the directory version is 28.8, by Yoast, requiring at least WordPress 7.0 and PHP 7.4, with testing declared up to WordPress 7.1.4. This free tool does not require a Premium purchase for this operation.

First save the existing file in a private location. If the menu is missing or refuses writes, use the person responsible for files; do not change global editing restrictions to force this tool to work. The expected outcome remains the corrected public response, not merely a save message.

1. In the dashboard, open Yoast SEO, Tools, then File Editor.
2. In the area titled robots.txt, locate the identified group and rule. Do not edit the separate .htaccess area that may appear on the same screen.
3. Change only that rule or add the targeted exception, then use Save changes to robots.txt.
4. Read the file’s public URL again and inspect the control page.
5. To undo the change, put the initial text back in the same area and save.
6. If only Create robots.txt file is offered, first check virtual generation and the effects of creating a file. Do not replace its rules with a new file before inventorying them.

Handle virtual generation without adding a second file

WordPress can serve robots.txt without a corresponding physical file. Its do_robots function produces a base, then the robots_txt filter lets components modify it. The hosting platform, a theme, or a plugin may therefore explain a rule you cannot find in a file at the root.

Compare the output received with the owning component’s settings. If it is custom code, have someone locate the robots_txt filter and the exact line adding the prohibition in the child theme or relevant module. Keep its original, try correcting only that condition on a copy, and check the complete output. Do not directly edit the core file containing do_robots: an update could erase the change and the cause of the problem would remain hidden.

On a hosted service or multisite, access and scope may be limited. Use documented settings or the person responsible for that generator rather than a second robots.txt plugin. To undo a code test, restore only the modified file. A physical file created alongside it can hide virtual generation; determine which one is responding before concluding that the filter does not work.

Preserve access to resources required for rendering

An allowed HTML page may depend on styles, scripts, or data that the crawler cannot retrieve. Its content may then render incompletely. Examine the resources actually used by a control page instead of checking only its main address.

In live URL inspection, open View tested page, then More info for available resources and errors; also view the screenshot when the tool provides one. If essential content is missing or a necessary public resource is blocked, compare its path with the rule applicable to its own host. A resource subdomain has its own robots.txt.

Avoid disallowing all scripts, styles, or the content directory merely to reduce crawling. Correct the paths that really prevent rendering, while preserving access controls for private data. A robots.txt allowance does not give the crawler a user session and must not make sensitive files public. Then repeat the same test and compare the rendered content, without treating a correct screenshot as a guarantee of indexing.

Distinguish a restrictive rule from an unavailable file

A robots.txt that cannot be found and returns status 404 is not equivalent to a file containing Disallow: /. For Google, 4xx errors for this file, except 429, are treated as an absence of robots restrictions. A 5xx error, timeout, or request limit instead calls for examining retrieval and the last known version, rather than interpreting text Google never received.

If the response is a protection page, correlate its time with logs from the relevant layer. Have the block on this legitimate read corrected without disabling all site security. Keep the previous settings so that an overly broad or ineffective exception can be undone.

A cache can retain the old file after you save. Correct its source first, then invalidate the robots.txt copy using the existing system if necessary. Read it again once the cache has been rebuilt. If the content changes again after synchronization, identify the deployment step that overwrites it. Repeated purges do not resolve this reintroduction of the rule. The version retained by Google is another cache, separate from your hosting cache. Your local purge does not automatically trigger its update.

Have the corrected file and affected page checked

After an important correction, first read the public file again to confirm that the right version is actually being served. In Search Console’s robots.txt report, the settings menu beside the file lets you request a recrawl. Use this action for a critical change or a resolved error; it does not guarantee immediate retrieval or crawling of the unblocked pages.

Then inspect the control page’s final URL and run Test live URL. Check whether the current blocking reason has disappeared. Keep the last known indexed version separate: it may still describe the previous state. Also check the address that was supposed to retain its initial treatment, taking the applicable group and path into account.

If crawling is now possible but a noindex instruction remains, handle that other source separately. If the page is technically accessible and still not indexed, continue analyzing its content and discovery. Removing a robots restriction resolves an access obstacle; it does not by itself determine indexing or ranking.

Prevent the block from returning and share useful evidence

Assign responsibility for the file to one clearly identified system: a physical file, WordPress generation, or a hosting service. Document exceptions and their reasons so that future cleanup does not remove them without understanding their scope. A rule commented with its purpose is easier to maintain than a long list copied from several recipes.

In your go-live procedure, compare production and staging. Check their origins, their protections, and the file actually served after deployment. Keep a small set of public URLs and deliberately restricted URLs for this check. Do not use only the filename or a successful transfer as evidence that the correct host serves it.

To request targeted help, provide the robots.txt URL, the affected page, the relevant user-agent, the suspected group and rule, and the before-and-after versions with their dates. Add the current inspection result and the file retrieval status. Do not attach SFTP credentials or a complete site backup. These items help explain a precise block and verify the correction without redoing the entire file.

Check variants of the blocked path

The example block that disallows /dossiers/ while allowing /dossiers/publics/ deserves checking against several addresses, not just the homepage. With only this group applying to the crawler being tested, the expected results are:

These examples describe Google’s path comparison; they are not test results from your server. A rule dedicated to Googlebot can make the generic group inapplicable to that crawler. Find its complete group before drawing a conclusion from two isolated lines. Also record redirects: the starting address and its destination are two URLs to consider, and a blocked path may prevent the crawler from discovering its redirect.

For beginners, open the public file, identify the group, and compare paths character by character. For an important configuration, have the complete file validated using your organization’s designated checking tool. After the correction, read the public response again and keep the previous file so you can undo only the change. No plugin is needed for this comparison. A private area must always be protected by its access control; removing it from this list neither grants nor removes authentication.

• /dossiers/publics/guide/: allowed by the most specific prefix;
• /dossiers/interne/: disallowed;
• /dossiers/publics without a trailing slash: still disallowed by /dossiers/, because it does not match the allowed prefix ending in a slash;
• /Dossiers/interne/: not blocked by these two lowercase rules.