Perfect Robots.txt for WordPress
When it comes to search engine optimisation for WordPress, some key things need to be set up for all websites. The Robots.txt for WordPress file is one such file that needs to be set up and configured to ensure search engine crawlers are optimised when reading and analysing your website.
In most cases, you will want a search engine to crawl and index all the pages on your websites, but there are some pages that shouldn’t be accessed or, if indexed, don’t provide any real value to the users that are seeing it in search results.
Pages such as the admin login page, search results pages on WordPress or taxonomy related pages don’t provide value to a search users and shouldn’t appear in the search engine results.
Applying appropriate rules in your robots.txt file will ensure that you have optimal search results for your audience.
Here are some areas that you do not want to be crawled on your websites.
- Admin section
- Internal search results
- Tag, category and author pages
- 404 and error related pages
This is our standard robots.txt file that will help you achieve just that!
#This section blocks AI bot crawlers scraping your website
User-agent: Amazonbot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: *
# Block WordPress core admin and login pages
Disallow: /wp-admin/
Disallow: /wp-login.php
# Allow admin-ajax.php (required for WordPress front-end)
Allow: /wp-admin/admin-ajax.php
# Block common WordPress paths not needed for indexing
Disallow: /?s=
Disallow: /search
Disallow: /cgi-bin/
Disallow: /xmlrpc.php
Disallow: /trackback/
# Block WordPress core files
Disallow: /wp-content/plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/
Disallow: /wp-includes/
# Block duplicate content
Disallow: /tag/
Disallow: /author/
Disallow: /archives/
# Allow media uploads
Allow: /wp-content/uploads/
Sitemap: https://yourwebsite.com/sitemap_index.xmlThe first section of the file covers all of the AI crawlers that scan through websites, chewing up resources and bandwidth.
The second half blocks search engine crawlers from snooping around WordPress related folders and duplicate content areas while still allowing crawlers to scan and index images and other assets that are on your website.
Additionally, you will want to ensure that your website’s Sitemap.xml is included in the robots.txt file, so search engines know which pages to index and include in their search results.
How to Modify Your Robots.txt File with RankMath
If you’re using RankMath to manage your WordPress website’s SEO, then you’re on easy street when it comes to modifying your robots.txt file.
These steps will guide you through modifying your robots.txt file and updating it with all the additional features to protect your site and optimise it for search engine crawlers.
Navigate to RankMath Robots Editor
Once you’ve logged into the admin area of your website, from the main admin menu, navigate to:
RankMath > General Settings
From here, you will see all of the general settings RankMath can manage.
You will see an additional RankMath menu.
Click on “Edit robots.txt“
Toggle Editor Mode
To edit your website’s Robots.txt file, you will need to toggle the Editor option to be able to modify the file.
Click on Editor in the top right corner, and copy and paste in the contents of the perfect robots.txt file into the editor.
BUT LEAVE YOUR SITEMAP REFERENCE INTACT.
Ensure Your Sitemap Reference Remains
Lastly, make sure that your sitemap reference remains intact.
It would look something like this:
Sitemap: https://yourwebsite.com/sitemap_index.xml
Replace https://yourwebsite.com/ with your primary domain. The sitemap_index.xml part should remain the same, as RankMath generates it.
Save and Test
At this point, you can now press Save Changes in the bottom right corner.
You’re now done!
However, you can now test your file to ensure it is valid.
In a new browser tab, navigate to https://rankmath.com/tools/robots-txt/
Enter in the URL of your website and choose a crawler such as Googlebot.
Press TEST
This will give you a rundown of any issues that need to be fixed on your robots.txt file, but if you followed the instructions carefully, you should be perfectly fine.
Congratulations, you have just levelled up your robots.txt file to the perfect one for WordPress.
You can learn more about search indexing and sitemaps in our other insights and resources.
If you need help, advice, custom WordPress development, or further work on optimising your website’s search presence, sign up for our newsletter for more regular hints and tips.
Ready to build a project?
Let's create something amazing together.