Plugin
Updated
Better Robots.txt - AI-Ready Crawl Control & Bot Governance (Premium)
Better Robots.txt (Premium) by Pagup is a WordPress plugin for creating and managing your site's robots.txt file. It replaces manual editing with a guided workflow, presets, visual controls and a preview step, so you can decide which search engines, AI crawlers, bad bots and SEO tools may crawl your site.
- Version
- 3.0.1
- Released
- Apr 15, 2026
- Developer
- Pagup
- Type
- WordPress plugin
- Licence
- GPL
- Included in
- Every plan, or $5.59 a year on its own
- Topics
- SEO Plugins
What’s new in Better Robots.txt - AI-Ready Crawl Control & Bot Governance (Premium) 3.0.1
Version 3.0.1
Latest
- FixedRemoved Push notification popup during freemius opt-in.
- ImprovedUpdate Freemius to v2.13.1
What it does
A robots.txt file tells crawlers which parts of a site they may visit, and a single typo can block pages you want found or open areas you meant to keep closed. Editing it by hand is easy to get wrong, especially if you rarely touch code. This plugin gives you a guided process inside the WordPress dashboard, with ready-made presets for common crawler rules and toggle controls for adjusting access to specific bots.
Before anything goes live, you can preview the finished robots.txt file and check that the rules match what you intended. If a change causes trouble, earlier versions can be restored. Beyond search engines, the plugin covers AI crawlers, unwanted bots and SEO tools, and it can block low-value URLs to tidy up crawl behavior. Multisite support lets agencies and network admins manage robots.txt files for several sites from one interface.
Best for
- Site owners who want crawl control without editing code
- Publishers deciding how AI crawlers access their content
- Agencies and network admins running WordPress multisite
Features
Guided setup workflow
Walks you through building a robots.txt file step by step, so you can set crawl rules without writing or editing the file by hand.
Presets for common rules
Ready-made configurations cover typical crawler scenarios, giving you a sensible starting point that you can adjust to fit your own site.
Visual rule editor
Drag-and-drop and toggle controls let you allow or block crawler access visually, which keeps the process approachable for people who prefer not to code.
Preview before publishing
See exactly how your robots.txt file will look before it goes live, so you can catch mistakes and confirm each rule is correct.
Control over many bot types
Set separate allow and block rules for search engines, AI crawlers, bad bots and SEO tools, giving you detailed control over who crawls your content.
Low-value URL cleanup
Blocks low-value URLs from being crawled, helping crawlers spend their time on the pages of your site that actually matter to you.
Rollback to earlier versions
Changes to your crawl policies can be rolled back to previous versions, so an unwanted edit is simple to undo.
Multisite management
Manage robots.txt files across multiple WordPress sites from a single interface, which suits agencies and network administrators handling several properties.
Questions about Better Robots.txt - AI-Ready Crawl Control & Bot Governance (Premium)
Can I block AI crawlers while still allowing search engines?
Yes. The plugin lets you set separate rules for search engines, AI crawlers, bad bots and SEO tools, so you can allow some groups and block others. The visual controls and presets make it practical to set these choices without writing robots.txt directives yourself.
What happens if I publish a rule by mistake?
You can preview the robots.txt file before publishing to catch errors early. If a change still turns out to be wrong, the plugin supports rolling back to previous versions of your crawl policies, so you can restore a setup that worked.
Do I need to know robots.txt syntax to use it?
No. The guided workflow, presets and toggle-based rule editor handle the file structure for you. Knowing the basics can help you make informed choices, but the plugin is built so you can manage crawl rules without coding.
Does it work on a WordPress multisite network?
Yes. It includes multisite support, letting you manage robots.txt files for multiple WordPress sites from a single interface. That is handy for agencies and network admins who look after several sites at once.
Is Better Robots.txt - AI-Ready Crawl Control & Bot Governance (Premium) free?
The code of Better Robots.txt - AI-Ready Crawl Control & Bot Governance (Premium) is licensed under the GPL, so anyone may use, change and share it. GPL Feast is a paid service: a plan or a single purchase gives you the download and automatic updates. The developer's own support, licence key and online services are not included.
Which version of Better Robots.txt - AI-Ready Crawl Control & Bot Governance (Premium) is this?
Version 3.0.1, released Apr 15, 2026. New versions appear here when the developer publishes them, and members get them as automatic updates.
How do I update Better Robots.txt - AI-Ready Crawl Control & Bot Governance (Premium)?
Install the free GPL Feast Updater on your site and connect it with your API key. It installs new versions of Better Robots.txt - AI-Ready Crawl Control & Bot Governance (Premium) automatically and keeps a backup, so you can roll back.
You might also like
Better Robots.txt - AI-Ready Crawl Control & Bot Governance (Premium)v3.0.1 · From $14/month or $5.59/yr
See plans