All about robots.txt

If you manage a website, you may have heard of something called a robots.txt file.

It sounds technical, but the concept is pretty straightforward—and it can play a small but important role in how search engines and other bots interact with your site.

This guide will help you understand what a robots.txt file is, why it might matter to your site, and how to create a very basic ‘robots’ file. No coding knowledge needed!

---

What is robots.txt?  

A robots.txt file is a simple text file placed at the root of your website. It tells search engine bots/crawlers (such as Googlebot or Bingbot) which parts of your site they should or shouldn’t access.

The instructions within the robots.txt file can be considered as a ‘code of conduct’ for search engine bots when they come to your site.

For example:

• You might want search engines to skip private admin pages.
• Or block known “bad bots” that eat up server resources.

It’s also important to note that robots.txt is a request, NOT a rule.

Most well-behaved search engines will respect it, but not all bots will follow the instructions, especially less reputable ones.

Please use caution: blocking the wrong pages could prevent your site from being indexed correctly in search engines.

---

What can robots.txt be used for?

Here are a few common uses:

• Preventing duplicate content from being indexed (like search results pages)
• Blocking access to sensitive or irrelevant areas, such as /wp-admin/ or /cgi-bin/
• Reducing load from bots that crawl too aggressively
• Improving SEO by focusing crawlers on your important pages

What robots.txt CANNOT do

• It won’t prevent a page from appearing in search results if it’s linked elsewhere
• It doesn’t always block access from ‘bad’ bots – less reputable bots often ignore it
• It’s not a security feature. Sensitive info should be protected through proper permissions, not via robots.txt

---

A Basic Example

The format in this example is commonly used on WordPress sites to stop crawlers from wasting time on backend directories that aren’t useful for search results.

Basic directives include: User-agent, Disallow, and Crawl-delay.

Note: Not all bots respect the crawl-delay directive. For instance, Googlebot does not support it (use Google Search Console instead). However, Bingbot and some other crawlers do.

• User-agent: The user agent is the name of the bot/crawler. You can specify certain bots with multiple ‘User-agent:’ entries. An asterisk (*) is used as a catch-all and means the rules apply to all bots.
  
• Crawl-delay: This asks bots to wait the specified amount of time (in seconds) between requests. This can help to reduce server load
  
• Disallow:These lines tell bots not to visit the locations/folders specified. You can list specific directories such as /wp-admin/. Using a single slash (/) indicates the root or all directories.
  

Here’s an example of a simple robots.txt file:

User-agent: *
Crawl-delay: 5
Disallow: /wp-admin/
Disallow: /cgi-bin/


User-agent: NameOfBadBot
Disallow: /

What this does:

Each block of text begins with the user agent/s that you want the following set of rule/s to apply to.

So, in our example above, there are two blocks of text. The 1st will apply to all bots (remember the * means everyone), and the 2nd will apply only to the user agent listed.

• User-agent: *
  The asterisk (*) is asking that the rules apply to all bots.
  
• Crawl-delay: 5
  This asks bots to wait 5 seconds between requests.
  
• Disallow: /wp-admin/:
  This asks bots not to visit the locations/folders specified. In this case, the “wp-admin” folder.
  

• User-agent: NameOfBadBot
  This specifies that the rules only apply to the bot named
• Disallow: /
  Using / (the root directory) as the location is asking the bot not to access any part of the site, essentially asking bots not to visit at all.

---

How to create a Robots.txt file  

A robots.txt file is a simple text file that can be created offline (using Notepad, etc) and then uploaded to your site’s root directory.

Alternatively, you can create the file directly on your web server/cPanel via the File Manager tool:

• Log into your hosting/cPanel
• Navigate to the File Manager
• Go to your site’s root directory (usually /public_html)
• Create a new file named “robots.txt”
• Edit the file to add your robots.txt instructions
• Save the file when you’re done

---

Robots.txt and Search Engine Optimisation (SEO)

If you’re looking to get serious about SEO or bot traffic management, we recommend:

• Using SEO plugins or tools (like Yoast, Rank Math, or Screaming Frog)
• Consulting an experienced SEO specialist
• Reading Google’s official documentation on robots.txt

Every website is different, and there’s no one-size-fits-all setup. A professional can help craft a strategy tailored to your content, goals, and audience.

---

Summery

• robots.txt helps guide bots on how to crawl your site
• It’s simple to set up, but should be used with care
• Please use caution: blocking the wrong pages could prevent your site from being indexed correctly in search engines.
• It’s not a security tool or foolproof barrier
• You can also include your sitemap to help search engines find your content
• For anything beyond a basic setup, we recommend further research or an SEO consultation

Creating a basic robots.txt file is a great first step in taking control of how your website interacts with the wider web.

---