Each line in the file is a rule (a directive) that a bot will read and follow when it visits the site. Let's look at what each line is doing:
The User-agent directive tells the bot to follow all the rules underneath. The colon separates the directive from its value. The value of the user agent, in this case, is *. This is a wildcard and it will match all user agents.
What are these user agents? The user agent is the way a bot identifies itself when it visits your site. Here is a list of common user agents:
Googlebot - Used for Google Search
Bingbot - Used for Bing Search
Slurp - Yahoo's web crawler
DuckDuckBot - Used by the DuckDuckGo search engine
Baiduspider - This is a Chinese search engine
YandexBot - This is a Russian search engine
facebot - Used by Facebook
Pinterestbot - Used by Pinterest
TwitterBot - Used by Twitter
You can use these user agents to create rules for specific bots. For example, you can block facebot from visiting certain parts of your website like this:
WordPress by default has two rules all bots should follow.
This directive is “Disallow”. This tells the bot that they are not allowed to go to a certain area of the website. The value is /wp-admin/ which is a folder on the website. This means that all bots are not allowed to visit the admin area of the WordPress site.
The directive “allow” will give access to an area of the site to a bot. In the last rule, we removed access to the admin area. This rule then gives access to a single file within that admin area.
You may be wondering why Google needs to read this file when it is in the admin area.
Do not delete or remove this rule unless you know what you are doing.