Robots.txt File

There are three steps to creating a robots.txt file. You create the file itself, decide what to include in the file, and upload it to the correct location on your server.

Choosing the right program to create a robots.txt file

A robots.txt file is a plain text file. So, you'll want to be sure to create the file using a plain text editor, such as Notepad. Do not use a program such Microsoft Word, because programs like these will add unwanted code to the file.

Pay special attention to the .txt file extension when creating the file.

Deciding what to include in the file

What is your goal in creating the robots.txt file? Do you want to grant access to all robots, or only to certain robots? Do you want to restrict access to certain files or folders?

Head over to the robots.txt examples page for examples of what you can do with a robots.txt file.

There are two main elements that must be included in the file. These two elements are User-agent: and Disallow:

User-agent: 
Disallow: 

User-agent: defines which robot the next line refers to.

Disallow: tells the robot identified above what files or folders not to access.

While most good robots will listen to what the robots.txt file says, bad robots will not. A robots.txt file is not a guarantee that all robots will behave as requested.

To allow all robots access to your entire site, copy and paste the following code into your plain text editor.

User-agent: *
Disallow:

Where to put the robots.txt file

The robots.txt file needs to be placed at the root of your server. That means that it would be visible in your browser when you look here:

http://www.your-site.com/robots.txt

Note that you would replace 'your-site.com' with your domain name.

The file will not work if you place it into a sub-folder, like this:

http://www.your-site.com/somefolder/robots.txt