Allow indexing of everything
User-agent: *
Disallow:
Disallow indexing of everything
User-agent: *
Disallow: /
Disawllow indexing of a psecific folder
User-agent: *
Disallow: /folder/
Disallow Googlebot from indexing of a folder, except for
allowing the indexing of one file in that folder
User-agent: Googlebot
Disallow: /folder1/
Allow: /folder1/myfile.html
To block access to all URLs that include a question mark
(?), you could use the following entry:
User-agent: *
Disallow: /*?
You can use the $ character to specify matching the end of
the URL. For instance, to block an URLs that end with .asp, you could use the
following entry:
User-agent: Googlebot
Disallow: /*.asp$
