A Small File That Can Make A Big Difference

**the-ref** · 11-19-2007, 02:14 PM

Another thing to keep in mind when it comes to using robots files is there is now law that states search engines need to follow robots files. While it hasn't really happened yet with any of the big search engines, it will be interesting to see what happens when one of the big players stops reading robots files.

**Greg** · 11-20-2007, 04:57 AM

IMHO why would anyone have pages on their site that are not optimized. That is just foolish or lazy. I have pages that I thought would never go anywhere and now they have a PR3 and are picking up long tail hits. I have registration forms that show up #1 for their optimized kw. How cool is that?

The more pages the se sees the better. IMHO

**MAAOnline** · 02-06-2008, 11:34 AM

This is really interesting but it looks like it's a lot to remember. Does anyone have any experiences with using the robot.txt file on there websites? How has it effected your rating?