View Full Version : When bots crawl the site?
Bogzz
07-10-2007, 01:10 PM
hi.. please kindly feedback regarding for the question a above.. thanks in advance.
Nealrm
07-24-2007, 11:52 AM
You will need to reword the question before anyone will be able to answer. As stated it does not make sense.
agentstealth.com
07-30-2007, 06:12 PM
You should reword your question. I think you are asking - how often do search engine bots crawl your site to index content and changes?
If you are concerned about the content in your site being updated and indexed, you should use meta tag "robot". You can enter a frequency of time between when robots will spider your website and what pages it will include.
the-ref
11-19-2007, 05:14 PM
It really depends how frequently updated your site is with new content. The more times it is updated, the more times spiders will crawl through your website.
refid
11-19-2007, 07:13 PM
The best way to have your site crawled more often is by having outbound links.
The suggestion given by the-ref is also true.
About the Meta robot mentioned above doesn’t help much. This Meta which is the: name="revisit-after" content="whatever days” doesn’t instruct the robots to return, but not to return before the amount of days chosen by you.
FlipItBig
03-02-2008, 10:41 AM
I constantly see yahoo spiders, yahoo slurp spiders etc at my site. the-ref is right on, more content!!!
tina.anderson
03-05-2008, 09:13 AM
Does that mean a blog would tend to be crawled more often than a static website because the content is always changing and being updated?
FlipItBig
03-05-2008, 10:14 AM
That's how it has been explained to me Tina. Make sure to use a lot of key phrases in the last few paragraphs of your blog too...helps with search engines.
adamwealth
03-06-2008, 08:06 AM
My understanding is that more active blogs and sites get crawled more often. Also, having links to your site from external sites helps a ton too!
adamwealth
03-06-2008, 08:09 AM
And of course providing outbound links too!!!! This explands your relevance big time!
tina.anderson
03-13-2008, 09:25 AM
That's how it has been explained to me Tina. Make sure to use a lot of key phrases in the last few paragraphs of your blog too...helps with search engines.
Ok thanks! I have another question though. If you make frequent changes to your static website, say maybe once a month, would the crawl that happens as a result carry more weight than say a blog that was updated every day, or doesn't it work like that?
vBulletin® v3.7.0, Copyright ©2000-2009, Jelsoft Enterprises Ltd.