Is there any FREE guide or lesson how to write a robot?

From reading "Will
A Sitemap Get Google To Crawl My Site Faster?" in
http://www.startupnation.com/forums/6814/1/1. I am adding a site map to
my site as soon as I can.Right now, I wanted to optimize my website ranking and indexing in Live.com (being no.2 website worldwide). At this point I am really concern that my traffic from Live.com is still 0 (zero).I was reading from Live.com web ranking and indexing guidelines and Live.com suggested to develop a robot.txt. I`ve been to http://www.robotstxt.org/wc/robots.html and it did not provide help in how to write a robot. Many of their books recommendation were written back in the 90`s.I was wondering if anybody know a website that can give us a FREE complete guidline on how to write a robot.Irwan.
A Sitemap Get Google To Crawl My Site Faster?" in
http://www.startupnation.com/forums/6814/1/1. I am adding a site map to
my site as soon as I can.Right now, I wanted to optimize my website ranking and indexing in Live.com (being no.2 website worldwide). At this point I am really concern that my traffic from Live.com is still 0 (zero).I was reading from Live.com web ranking and indexing guidelines and Live.com suggested to develop a robot.txt. I`ve been to http://www.robotstxt.org/wc/robots.html and it did not provide help in how to write a robot. Many of their books recommendation were written back in the 90`s.I was wondering if anybody know a website that can give us a FREE complete guidline on how to write a robot.Irwan.
Sign In or Register to comment.
Comments
Please check out http://www.robotstxt.org/wc/norobots.html</A>. This is the standard for writing and consuming robots.txt. This page also provides several good examples.
The robot.txt file is basically an instruction set that informs web-crawlers what they should / shouldn`t crawl in your site.
There are some sites that provide free, online wizards for this tasks. This is a good one: http://www.mcanerin.com/EN/search-engine/robots-txt.asp</A>
Enjoy,
David
Please check out http://www.robotstxt.org/wc/norobots.html. This is the standard for writing and consuming robots.txt. This page also provides several good examples.
The robot.txt file is basically an instruction set that informs web-crawlers what they should / shouldn`t crawl in your site.
There are some sites that provide free, online wizards for this tasks. This is a good one: http://www.mcanerin.com/EN/search-engine/robots-txt.asp
Enjoy,
DavidSecond that. If you want to look at the robots.txt file of a heavily used site check out Wikipedia`s robot.txt file.As for the Google/Yahoo Sitemap Generator, try GSiteCrawler this sucker will crawl a given website and create sitemaps for Google and Yahoo.
<META NAME="robots" CONTENT="ALL">
This tag is a waste of time. If a search engine finds your page and wants to index it, and hasn`t been blocked from doing so, it will. And if it doesn`t want to index a page, it won`t. Telling the search engine to do so doesn`t make a difference.
Here is a special Google meta tag that you can use a couple of ways. Here`s one example:
<META NAME="googlebot" CONTENT="nosnippet">
This meta tag tells Google not to use the description snippet, the piece of information it grabs from within a Web page to use as the description; instead it will use the DESCRIPTION meta tag. Here is another example
<META NAME="googlebot" CONTENT ="noarchive">
Using the ROBOTS meta tag or the robots.txt file, you can tell the search engines to stay away . The meta tag looks like this:
<META NAME="robots" CONTENT="noindex, nofollow">
check on this link https://www.youtube.com/watch?v=Hi41dBV3BxM