WE'VE MOVED!
We are proud to announce our NEW community destination. Engage with resident experts and fellow entrepreneurs, and learn everything you need to start your business. Check out the new home of StartupNation Community at startupnation.mn.co
We are proud to announce our NEW community destination. Engage with resident experts and fellow entrepreneurs, and learn everything you need to start your business. Check out the new home of StartupNation Community at startupnation.mn.co
Is there any FREE guide or lesson how to write a robot?

From reading "Will
A Sitemap Get Google To Crawl My Site Faster?" in
http://www.startupnation.com/forums/6814/1/1. I am adding a site map to
my site as soon as I can.Right now, I wanted to optimize my website ranking and indexing in Live.com (being no.2 website worldwide). At this point I am really concern that my traffic from Live.com is still 0 (zero).I was reading from Live.com web ranking and indexing guidelines and Live.com suggested to develop a robot.txt. I`ve been to http://www.robotstxt.org/wc/robots.html and it did not provide help in how to write a robot. Many of their books recommendation were written back in the 90`s.I was wondering if anybody know a website that can give us a FREE complete guidline on how to write a robot.Irwan.
A Sitemap Get Google To Crawl My Site Faster?" in
http://www.startupnation.com/forums/6814/1/1. I am adding a site map to
my site as soon as I can.Right now, I wanted to optimize my website ranking and indexing in Live.com (being no.2 website worldwide). At this point I am really concern that my traffic from Live.com is still 0 (zero).I was reading from Live.com web ranking and indexing guidelines and Live.com suggested to develop a robot.txt. I`ve been to http://www.robotstxt.org/wc/robots.html and it did not provide help in how to write a robot. Many of their books recommendation were written back in the 90`s.I was wondering if anybody know a website that can give us a FREE complete guidline on how to write a robot.Irwan.
Sign In or Register to comment.
Comments
Please check out http://www.robotstxt.org/wc/norobots.html</A>. This is the standard for writing and consuming robots.txt. This page also provides several good examples.
The robot.txt file is basically an instruction set that informs web-crawlers what they should / shouldn`t crawl in your site.
There are some sites that provide free, online wizards for this tasks. This is a good one: http://www.mcanerin.com/EN/search-engine/robots-txt.asp</A>
Enjoy,
David
Please check out http://www.robotstxt.org/wc/norobots.html. This is the standard for writing and consuming robots.txt. This page also provides several good examples.
The robot.txt file is basically an instruction set that informs web-crawlers what they should / shouldn`t crawl in your site.
There are some sites that provide free, online wizards for this tasks. This is a good one: http://www.mcanerin.com/EN/search-engine/robots-txt.asp
Enjoy,
DavidSecond that. If you want to look at the robots.txt file of a heavily used site check out Wikipedia`s robot.txt file.As for the Google/Yahoo Sitemap Generator, try GSiteCrawler this sucker will crawl a given website and create sitemaps for Google and Yahoo.
<META NAME="robots" CONTENT="ALL">
This tag is a waste of time. If a search engine finds your page and wants to index it, and hasn`t been blocked from doing so, it will. And if it doesn`t want to index a page, it won`t. Telling the search engine to do so doesn`t make a difference.
Here is a special Google meta tag that you can use a couple of ways. Here`s one example:
<META NAME="googlebot" CONTENT="nosnippet">
This meta tag tells Google not to use the description snippet, the piece of information it grabs from within a Web page to use as the description; instead it will use the DESCRIPTION meta tag. Here is another example
<META NAME="googlebot" CONTENT ="noarchive">
Using the ROBOTS meta tag or the robots.txt file, you can tell the search engines to stay away . The meta tag looks like this:
<META NAME="robots" CONTENT="noindex, nofollow">
check on this link https://www.youtube.com/watch?v=Hi41dBV3BxM