How do you get a website spidered to all you pages?

[citibank-credi-card]

I have another web site...http://www.webfina.com (not sure if i can post my urls here :oops: ) I noticed only a couple of my pages get spidered...but not all of them...esp from Google.

Any one out there know how to get all the pages get spidered? :lol:

Let me know...

Thanks...

[XLinux]

You need to have a robots.txt in the main directory of your website. When a spider goes through your site, it will read what to spider, and what not. If you need an example, let me know :-P

[FREE33]

I deleted robots.txt when I saw it on my new hosting account.

Thanks for your adive.

I will appreciate if you can show me further example.

[webdirect.bizhat.com]

Hey

Worked for me:

See here for an explanation of the robot.txt
http://www.searchengineworld.com/rob...s_tutorial.htm

I suggest you create a page on your site called Site Map which is a page that basically just has links to all the pages of your site in plain format - no graphics just html. A spider will crawl this better and all of your pages will become spidered.

Hope this helps....

[overhung]

no way..i've always wondered what the whole robot/spider thing was

i will deffinately have to check that out!

cheers

[othila]

Thanks for the information, this is really helpfull!