hallo

I've been looking at the number of pages our site has in the indices and, particularly in Google, how many of those pages are in the supplemental index. I've noticed that most of our pages that are in the supplemental index consist of URLs that are refinements of the search results pages on our site.

Basically we have geographically based serach results pages, which a user can then refine by certain attributes to hone in on exactly what they are looking for.

I am starting to wonder whether these URLs are just a distraction for the bots. Our most important pages are the actual listing pages we want people to visit and the main navigation pages.

By blocking the bots from those refinement URLs couldn't we tighten the bot's focus on our most important content, get it crawled better, possibly more quickly and imply its importance better?

I have been looking at some of our main competitors and I notice that something almost all have in common is a very focused structure. They have fewer numbers of pages indexed, but also much fewer pages in the supplemental index. They are doing quite well ranking-wise and, from what I can gather, traffic-wise.

It seems like the efficiency of a leaner site would be valued by the bots and the users and such an approach would be worth emulating. Or maybe leaving the refinement functionality for the users, but blocking the bots from it to increase the efficiency of their crawls of the site.

Any thoughts?

CristianR

yes , it is actually a good idea to use the robots.txt , and don't let them index everything , keep them away from parts of your site

njoker555

Well I am oppositve, I let them index what they want, although I use a forum, I can block out bots from some boards and I do, from staff boards

__________________
Gamers Forum << PR3 Forum
WebZ Junk Zone << PR3 Blog

CristianR

the problem with letting them index whatever they want is that they can even index your login page , and sometimes this makes you vulnerable and data leaking

njoker555

it indexes /login.php no vulnerability in that, no usernames or passwords gets leaked that way, just another page

__________________
Gamers Forum << PR3 Forum
WebZ Junk Zone << PR3 Blog

CristianR

it doesn't just index.php , it indexes all of your .php pages , including the login , let me give you an example .. a live one .. type this in google , and you'll see what comes up , fortunetly these accounts will not be valid since rapidshare solved this problem