Hello World,

Well in short my problem is as follows,
Google is unable to crawl my site. I don't know why or where to even start.

I have created several nuke sites and they all can be crawled accept one.
For example, I'm testing it by using the search tool at googles webmaster central here http://www.google.com/sitesearch/#utm_source=webmaster_central&utm_medium=et&utm_campaign=en

One nuke site works great using the following perams....
URL = www.austinabc.com
SearchTerm = *

Yet when I try
URL = www.dreamteamrealty.com
SearchTerm = *
Nothing shows up, yet there is content there.

I'm using RavenNuke(tm) Version 2.20.01
I have submited the nukeseo sitemap and it is working and verified, yet google has not yet indexed www.dreamteamrealty.com.
I submited it about 32 days ago.

Any suggestion?

You4eea

I think I may have found the problem.

I used this tool http://www.google.com/sitesearch/#utm_source=webmaster_central&utm_medium=et&utm_campaign=en

and searched other sites hosted on this server and I even changed the index.php to index.html on www.dreamteamrealty.com and found that it just wasn't getting reached by the above tool where as all other domains were.

I have had whois data get corrupted before due to some hosting company botching up the transfer process.

So I'm transfering the registrar and I will update you all to see if that fixes it.

Interesting. I have never had this issue of not crawling. Curious as to what your robot.txt file looks like. If it is an unmodified version from RavenNuke 2.20.01 then that should be fine. Definitely keep us posted.

Hey montego,

I'm still having the same issue, very strange.
After reading your post above I went and copied the robots.txt file
from one of my other sites and same problem. It just isn't indexing
it. In the webmaster tools -> Sitemaps the overview gives this info:

Sitemaps
dreamteamrealty.com
Sitemap statistics:
Total URLs: 18
Indexed URLs: 0

Here is what my robots.tx file looks like,
User-agent: Mediapartners-Google*
Disallow:
User-agent: *
Disallow: /admin.php
Disallow: /admin/
Disallow: /images/
Disallow: /includes/
Disallow: /themes/
Disallow: /blocks/
Disallow: /modules/
Disallow: /language/

Any ideas?

Ok, that is not the default for RavenNuke. Possibly try this:

User-agent: *
Crawl-delay: 5
Disallow: /abuse/
Disallow: /admin/
Disallow: /blocks/
Disallow: /cgi-bin/
Disallow: /classes/
Disallow: /db/
Disallow: /images/
Disallow: /includes/
Disallow: /language/
Disallow: /modules/
Disallow: /ShortLinks/
Disallow: /themes/
Disallow: /admin.php
Disallow: /config.php

This is definitely an unusual one.
As far as I'm aware the Google Mediapartner useragent is specifically for the adsense bot but it makes perfect sense to me to remove that one from the robots.txt any way.
Something you might want to do is check Nuke Sentinels tracked referrers - if NS is seeing the Google bot referrer and your still not getting indexed this might provide evidence that your site or server has somehow been blacklisted by Google.

If you are not seeing any referrers for Google than it is certainly worth digging deeper - perhaps you may have (at some time) had an admin link accessible and Google might have been blocked for trying to access it.

Hey Gard,

Thanks for the reply,

I have taken out the suggested line from my robots.txt file.
I have also check NS and the only line with google in it is as follows:
http://www.google.com/search?hl=en&safe=off&pwst=1&q=%22houses+for+rent%22+%22austin%22&start=20&sa=N

No evidence of google bot at all.
I also checked my http referrers module and nothing there either.

Do you have any suggestions as to who to email over at google?


Thanks

Actually, I would also look for the range of IPs that are for the GoogleBot to see if you have one blocked in that range. You can also set that range as protected. I think some of them already are, but not sure. You may need to do a search on Google for their latest bot IP addresses.

Ip numbers doesn't seem to be a problem.

Here are the last result from my sitemap details
Property Status
Sitemap type Web
Format Sitemap
Submitted Apr 28, 2008
Last downloaded by Google Jun 23, 2008
Status OK
Total URLs in Sitemap 21
Indexed URLs in Sitemap 0

Does anyone now how to get a hold of google to figure out
what the problem is?

It absolutely could still be the IP addresses of the BOTs themselves. The Sitemap download could very well be coming from a completely different set of IP addresses. Have you checked for the GoogleBot IP addresses against your blocked IPs?

You should be able to find the list of IP addresses by searching google for "googlebot ip addresses" or something like that.

Montego,

I got an interesting reply from a post i put over at the google group forum

Here is what one guy said

You can see the complete post here:
http://groups.google.com/group/Google_Webmaster_Help-Indexing/browse_thread/thread/60b56c08f28b2d70/2f4ad38b8d6816dd?lnk=gst&q=you4eea#2f4ad38b8d6816dd[/quote]

[quote]
What total CRAP.

Have you even spent one millisecond considering exactly what that
sitemap says?

Every page changed today? Really? When the Googlebot crawls and
compares what it gets with what it got last time, is it going to thank
you for telling it about a real change, or is it going to find the
page hasn't changed today?

And every page changes every day? Really? Is that the truth -
because it's what you're telling Google.
The best thing you can do with this particular sitemap is delete it.
"end quote"

Any Suggestions?
or do you think I need to still check the ip address?

I think he over reacted. Yes, it is better to give Google a more accurate picture of your site i.e. if you have constantly changing content then 1 day is ok for a refresh, if it changes less often a week is long enough etc.
To imply that Google is not crawling your site because of the sitemap setting being 1 day is just well, horse manure.

Google is not crawling your site for a reason and so far the only reason seems to be a possible blocked IP range. There is nothing on your site that would turn Google away as far as content is concerned so we are going with a blocked range as the most likely suspect.
There are others, if for example the domain has been previously owned and they told Google not to index it but I don't think that applies in this case.

I don't think it is Google just being plain slow either, I can get a new site indexed in less than 24 hours every time.

Guardian is correct. Although it probably doesn't help that the date time is current, that certainly wouldn't stop Google from reading it. It might lower your rankings, but I think it's just ignored as far as placement goes (though I'm sure your "friend" on Google groups knows the algorithm). Hopefully, the next update to the sitemap (next on the list after RNYA) will include the ability to configure both the refresh setting and whether or not to display the date time (and improvements for displaying the date time).
_________________

Thanks Guys,

Yeah, I'm pretty sure he is over reacting, and NO NOT MY FRIEND

Ok, so this domain was actually owned previously, and was my last thought on the reason why it isn't indexing.

Just to be sure we are on the same page, it does appear as though it is crawling, but not indexing. I can see that it is crawling in my webmaster tools, and it see all the new URL's, but it is not indexing.

So I checked all the ip's that are from US in Sentinal Module and none of the appear to be from google. I will do some more investigation.

Also, in my webalizer stats, I'm not seeing any googlebot stuff
Here are my stats dreamteamrealty.com/webalizer

Thanks for all the help.

Looks like you might have been punished by Google for the previous owners abuse (they probably had a CNAME record pointing to Slashdot)
http://www.domaintools.com/historic-thumbnails/?domain=dreamteamrealty.com