nukeSEO.com - PHPNuke SEO Search engine optimization, professional tools including nukeSEO, nukeSPAM, nukeFEED, nukePIE, nukeWYSIWYG and more

 

. Welcome to nukeSEO.com  ! 
.
.
.

Tag This



.
nukeSEO.com: Forums


 Forum FAQForum FAQ   SearchSearch   UsergroupsUsergroups   ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Google Not Crawling Site
 Goto page 1, 2  Next
 
Post new topic   Reply to topic    nukeSEO.com Forum Index -> nukeSEO (tm)
View previous topic :: View next topic  
Author Message
You4eea
webmaster


Joined: May 27, 2006
Posts: 11

PostPosted: Fri Jun 06, 2008 8:34 am    Post subject: Google Not Crawling Site Reply with quote

Hello World,

Well in short my problem is as follows,
Google is unable to crawl my site. I don't know why or where to even start.

I have created several nuke sites and they all can be crawled accept one.
For example, I'm testing it by using the search tool at googles webmaster central here Only registered users can see links on this board! Get registered or login!

One nuke site works great using the following perams....
URL = Only registered users can see links on this board! Get registered or login!
SearchTerm = *

Yet when I try
URL = Only registered users can see links on this board! Get registered or login!
SearchTerm = *
Nothing shows up, yet there is content there.

I'm using RavenNuke(tm) Version 2.20.01
I have submited the nukeseo sitemap and it is working and verified, yet google has not yet indexed Only registered users can see links on this board! Get registered or login!
I submited it about 32 days ago.

Any suggestion?

You4eea
  
Back to top
View user's profile Send private message
You4eea
PostPosted: Fri Jun 06, 2008 5:16 pm    Post subject: Re: Google Not Crawling Site Reply with quote

I think I may have found the problem.

I used this tool Only registered users can see links on this board! Get registered or login!

and searched other sites hosted on this server and I even changed the index.php to index.html on Only registered users can see links on this board! Get registered or login! and found that it just wasn't getting reached by the above tool where as all other domains were.

I have had whois data get corrupted before due to some hosting company botching up the transfer process.

So I'm transfering the registrar and I will update you all to see if that fixes it.
  
Back to top
montego
webmaster


Joined: Dec 26, 2005
Posts: 254

PostPosted: Sat Jun 07, 2008 7:49 am    Post subject: Re: Google Not Crawling Site Reply with quote

Interesting. I have never had this issue of not crawling. Curious as to what your robot.txt file looks like. If it is an unmodified version from RavenNuke 2.20.01 then that should be fine. Definitely keep us posted.
  
Back to top
View user's profile Send private message Visit poster's website
You4eea
PostPosted: Fri Jun 20, 2008 2:10 pm    Post subject: Re: Google Not Crawling Site Reply with quote

Hey montego,

I'm still having the same issue, very strange.
After reading your post above I went and copied the robots.txt file
from one of my other sites and same problem. It just isn't indexing
it. In the webmaster tools -> Sitemaps the overview gives this info:

Sitemaps
dreamteamrealty.com
Sitemap statistics:
Total URLs: 18
Indexed URLs: 0

Here is what my robots.tx file looks like,
User-agent: Mediapartners-Google*
Disallow:
User-agent: *
Disallow: /admin.php
Disallow: /admin/
Disallow: /images/
Disallow: /includes/
Disallow: /themes/
Disallow: /blocks/
Disallow: /modules/
Disallow: /language/

Any ideas?
  
Back to top
montego
PostPosted: Sat Jun 21, 2008 7:12 am    Post subject: Re: Google Not Crawling Site Reply with quote

Ok, that is not the default for RavenNuke. Possibly try this:

User-agent: *
Crawl-delay: 5
Disallow: /abuse/
Disallow: /admin/
Disallow: /blocks/
Disallow: /cgi-bin/
Disallow: /classes/
Disallow: /db/
Disallow: /images/
Disallow: /includes/
Disallow: /language/
Disallow: /modules/
Disallow: /ShortLinks/
Disallow: /themes/
Disallow: /admin.php
Disallow: /config.php
  
Back to top
Guardian
webmaster


Joined: Dec 25, 2005
Posts: 364
Location: Vsetin, Czech Republic

PostPosted: Sat Jun 21, 2008 8:49 am    Post subject: Re: Google Not Crawling Site Reply with quote

This is definitely an unusual one.
As far as I'm aware the Google Mediapartner useragent is specifically for the adsense bot but it makes perfect sense to me to remove that one from the robots.txt any way.
Something you might want to do is check Nuke Sentinels tracked referrers - if NS is seeing the Google bot referrer and your still not getting indexed this might provide evidence that your site or server has somehow been blacklisted by Google.

If you are not seeing any referrers for Google than it is certainly worth digging deeper - perhaps you may have (at some time) had an admin link accessible and Google might have been blocked for trying to access it.
  
Back to top
View user's profile Send private message
You4eea
PostPosted: Sun Jun 22, 2008 9:49 am    Post subject: Re: Google Not Crawling Site Reply with quote

Hey Gard,

Thanks for the reply,

I have taken out the suggested line from my robots.txt file.
I have also check NS and the only line with google in it is as follows: Only registered users can see links on this board! Get registered or login!

No evidence of google bot at all.
I also checked my http referrers module and nothing there either.

Do you have any suggestions as to who to email over at google?


Thanks
  
Back to top
montego
PostPosted: Sun Jun 22, 2008 11:17 am    Post subject: Re: Google Not Crawling Site Reply with quote

Actually, I would also look for the range of IPs that are for the GoogleBot to see if you have one blocked in that range. You can also set that range as protected. I think some of them already are, but not sure. You may need to do a search on Google for their latest bot IP addresses.
  
Back to top
You4eea
PostPosted: Mon Jun 23, 2008 1:29 pm    Post subject: Re: Google Not Crawling Site Reply with quote

Ip numbers doesn't seem to be a problem.

Here are the last result from my sitemap details
Property Status
Sitemap type Web
Format Sitemap
Submitted Apr 28, 2008
Last downloaded by Google Jun 23, 2008
Status OK
Total URLs in Sitemap 21
Indexed URLs in Sitemap 0

Does anyone now how to get a hold of google to figure out
what the problem is?
  
Back to top
montego
PostPosted: Tue Jun 24, 2008 6:19 am    Post subject: Re: Google Not Crawling Site Reply with quote

It absolutely could still be the IP addresses of the BOTs themselves. The Sitemap download could very well be coming from a completely different set of IP addresses. Have you checked for the GoogleBot IP addresses against your blocked IPs?

You should be able to find the list of IP addresses by searching google for "googlebot ip addresses" or something like that.
  
Back to top
You4eea
PostPosted: Tue Jun 24, 2008 9:16 am    Post subject: Re: Google Not Crawling Site Reply with quote

Montego,

I got an interesting reply from a post i put over at the google group forum

Here is what one guy said

You can see the complete post here: Only registered users can see links on this board! Get registered or login!

[quote]
What total CRAP.

Have you even spent one millisecond considering exactly what that
sitemap says?

Every page changed today? Really? When the Googlebot crawls and
compares what it gets with what it got last time, is it going to thank
you for telling it about a real change, or is it going to find the
page hasn't changed today?

And every page changes every day? Really? Is that the truth -
because it's what you're telling Google.
The best thing you can do with this particular sitemap is delete it.
"end quote"

Any Suggestions?
or do you think I need to still check the ip address?
  
Back to top
Guardian
PostPosted: Tue Jun 24, 2008 9:59 am    Post subject: Re: Google Not Crawling Site Reply with quote

I think he over reacted. Yes, it is better to give Google a more accurate picture of your site i.e. if you have constantly changing content then 1 day is ok for a refresh, if it changes less often a week is long enough etc.
To imply that Google is not crawling your site because of the sitemap setting being 1 day is just well, horse manure.

Google is not crawling your site for a reason and so far the only reason seems to be a possible blocked IP range. There is nothing on your site that would turn Google away as far as content is concerned so we are going with a blocked range as the most likely suspect.
There are others, if for example the domain has been previously owned and they told Google not to index it but I don't think that applies in this case.

I don't think it is Google just being plain slow either, I can get a new site indexed in less than 24 hours every time.
  
Back to top
kguske
Site Admin
Site Admin


Joined: May 12, 2005
Posts: 875

PostPosted: Tue Jun 24, 2008 10:53 am    Post subject: Re: Google Not Crawling Site Reply with quote

Guardian is correct. Although it probably doesn't help that the date time is current, that certainly wouldn't stop Google from reading it. It might lower your rankings, but I think it's just ignored as far as placement goes (though I'm sure your "friend" on Google groups knows the algorithm). Hopefully, the next update to the sitemap (next on the list after RNYA) will include the ability to configure both the refresh setting and whether or not to display the date time (and improvements for displaying the date time).
_________________
  
Back to top
View user's profile Send private message Visit poster's website
You4eea
PostPosted: Tue Jun 24, 2008 12:42 pm    Post subject: Re: Google Not Crawling Site Reply with quote

Thanks Guys,

Yeah, I'm pretty sure he is over reacting, and NO NOT MY FRIEND Smile

Ok, so this domain was actually owned previously, and was my last thought on the reason why it isn't indexing.

Just to be sure we are on the same page, it does appear as though it is crawling, but not indexing. I can see that it is crawling in my webmaster tools, and it see all the new URL's, but it is not indexing.

So I checked all the ip's that are from US in Sentinal Module and none of the appear to be from google. I will do some more investigation.

Also, in my webalizer stats, I'm not seeing any googlebot stuff
Here are my stats dreamteamrealty.com/webalizer

Thanks for all the help.
  
Back to top
Guardian
PostPosted: Wed Jun 25, 2008 12:38 pm    Post subject: Re: Google Not Crawling Site Reply with quote

Looks like you might have been punished by Google for the previous owners abuse (they probably had a CNAME record pointing to Slashdot) Only registered users can see links on this board! Get registered or login!
  
Back to top
Display posts from previous:       
Post new topic   Reply to topic    nukeSEO.com Forum Index -> nukeSEO (tm) All times are GMT - 5 Hours
 Goto page 1, 2  Next
 Page 1 of 2

 

Jump to:   
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum

Powered by phpBB © 2001-2008 phpBB Group


Page Generation: 0.12 Seconds