This site is now slightly over a month old I'm thrilled by the number of visitors that have already found their way here. What I find perplexing, however, are the number of search engine spiders and other "bots" that find frequent this site daily.

"A web crawler (aka spider, robot or bot) is a computer program that scours the web gathering content. Some crawlers are specific in what they are looking for, while others are just interested in gathering as much content as possible."

Super Fast Computer is one of the few sites you'll find that makes its visitor logs available to the public. I glean all kinds of interesting information from the logs. For example, the vast majority of you are using Internet Explorer to browse this site, while only a small percentage of people are using (yuck) Netscape. Sadly, Opera users only account for about 1% of visitors.

My other websites are visited about once a month from search engines such as Google and AOL. Because of the fact that I'm constantly updating this site, Google sends its search spiders to this site every other day-- sometimes daily. (Even stranger is the fact that, despite all of the visits, Google is not currently displaying this site in its index, unless you trick it).

Within the two weeks of launching this site, I found that it was being spidered by so many bots that I took the step of telling my logger to ignore about a dozen of them. More bots continue to find the site, however, and I can't keep up with them all. As this graph shows, almost 32% of the visitors logged are using an "Unknown Platform"; the majority of these you can bet are non-human visitors. Look at them all!

You would think that for every search engine or weblog index that is listing this site, it would translate into lots of human visitors following links and showing up here. It does not appear to be so. I find myself wondering who is paying for all of these computers, and for what purpose?

08 February 2003  |   permalink