Blog Post: Updated: When you accidentally index 5.5 billion pages of spam, something's rotten in the state of Googledom. Google Search is good. Maybe too good?
Reports from the
Monetize blog last weekend demonstrated that a spammer -- since revealed to be a Moldavian malcontent -- managed to get 5.5 billion spurious pages indexed by Google. Google has removed the pages from their index, and are reportedly working on a solution, according to several sources.
Google, however, has not returned requests for comment.
The Moldavian spammer's pages, which were created using a variety of
readily-available and easily learned techniques, were also partially indexed by Yahoo. MSN's new live search, meanwhile, indexed only 18 pages.
"It is important for the entire Search industry to continue to focus on reducing spam, and this an area our development teams are working on as we build out Windows Live Search," said Justin Omer, senior product manager for Windows Live Search, in an e-mail to Google Watch. "Our goal is to help our customers find the information they are looking for vs. useless information that clog results pages and reduces relevancy."
Yahoo, for its part, said in an email to Google Watch that the number of spammer pages indexed was "several orders of magnitude lower" than the numbers reported on John Battelle's Searchblog. However, that particular entry did not specify how many spurious pages Yahoo indexed. Yahoo has not returned a request to clarify this statement.
Representatives from Yahoo did say that "a very tiny percentage of our index was impacted by these spam sites and most of these spam sites are already eliminated. We believe our user experience was not compromised."
The big problem, however, remains to be Google, which apparently indexed more spurious domains faster than any other search engine.
Some critics see Google's indexing problem as more evidence that their core business is hurting. Others have noticed that Google is having problems with their site: operator, which allows a user to search a single domain or multiple domains for keywords.
"Since they introduced the new infrastructure, it has been a state of constant change," said Merge Database's Alex Firmani in an IM interview. "It is like swapping out the engine of a car while it is still running... in fact, going 80 MPH down the highway. So now Google has swapped the engine and all sorts of flaws are showing up. Flaws that were not evident in their previous infrastructure, but are hurting the new methodology."
Firmani also noted that Google, which recently made major changes to its indexing engine, dropped quite a few non-spam sites from its index at the same time it was indexing the spurious results.
In some cases, Google has also reverted to two-year old page caches for sites.
In April, Google CEO Eric Schmidt told the New York Times that Google's indexing servers are running near capacity. "Those machines are full. We have a huge machine crisis."
The Times revealed recently that Google is building a large facility in Oregon, presumably to house more computing power.
This article originally appeared on Steve Bryant's blog, GoogleWatch.
