Wikia launches open source search engine 8 Jan 2008

Wikia Search, the open source search engine project backed by Wikipedia co-founder Jimbo Wales, officially launched yesterday after more than a year of development.

Wikia Search was originally announced at the end of 2006, but for all the talk of "openness" and "transparency", there were no real details about how the system would work. While the project came into sharper focus in July 2007, when Wikia acquired the Grub indexing system and released it under an open source license, it took until last month to enter a "pre-alpha" state suitable for limited user testing.

In an interview with ars technica, Wales says that "Wikia Search will be open to users in two ways. First, anyone can rank search results by using a simple five-star system. Second, the backend systems that power Wikia Search will all be open-source projects like Grub, and anyone with a bent for coding can get their hands dirty with the actual search algorithms that operate in the background."

Sounds like some major battles lie ahead with SEO spammers then! (It's not hard to imagine these unscrupulous parties hiring thousands of sweat shop workers to give their content millions of five star rankings, for example).

Interestingly, Wales sees "algorithmic transparency" as Wikia Search's ultimate guiding principle and makes a great point about how search engine algorithms are essentially little more than political / editorial statements:

"Other search engines keep their algorithms hidden in an attempt to keep site operators from gaming the system and competitors from seeing how it works. But Wales believes that it's important to know how the algorithm works, since each algorithm makes an editorial statement.

"This isn't the way that most people think about algorithms, which seem like the ultimate expressions of unbiased machine processing. Wales turns to an example to make his point: searching for 'Thomas Jefferson' might return a page of ten links. Those links ostensibly represent the most important information about him, but are they? Who says so? The way the algorithm was tuned and constructed means that even these results are 'an editorial statement' about the sorts of pages that are important. Algorithms of this kind aren't neutral any more than the people who create them are."