Rebuild search with trust
Finally there is a thing people can agree on:
- 2023-08-28, OSNews: The end of the Googleverse
- 2023-07-28, Cory Doctorow: Microincentives and Enshittification
- 2023-10-03, Cory Doctorow: Google’s enshittification memos
- 2024-01-15, Tim Bray: Mourning Google
Apparently, Google Search is not good anymore. And I’m not the only one thinking about decentralization to fix it:
Honey I federated the search engine - finding stuff online post-big tech - a lightning talk at the recent chaos communication congress
The speaker however did not mention, that there have already been many attempts at building distributed search engines. So why do I think that such an attempt could finally succeed?
- More people are searching for alternatives to Google.
- Mainstream hard discs are incredibly big.
- Mainstream internet connection is incredibly fast.
- Google is bleeding talent.
- Most of the building blocks are available as free software.
- “Success” depends on your definition…
My definition of success is:
A mildly technical computer user (able to install software) has access to a search engine that provides them with superior search results compared to Google for at least a few predefined areas of interest.
The exact algorithm used by Google Search to rank websites is a secret even to most Googlers. Still it is clear, that it relies heavily on big data: billions of queries, a comprehensive web index and user behaviour data. - All this is not available to us.
A distributed search engine however can instead rely on user input. Every admin of one node seeds the node ranking with their personal selection of trusted sites. They connect their node with nodes of people they trust. This results in a web of (transitive) trust much like pgp.
For comparison, imagine you are searching for something in a world without computers: You ask the people around you. They probably forward your question to their peers.
I already had a look at YaCy. It is active, somewhat usable and has a friendly maintainer. Unfortunately I consider the codebase to show its age. It takes a lot of time for a newcomer to find their way around and it contains a lot of cruft. Nevertheless, YaCy is a good example that a decentralized search software can be done even by a small team or just one person.
I myself started working on a software in Haskell and keep my notes here: Populus:DezInV. Since I’m learning Haskell along the way, there is nothing there to see yet. Additionally I took a yak shaving break to learn nix.
By the way: DuckDuckGo is not the alternative. And while I would encourage you to also try Yandex for a second opinion, I don’t consider this a solution.