librarian
Custom meta-search
Search engines are the default way of finding information on the internet. Although there is a host of search engines for users to choose - from library catalogs to cooking portals - there is currently only a small number of dominant search engines that practically decide who finds what on the internet. This situation has the following disadvantages: 1) by designing their algorithms these dominant search engines influence our world view, 2) the huge amounts of user data they record, creates sever risks of data leaks and misuse, finally 3) search engines can misuse their market power to gain advantages in other lines of business (e.g. the mobile phone market).
Federated web search is a technology where users connect to a so-called broker which forwards their search request to suitable search engines and combines the results. Using federated search lessens the risks of few dominant search engines: it shows a blend of search results created by different algorithms, it prevents the search engine to record data of individual users, and its search results are usually more divers. Still, for federated web search to become widely used, it faces the following challenges: 1) while exploiting user behavior is known to improve search effectiveness, brokers exploiting this data also risk leaks and misuse, 2) as brokers typically serve many users, they are not able to include search engines for personal content, such as email, social media or cloud storage because the public broker cannot know the user’s credentials to access these services, finally 3) brokers consider for every user the same base set of search engines, while considering a more focused set of engines could improve search results, given the diversity of users.
To improve upon these challenges, while avoiding the disadvantages of dominant search engines, this project will investigate a radical change to the federated search architecture: users run a broker on their own computer using a browser plugin. In this architecture the broker can safely analyze the user's behavior to improve search results as the data is accumulated on a per-user basis on disconnected computers. Furthermore, the search requests forwarded to search engines use the user's credentials and thus can access search engines for personal data, such as email etc. Finally, starting from sensible defaults, each user can configure its broker with his or her individual needs.
Why does this actually matter to end users?
Search and discovery are some of the most important and essential use cases of the internet. When you are in school and need to give a presentation or write a paper, when you are looking for a job, trying to promote your business or finding relevant commercial or public services you need, most of the time you will turn to the internet and more importantly the search bar in your browser to find answers. Searching information and making sure your name, company or idea can be discovered is crucial for users, but they actually have little control over this. Search engines decide what results you see, how your website can be discovered and what information is logged about your searches. What filters and algorithms are are used remains opaque for users. They can only follow the rules laid out for them, instead of deciding on their own what, where and how to find the information they are looking for.
How can users get back into the driving seat and decide for themselves how they search and find online? One of the ways to get a better grip on search engines is to combine them into a single result page and letting users decide what engine they want to use in what way. That is what federated web search can do: your search request is forwarded to a selection of search engines and you get an aggregate of the results they find. Brokering search can give users more control over what they see and discover online, but not every federated web search solution actually lets users push the buttons themselves. They cannot always control what search engines are used and what is done with their search request, nor can they use the search broker to go through their own files like their email, calendar, or cloud drive.
Librarian is an effort to make federated web search really work for the user by running the broker solution that will be developed entirely on your own computer. That way your queries can be forwarded to online search engines (that you select yourself) but also be used to go through your own personal files stored on your computer. The search broker can also learn from what you are looking for and want to find, without giving away private details about your preferences to some unknown company. This makes federated web search a more privacy-friendly and user-centric alternative to the existing dominant search engines, giving users a fresh and more transparent view of the world wide web and the world itself.
This project was funded through the NGI0 Discovery Fund, a fund established by NLnet with financial support from the European Commission's Next Generation Internet programme, under the aegis of DG Communications Networks, Content and Technology under grant agreement No 825322.
