Calls: Send in your ideas. Deadline August 1st, 2022.

URL Frontier 2.0

Enterprise features for URLFrontier

URLFrontier provides a crawler-neutral API and service implementation for a crawl frontier, which can power various web crawlers independently from their implementation language and scalability. This API defines the operations that a web crawler typically does when communicating with a web frontier e.g. get the next N URLs to crawl, update the information about URLs already processed, change the crawl rate for a particular hostname, get the list of active hosts, get stats, etc… The aim of this project is to turn what is currently a working piece of software (the result of an earlier grant from NGI Zero Discovery) into an enterprise-grade solution. The improvements will mainly concern the service implementation, eg. monitoring/reporting, clustering/discovery and robustness/resilience. The project will improve the usability of the system by adding configurable logging and metrics reporting, improve the performance of the service for very large volumes of data by adding efficient parallelization across multiple nodes; and improve the overall robustness through more graceful failure modes and more efficient restarts .

Run by DigitalPebble Ltd

Logo NLnet: abstract logo of four people seen from above Logo NGI Zero: letterlogo shaped like a tag

This project was funded through the NGI0 Discovery Fund, a fund established by NLnet with financial support from the European Commission's Next Generation Internet programme, under the aegis of DG Communications Networks, Content and Technology under grant agreement No 825322.

Navigate projects

Please check out NLnet's theme funds, such as NGI Assure and NGI Zero Entrust.

Want to help but no money to spend? Help us by protecting open source and its users.

.