A federated search engine for frequently updated Webpages. Although Google is great for large and small static sites that change infrequently, its method sucks for frequently updated small sites. Why? Google doesn't know when a site has been updated. The result: out of date results and added expenses for needlessly crawled sites. This was very apparent during 9-11, when the need for up to the minute news was vital.
A better system is one where a site informs the search engine when it is ready to be crawled (even 10 times a day if it is active). Dave already announced that we are doing that with Weblogs.com. An even better system, suggested by Dries Buytaert, is one that shares the crawling duties with other systems and tells them to crawl sites that have inform the central search engine that they are ready to be crawled.
The benefits to this system are enormous. A search engine that has an up to the minute index. A system that doesn't transfer costs to crawled sites needlessly. It's the kind of system that will make all the difference during the next big problem we experience. Some added functionality that could be added over time:
1) Ranked results by site popularity (traffic).
2) Ranked results by time of update.
3) Ranked results by number of times it was referenced (referrers or blogdex results for specific topics referred to).
It's nice to see the technology community starting to respond to the situation. [John Robb's Radio Weblog]