Building a Better Search Engine

By Bates, Mary Ellen

One way that we distinguish ourselves from Google-at least in the eyes of our clients or users-is by finding content that they can’t. But we can also beat the search engines at their own game by creating a customized filter that can be overlaid on search results. There’s nothing as compelling as showing clients that not only are we not threatened by search engines, but we are also able to improve upon them.

There are a number of tools for building customized, filtered versions of the leading search engines. They offer features that allow you to do the following:

* Limit the search to what you judge to be the most authoritative sites on the Web.

* Filter the search so that it retrieves only sites that have specific words or phrases in them.

* Accept recommendations for additional “best of breed” sites to add to your filter.

* Make your customized search engine available on your intranet Web site.

These filtering services are free and amazingly simple to set up. The most time-consuming part of the process is building the list of sites that you think are most important to your clients. In fact, you can build multiple customized search engines, specific to each client’s or department’s needs and interests. After you have customized the filter, you can copy the HTML code for your filtered search engine and insert it into any Web page. These customized search tools can be used to help users find information and to build ready reference search engines for your fellow info pros.

You can see a rough example of two different customized search engines by going to my Web site (www.batesinfo .com/testhtml). The first link is to Yahoo! Search Builder. You can search with any words you want: Behind the scenes, the word energy is appended to your search. The second search box uses Google Co-op. The difference between my customized search and regular Google search results is that Web pages from my collection of favorite sites are given more weight in the relevance ranking.

There are a number of choices for creating your customized search engine. The services I have found most useful include the following:

* Yahoo! Search Builder (http://builder.search.yahoo.com)

* Google Co-op (www.google.com/coop/cse)

* Gigablast Custom Topic Search (www.gigablast.com/cts.html)

* Rollyo (www.rollyo.com)

* Swiki (http://swicki.eurekster.com)

Each service works slightly differently. Your decision on which to use will be based on how you want to limit the search results and how much flexibility you need. Most let you test the search filters before going live-a handy way of catching unexpected results before your clients do.

Yahoo! Search Builder lets you limit the search to specific Web sites, automatically append key terms to each search run through your filter, or exclude specific sites or terms. You can include an entire domain or limit by subdomain or specific Web page. For example, you could focus the search on the U.S. Environmental Protection Agency’s Web site (www .epa.gov), or you could narrow it to just the area of the site that addresses the issue of radon (www.epa.gov/radon).

Google Co-op custom search tool lets you limit the search to specific sites or domains, or, as with my example above, you can simply have pages from specific sites ranked more relevant. One strange “feature” is that you are required to supply at least one keyword that must appear in all the retrieved search results. If you don’t want to limit searches by keyword, your best option is to include a noise word such as and.

Gigablast’s Custom Topic Search is a fairly rudimentary service; you specify the Web sites or subdomains you want searched, and only pages from those sites are included in the search results. Gigablast claims to support a customized search with up to 500 sites, but reports from the field suggest that performance degrades significantly when that many sites are included.

Rollyo, one of the earliest customized search builders, restricts you to a maximum of 25 Web sites in your filter, which makes it less attractive than Yahoo! or Google. However, many users’ “searchrolls” are public; you can browse the directory of searchrolls and see, for example, what one user considers the key sites for searching knowledge management-related issues (http://snurl.com/16×80).

Swiki, developed by Eurekster, offers a different take on the idea of filtered searching and relies on feedback from users of your search tool to refine the filtering. You initially supply keywords sites that your swiki will consider most relevant; what’s unusual is that Eurekster monitors what sites from the search results are clicked on by people using the swiki and ranks those pages more highly in future searches. This collaborative approach would work well in some settings, particularly libraries.

Mary Ellen Bates (mbates@batesinfo.com, www.batesinfo.com) needs a custom search filter for her office desk.

Copyright Information Today, Inc. Mar/Apr 2007

(c) 2007 Online. Provided by ProQuest Information and Learning. All rights Reserved.