After All the Agony, Has Site Search Become a Free Lunch?
By Arnold, Erik
Thanks to Google-and, yes, even the dramatic improvement of Yahoo! Search and Microsoft’s Live Search over the past few years – people now use search whenever and wherever they find it. Studies show that more than 50% of users go straight to the site search box instead of navigating through links. This fact has led to consternation among companies, interactive ad agencies, and other groups who spend so much time pining over every detail of their websites but gladly hand off the search feature to the corporate IT departments as if it were a pure technology issue. As IT departments usually have lots of problems to worry about, staff tend to “turn on” the search baked into the content management system and move on with their lives. Times have changed. Statistics show irrefutably that people now use the search function, and controlling search is more difficult than controlling a website. People’s reactions to this issue vary widely. Some customize the search in the content management system; some purchase a Google Search Appliance; and some spend millions of dollars on enterprise search systems meant to organize intranet content.
However, rarely do organizations take a step back and consider using the actual web search engines that most people use already, namely, Google, Microsoft, and Yahoo!. Few people realize or understand that all three of these massive internet companies have opened up their infrastructure for use on any website. I know that I am not alone in leveraging Google, Yahoo!, or Microsoft to find content on a specific website because the site search yielded poor results, yet so few use the full capabilities of these services to deliver their own site search.
I grew up in Kentucky and, as I write this column, the Derby is corning. However, every year I get frustrated by a fruitless quest on the Derby site [http://www.kentuckyderby.corn] to actually get tickets to the event. The folks at the Derby obviously spend a lot of time and money on the website, but the search for tickets still falls prey to the scenario described above.
A search for “tickets” does not lead me to the path that most people would expect. Armed with the commercial tools that have developed over the past year, I decided to see if I could run the website and leverage one of the commercial web search engines as my site search.
Google’s Custom Search [http://www.google.com/coop/cse] engine is the most popular of the services that offer access to the commercial web indexes. It continues to gain more and more popularity on government and smaller websites, as well as some larger publishers, such as MacWorld.com.
With all these systems, it helps to have some knowledge of how search works, as when I log in to create a new search engine, I get the screen shown in Figure 1 below.
First I populate the fields. Though I only wish to search Kentuckyderby.com, I could place up to 5,000 sites in the Custom Search Engine if I wanted to expand the search. Here we go. Voila! I now have a functioning search engine. Wow, that was fast! And the results are already far superior to the current search system (see Figure 2 below).
I can customize the search experience using the CSE control panel (see Figure 3 top right).
One of my main complaints with the KentuckyDerby.com website is that it mixes blog posts with the official information on the site. Since I sought official Derby information, having to delve through random blog postings from random horse racing gurus was disconcerting. With the CSE control panel, I can separate the blog posts from the official Kentucky Derby content (see Figure 4 at right).
From the preview tab, I can see my Blog category above the search results and no blog listings appear on the first of my search results pages. However, clicking on the blog tab, I can see the related postings to my keyword. While that may seem and sound simple, in a just a few minutes, I have created a better site search engine than the one that exists on KetuckyDerby.com.
The CSE Control Panel offers the following customizations:
* Look and Feel offers some basic customizations of the search engine results pages.
* Code enables web publisher to host the search pages as opposed to Google.
* Collaboration enables others to share in the search by adding websites or comments (foundation for a social search application).
* Make Money enables the AdSense Program.
* Business Edition enables websites to pay for the raw XME, which enables publishers to completely customize the search results.
* Advanced allows you to further customize search results.
In sum, most tools available from enterprise systems are available with CSE.
Reporting
A few months ago, Google announced that Google Analytics, the popular free website measurement service, could now analyze site search results in addition to web traffic. This important feature can show web managers, few of whom analyze search results, how people search on their site.
From your Custom Search Engine home page, a statistics “tab” will hopefully show you your search statistics. At the present time, though, the tab is a link to Google Analytics. Google Analytics is a sophisticated product. Many Google Analytics users do not know about this site search upgrade or that it is not turned on by default (see Figure 5 below).
To activate site search, select the “Do Track Site Search” option and input the search parameter (q for Google custom search). You can use Google Analytics for any site search engine that you use, even the services listed below.
Once you complete these steps, you will see search statistics in the following categories: search queries usage, start pages, destination pages, categories, and trending (see Figure 6 at left).
Previously, to obtain such data, a website would have had to purchase an advanced web analytics tool such as Omniture, as most search engine vendors do not provide adequate reporting capabilities.
Advanced Features
Google (and Microsoft and Yahoo!) offer the ability to customize your search service even more through APIs (application programming interfaces) found at https://code.google. com. The Google Ajax API documentation allows publishers to manipulate custom search results in too many ways to mention. While this falls out of the “easy” category for most of us, a good webmaster would think of integrating APIs as “easy.”
Microsoft
Microsoft offers customized search in the form called search “macros.” While the name did not go through the marketing department and the feature seems amazingly hard to find, it is a good service. Live.com is the latest name for Microsoft’s search engine, whereas http://www.msn.com is still the “portal” home, go to http:// search.live.com for searching. For Microsoft’s custom search engine, go directly to http://search.live.com/macros. (see Figure 7 below.)
Following pretty much the same procedure, I again quickly created my Kentucky Derby site with good results (see Figure 8 below). In fact, Microsoft generated very similar results to my Google CSE. This should not prove too surprising, as both Microsoft and Yahoo use similar algorithms to Google.
Reporting
For Microsoft devotees, Microsoft acquired Deep Metrix in 2007 and currently has the program in private beta. Deep Metrix had a lot of fans and the early results are positive about Microsoft’s upcoming analytics service. However, currently, turnkey reporting is not available.
Advanced Features
Microsoft does not provide the same level of customization through a web-based interface as Google, but it does provide an interactive SDK (software development kit) at http://dev. live.com/ livesearch/sdk. While SDKs are targeted for programmers, the web- based SDK makes it possible for search managers to put in parameters and see results, as shown in Figure 9 below.
You can find Microsoft’s downloadable SDK for traditional development work at http://www.rnicrosoft.corn/downloads/ details.aspx?FamilyID=c271309b-02de-42a7-b23e-el9f6866 7197&DisplayLang=en..
Yahoo!
Last, but not least, Yahoo! offers Search Builder in its traditional user-friendly design. At the intro page, I immediately selected site search and input my domain name (see Figure 10 below).
Yahoo! seems to target smaller sites. It allows little customization of search engine results from its control panel (see Figure 11 top right).
Reporting
Yahoo! does offer a nice, integrated reporting interface for its Search Builder service (see Figure 12 below). Users can view the top referrals and top queries from their Yahoo! search box. While this is not as advanced as Google (and potentially Deep Metrix from Microsoft), it does provide immediate access to the most important information.
Advanced Features
While it may seem like ancient history (a full 18 months ago), Yahoo! took a leadership position in opening up its search services to third parties through APIs and in creating sophisticated online development tools such as Pipes. Yahool’s developer website [http:// developer.yahoo.com/search] offers similar customization capabilities to http://code.google.com. A savvy programmer can customize the Yahoo! search capabilities to meet specific requirements, but it will require more programming then the Google CSE Control Panel or the Live Interactive SDK.
Catches?
For anyone fighting a search battle on a website facing the public, these services may sound amazing and, if you take a step back, they are incredible. All three of these companies empower any organization to tap into their web services (that cost billions of dollars to build) and use them to their advantage (not to mention the cost savings compared to other approaches). Most web surfers have a basic understanding that link popularity drives the relevancy of these services – the relevancy that your users expect from your public-facing search engine too. Public-facing websites generally draw low customer satisfaction when it comes to search, dissatisfaction that drives users back to Google, Yahoo!, or Microsoft. The pitfalls lie in two categories: terms of services and the content in the indexes. These three services are free, but all display advertisements next to the search results. Google has productized its CSE as the Custom Search Business Edition. This will turn off the Google advertisements that appear in the free version. You may view Google’s pricing tiers at http://www. google.com/ enterprise/csbe/index.html. The pricing seems reasonable when you consider the cost of purchasing search software to power your public- facing websites. Remember that the Google Custom Search Engine comes with the ability to integrate your AdSense account into the search engine results. So you can earn money while leveraging Google’s search results.
The others are not quite there yet. You will just have to live with the advertisements if you want to leverage their indexes to power your site search. In their terms of service, both Yahoo! and Microsoft provide contact information if you want to discuss other arrangements. It seems safe to say that any website that would generate a significant amount of queries could garner enough attention for a custom deal. Should a publisher wish to use the advanced features from the respective search engines, the threshold is 5,000 queries per day at Yahoo!. Google and Microsoft don’t have a set limit in their Terms of Service, but there is limitation language, and anything more than 5,000 queries per day will show up on their radar screen.
The negative lies on the actual content in the their indexes. If your content is not in their system, then it will not be found in your site search. While most public-facing websites make efforts to get their content in search engines, many only focus on their popular pages. If you decide to apply this approach to improve searching on your site, you must put much more effort on getting your information into the search engines and on search engine optimization (SEO).
New content can also be a problem. Search engines will not index your information immediately, and many times a business needs to update its index immediately. While there are workarounds when you have programmers leveraging the Advanced Features of the services, you must take note of your special needs and manage them.
Almost a Free Lunch
Well, as economists have been saying since the late 1940s, “There ain’t no such thing as a free lunch.” It’s even an acronym, TANSTAAFL. From one angle, the Kentucky Derby spent a lot of time and effort to create a nice looking, user-friendly website, but if no effort was placed on search, which close to half their users may use this year … Well, where’s my ticket?
That said, in a matter of minutes, all the major web services provided me with the tools to create a superior search experience. All that remained was to customize the look and feel into each template. Since, as the same economists who tell you TANSTAAFL will tell you, “Time is money,” you have to take that effort into consideration. So there goes that free lunch.
But for those many websites that have purchased expensive enterprise search systems to power public-facing websites (ecommerce excluded, no email on that please), this approach will still sound like a very happy meal. Publishers can save an enormous amount of money in terms of software licenses, hardware costs, and maintenance (their “footprint”).
While many reasons may still exist for your organization to use different systems, business managers must weight the pros and cons in using these commercial web search tools-unless, of course, your managers only eat lunch at the Four Seasons.
In a matter of minutes, all the major web services provided me with the tools to create a superior search experience. All that remained was to customize the look and feel into each template.
by Erik Arnold
Director, Adhere Solutions, Inc.
Copyright Information Today, Inc. Jul/Aug 2008
(c) 2008 Searcher. Provided by ProQuest Information and Learning. All rights Reserved.