July 18, 2008
Yahoo Search Scientists Continue Amid Turmoil
Ever since February, when Microsoft first announced its intention to buy Yahoo, Andrew Tomkins, Yahoo's chief search scientist, has been receiving worried calls from his mother.
Will Microsoft buy Yahoo? Will Microsoft buy Yahoo's search business? Will her son, a widely recognized search genius, be out of a job?
Tomkins has tried to reassure her: "We are building stuff, and it is all coming together."
While news reports of the takeover attempt may seem grim to outsiders, inside the embattled Sunnyvale company a particularly stubborn breed of Yahoo is thriving. Amid the turmoil of high-profile executive resignations and recurrent reorganizations, Yahoo's research scientists have staged a quiet coup.
During the past three months, Yahoo has released two major search products and began rolling out a new advertising-management platform. Behind the bustle of business activity is a brain trust of academics that Yahoo began gathering together more than three years ago.
"When we got started with the research lab one of our concerns was, 'Are we going to create a lab that is going to be so far out there and so disconnected from the day-to-day realities that we are just not going to see any return on it?"' said David Filo, Yahoo's co-founder and chief Yahoo. "The feeling now is, if anything, they are too close to the actual business and their ideas are too influential.
"I think from the company's perspective it has worked out
Prabhakar Raghavan, head of Yahoo Research, began building the lab in July 2005. In addition to search scientists like Tomkins, Raghavan took the unusual step of hiring a team to focus on microeconomics.
Raghavan had an idea that they would be able to contribute, since the core of Yahoo's ability to profit from Web search rested on microeconomic principles. Within a year, they had a chance to prove their value as it became clear that Yahoo was getting clobbered by Google, which was able, by some estimates, to earn twice as much as Yahoo per search.
Michael Schwarz, an economist who lists Stanford University, University of California-Berkeley and Harvard University on his resume, suggested Yahoo try market-reserve pricing, basically using microeconomic theory to set minimum bids, based on the work of Roger Meyerson, who won the Nobel Prize in 2007. Yahoo rolled Schwarz's suggestion into its advertising at the end of April.
"There's a tremendous thirst for having the best and the brightest and sucking their ideas into the product as fast as we can," Raghavan said.
Yahoo is counting on these ideas to drive dramatic improvements in its financial performance. In a March presentation to investors, which was widely criticized as unrealistic, Yahoo predicted it could double operating cash flow, a measure of profitability, by 2010.
Equally bold is Yahoo's plan to unseat Google from the Internet search throne.
Just as open-source software eroded Microsoft's dominance of corporate computing, so too could open-search software cut into Google's growing lead, or at least that is what Yahoo is hoping.
In May, Yahoo released a developer platform, called SearchMonkey, that lets programmers customize how Yahoo's search results are displayed. This month, Yahoo went even further, providing developers with keys to its search kingdom in the form of a guide to software functions that let anyone access Yahoo's massive search infrastructure, and, in essence, build their own search service. (The software is called BOSS.)
The idea came from an economist, Preston McAfee, until recently a professor at the California Institute of Technology.
The idea was to basically remove all barriers to entry for software developers who have ideas about how to improve search, but lack the $300 million that is necessary to get into the game.
There was a risk: An innovative start-up could eat Yahoo's lunch while dining on Yahoo's platform. But Filo and other executives thought Yahoo was more likely to benefit from working with developers across the globe.
"Admittedly, we are not the market leader," Raghavan said. "So when we were getting ready to do this, the greatest risk was not to us."
Filo said BOSS is part of a more sweeping strategy to open up as much as Yahoo as possible, including its advertising business and popular services like Yahoo mail. It's an approach that the scientists and engineers love.
Andrei Broder, a vice president for computational advertising at Yahoo and a leading search scientist, said the reaction to the relentless barrage of takeover news has varied from employee to employee. "Some individuals are passionate readers of Valleywag," an online gossip site, he noted.
But the call of doing ground-breaking research is strong and, if anything, the takeover battle has pushed Yahoo's scientists into a frenzy as they race to finish projects that are under way.
"It's like a good book," Broder said. If you don't have time, you try to finish it quickly. "You want to get to the end. You want to see how these things play out."