The Good News on VoIP — Reliability Improves on Latest Keynote Study of Internet Telephone Service
Keynote Systems (Nasdaq:KEYN):
– Time Warner Digital Phone Ranks First for Both Reliability and Audio Clarity Among VoIP Providers, Edging out Winners in First Study
– Verizon DSL Most Reliable Network Carrier with Time Warner Cable Modem Service Winning as Carrier with Best Audio Clarity
– AOL TotalTalk, AT&T CallVantage, EarthLink trueVoice, Packet 8, Primus Lingo, SunRocket, Time Warner Digital Phone, Verizon VoiceWing, via:talk, Vonage and Vonics Digital Included in Study
– VoIP Competitive Monitoring Solution Announced
New competitive intelligence from the second Keynote VoIP Competitive Intelligence Study of Internet telephone service was released today by Keynote Systems (Nasdaq:KEYN), The Internet Performance Authority(R). The study revealed that while reliability has improved since the first Keynote study in June, 2005, the need still exists for considerable improvement in overall audio clarity from the leading providers of Internet telephone service (VoIP). In order to benchmark and rank the quality of consumer VoIP services, Keynote measured eleven leading VoIP service providers on critical performance factors that influence the end-user experience using Keynote’s VoIP Competitive Monitoring Solution.
Time Warner Digital Phone ranked first in the November/December Keynote study as the overall best VoIP service provider for both reliability and audio clarity, edging out the winning service providers, Vonage and AT&T CallVantage respectively, in the June study. Overall reliability is a computed index score based on performance measurements in three performance factors: service availability, average number of dial attempts, and dropped calls. Time Warner Digital Phone’s average availability for the entire measurement period was 99.91% contrasted with the lowest ranking provider coming in at 95.51%. Keynote found a significant gap in reliability and audio clarity between Time Warner Digital Phone and the lowest ranking providers.
Service Availability of VoIP providers has improved, with calls now completed 99.1% of the time across all VoIP providers in the November/December study. The average service availability of all VoIP providers was just 96.9% in June. The top providers were available 99.7% to 99.9% of the time — a significant improvement. All providers included in the last study and this one have improved in service availability.
Time Warner Digital Phone was one of three VoIP service providers in this study to achieve "toll quality" audio. The Mean Opinion Score (MOS) of three VoIP service providers exceeded 4.0 out of 5, a threshold that none of the VoIP service providers in the June study were able to achieve. Time Warner Digital Phone’s average MOS was 4.11, the best of all VoIP service providers measured in the study. There continues to be a large performance gap between VoIP providers with the best average audio quality and those with the worst average audio quality. Three providers, Time Warner, Vonage and Verizon, now have an average MOS over the 4.0 threshold for "toll quality," but five of the eleven providers in the study still have an average MOS lower than 3.5.
Although the top performers in the consumer VoIP services market have improved the quality of call audio, Keynote found that there was no significant change in the overall amount of audio delay present in VoIP calls over the past six months. Calls placed on VoIP phones continue to have up to twice as much audio delay as calls placed on traditional PSTN phones, which can lead to conversational disruption. The importance of conversation disruption grows with the importance of the information being imparted during a conversation. Overlapping conversation breaks down human ability to understand when to stop or start talking resulting in disruption of conversation and missed information, which creates frustration among users and inefficiency in business dealings.
The second Keynote VoIP Competitive Intelligence Study, just as the one conducted in June of 2005, assessed the market readiness of VoIP service in the New York and San Francisco metro areas, compared VoIP service providers for reliability and audio clarity over consumer cable, consumer DSL and business class network carriers, and evaluated network carrier performance on end user perceived call quality. The current study was expanded from six providers in the first study (see press release July 12, 2005) to eleven providers, including AOL TotalTalk (NYSE:TWX), AT&T CallVantage(NYSE:ATT), EarthLink trueVoice (Nasdaq:ELNK), Packet 8, Primus Lingo (Nasdaq:PRTL), SunRocket, Time Warner Digital Phone (NYSE:TWX), Verizon VoiceWing (NYSE:VZ), via:talk, Vonage and Vonics Digital. Data for this study was collected over a six week period, from November 18 through December 31.
The Keynote study provides an objective assessment of these critical performance factors that affect end-users’ perception of a VoIP service. While several providers and networks did well in certain areas, no single provider or network dominated the study in all metrics considered. The results of the study indicate that although overall reliability of VoIP service providers has improved, even the top providers are still performing below Public Switched Telephone Network (PSTN) standards and almost half the VoIP service providers measured in the study have an average MOS that is not just below the standard for "toll quality" audio, but well below the standards for wireless phones.
The Keynote study found that the underlying network carrier makes a difference in VoIP service reliability and audio clarity. To understand the impact of underlying network performance on call quality, the VoIP telephone calls were carried on three network providers in each city: Time Warner Cable (NYSE:TWX), Sprint (NYSE:PCS) and Verizon DSL (NYSE:VZ) in New York and Comcast Cable (Nasdaq:CMCSK), SBC DSL (NYSE:SBC) and Sprint business-class network in San Francisco. Verizon DSL was found to be the Most Reliable network carrier in the study, and Time Warner Cable Modem service was found to provide the Best Audio Clarity for calls made with all VoIP service providers measured in the study.
Analysts estimate that residential adoption of VoIP service will grow to over 26 million homes in 2008 in the US, up from 6.5 Million in 2004. The SMB and enterprise market forecasts are equally aggressive. Even so, VoIP reliability and audio clarity remain important factors that limit the widespread adoption of VoIP in consumer markets. A high rate of customer churn makes it difficult for carriers to break even. Some analysts estimate it can take as long as four years for some VoIP service providers to recoup the marketing and other costs associated with luring a subscriber.
"The consumer market VoIP performance gap in three areas studied by Keynote, service availability, average MOS and call delay could represent millions of dollars in lost revenue to service providers if customers get frustrated and change providers," said J. Jeffrey Nudler, senior analyst, Enterprise Management Associates. "The Keynote study is more exhaustive than any I have seen to date and the results, although not surprising, are very interesting with significant implications for the VoIP industry."
"As the number of VoIP providers and subscribers rises, users will have more choices and will switch companies if they do not receive the quality they desire," said Dharmesh Thakker, senior product manager at Keynote. "Indeed, Keynote found in its first study that VoIP providers were able to complete calls only 96.9% of the time with a Mean Opinion Score (MOS) of less than 3.5 — and that trend has continued for those ranking in the bottom of the second study. It should not be surprising then that 52% of respondents in a recent survey by Harris Interactive indicated VoIP call quality as a key barrier to adoption."
The study highlights the areas to which up-and-coming providers need to pay special attention in order to provide high service quality to end-users. Enterprises can also learn industry best practices they can implement during their branch-office VoIP deployments and when rolling out converged customer-care applications, to maximize employee productivity and retain high customer satisfaction.
With this study and the general release of the Keynote VoIP Competitive Monitoring Solution, Keynote continues to extend its expertise to embrace emerging technologies such as VoIP, streaming and wireless, which increasingly are core technologies of the enterprise. As Voice over Internet Protocol emerges as an influential technology that promises to cut consumer phone bills and enterprise communications expenses, the Keynote rankings help assess VoIP readiness and highlight market leadership among the various providers.
How the Study Was Conducted
The Keynote VoIP Competitive Intelligence Study was conducted using Keynote’s VoIP Perspective agent technology to evaluate critical performance factors that affect the consumer’s experience with Internet telephone service, in the New York and San Francisco markets. The study, based on an industry-standard methodology, helps service providers enhance their end-user experience and reduce customer attrition rates dramatically.
The study objectively ranked leading providers on critical factors that influence service quality for end-user experience when making VoIP calls. The study also analyzed various diagnostic metrics that contribute to the service performance factors and evaluated the impact of the underlying network carriers on the VoIP quality perceived by end-users. All calls were placed using Keynote VoIP Perspective(R) agents, which place actual calls across a VoIP service provider network to provide a realistic end user perspective.
Keynote placed local and long distance VoIP calls to destination phone numbers on a standard (PSTN) phone service. Calls were placed from San Francisco and New York once every 30 minutes on every VoIP provider and network carrier combination. A total of 190,000 calls were placed during the study, resulting in over 17,500 calls per VoIP service provider and over 30,000 calls per network carrier. Calls placed using VoIP providers were compared to traditional phone "toll quality" standards to determine what residential customers can expect when switching from traditional phone lines to VoIP.
To quantify the end-user experience, Keynote identified ten key performance factors based on multiple network and audio fidelity metrics that are collected during each call. These performance factors, to which every VoIP provider and network carrier should pay attention to ensure quality calls, include:
– Service Availability — the percentage of call attempts that were successful
– Average number of call attempts — how many times, on average, a caller must dial to place a call
– Dropped Calls — the number of times a call terminated prematurely
– Outage Minutes — periods of extended service unavailability
– Average MOS (Mean Opinion Score) measured using the ITU-T P.862 standard algorithm also known as Perceptual Evaluation of Speech Quality (PESQ) Percentage of Calls With an Acceptable MOS
– MOS geographic uniformity — a measure of the uniformity of MOS across different calling routes
– Average Audio Delay — The number of milliseconds of delay from one end of the call to the other
– Percentage of Calls with an Acceptable Audio Delay
– Audio Delay Geographic Uniformity
Keynote aggregated these ten factors into two distinct performance indices — Reliability and Audio Clarity. The Reliability index formula includes service availability, the average number of dial attempts, and dropped calls. The Audio Clarity index is computed based on a weighted average of the Average Mean Opinion Score (MOS) and Average Audio Delay including adjustments for percentage of calls below minimum acceptable thresholds and high geographic variability.
The full study, which is available for purchase from Keynote, includes detailed results, custom analysis of the data, and all raw measurement data. For more information on the study, to obtain an abstract or to purchase the study, please contact sales@keynote.com or go to http://www.keynote.com/solutions/ci_VoIP.html.
About Keynote VoIP Competitive Monitoring Solution
VoIP service providers can leverage the competitive insight provided by Keynote VoIP Competitive Monitoring Solution to narrow the gap between their call performance in target markets relative to PSTN and leading competitors. Combining Keynote’s competitive insight with internal network monitoring tools will allow operations and quality assurance teams to adopt an outbound focus and assure a quality experience for the end user. Keynote can design an ongoing VoIP Competitive Monitoring Solution in multiple markets including various competitors.
Service providers can leverage Keynote’s independent assessment and ongoing actionable recommendations to align their resources and infrastructure investments with customer acquisition and retention goals.
Keynote VoIP Competitive Monitoring Solution addresses the need of marketing and operations executives to understand their performance relative to competitors, and to gauge the impact on the end-user experience of both their infrastructure investments in new markets and their enhancements to services in existing markets.
Keynote VoIP Competitive Monitoring Solution provides ongoing performance monitoring on a monthly basis. In addition to ongoing competitive monitoring reports delivered monthly, Keynote includes custom analysis to help align operations teams with customer experience enhancement goals.
Availability and Pricing
The Keynote VoIP Competitive Monitoring Solution is available immediately. Pricing is customized based on the number of cities, providers and network carriers to be included in the ongoing measurements.
To learn more about the Keynote VoIP Competitive Monitoring Solution, send an email to sales@keynote.com or go to http://www.keynote.com/solutions/ci_VoIP.html.
About Keynote
Founded in 1995, Keynote Systems (Nasdaq "KEYN") is the worldwide leader in services that improve online business performance and communications technologies. Keynote helps approximately 2,300 corporate customers and 13,000 individual subscribers become "the best of the best" online. The business premise supporting Keynote’s mission is: "Online businesses can’t manage what they don’t measure." As an independent and trusted third-party, Keynote provides IT and marketing executives with unbiased benchmarking data, competitive analysis and operational metrics from the customer perspective. This data measures service levels and customer experience of Web sites, broadband services and mobile communications.
Known as The Internet Performance Authority(R), Keynote manages a market-leading infrastructure of 1,600+ measurement computers and mobile devices in over 114 locations and 66+ metropolitan areas worldwide that assess service levels and a panel of over 160,000 consumers who participate in interactive Web site tests that assess user experience. These online user experience tests capture customer attitude and behavior to answer the critical "why" behind the "what." Keynote’s geographically distributed measurement services, on-site monitoring appliances, competitive intelligence and custom studies ensure that its customers outpace their competitors in online service levels and overall user experience.
Keynote Systems, Inc. is headquartered in San Mateo, California and can be reached at www.keynote.com or by phone in the U.S. at 650-403-2400.
Keynote, The Internet Performance Authority and Perspective are registered trademarks of Keynote Systems, Inc. Other trademarks are the property of their respective owners. (C) 2006 Keynote Systems, Inc.