• E-mail
• Print
• Comment
• Font Size
• Digg
• del.icio.us
• Discuss article

Methodology for remote usability activities: A case study

Posted on: Saturday, 6 December 2003, 06:00 CST

Achieving high ease of use demands that an iterative design- evaluation-redesign process be followed. User interface evaluations and, sometimes, design work as well, require the participation of many target users, who often collaborate in the use of the product. In many cases, project members and potential users are spread worldwide, making face-to-face interaction difficult and expensive. Given these conditions and the ever-present consideration of keeping travel expenses down, human factors engineers and human-computer interaction experts are adopting a methodology that involves performing user interface design and assessment activities remotely, thus avoiding prohibitive travel costs while maintaining a high level of customer involvement. This paper presents a survey of various types of remote usability activities and some of the types of tools that are being used, and it describes a case study in which this methodology was applied; that is, the design of the Design Solutions application for the IBM Customer Connect portal.

The Internet's speed and reliability have made it possible to do work remotely in a way that just a few years ago was either impractical or impossible. Designers who are responsible for ensuring a product's ease of use are among those benefiting from this capability. Just a few years ago, the usability specialist was usually restricted to working on products that were being developed down the hall or perhaps in the same office complex, and conducting user interface (UI) design sessions, design reviews, and usability testing in a dedicated usability test lab where project members and user participants gathered. This was often a costly way to conduct such activities, due to travel and lab maintenance expenses, and it tended to reduce the number of users taking part in product trials. In addition, conducting such activities in a lab, rather than in a customer's actual work environment was, to some extent, an artificial evaluative environment. Test results may not have generalized to the intended population as well as they could have if a larger, more representative sample size were used and such testing were to occur in the users' actual working environment.

The capability to work remotely in an effective manner and to communicate with colleagues and customers residing in various geographical locations has enabled usability work to be expanded to a much larger user base in a cost-effective way, helping to ensure that more products are better tested and easier to use. Conducting usability work remotely simply means that those involved in the design, development, and use of a product are in different geographical locations and communicate by the use of various tools and the Internet or network infrastructure.

IBM Microelectronics products, technologies, and services are represented on the World Wide Web by an important Web site: the IBM Customer Connect Web portal.1 The IBM Customer Connect site is an Internet-based portal for product and technology information, documentation, and tools used by IBM Microelectronics customers, particularly chip designers. These customers include both IBM and non-IBM personnel, worldwide, and ease of use was a critical element in the design of this portal.

The portal consists of many applications that support well- defined tasks relevant to the design, testing, manufacturing, and distribution of microprocessors. Customer Connect is a secure portal that requires an IBM-registered user ID and password. External customers work with IBM field support engineers and others to gain access to the tools and sections of the site that best support their needs.

To ensure a quality experience for users of the Customer Connect Web site, human factors engineers and experts in Web application design worked with the development team and, just as important, with the people who would be using this portal, to ensure that typical user tasks could be performed easily. Various activities, most conducted remotely and all a part of a rigorous methodology called User-Centered Design (UCD), were conducted via the Customer Connect portal, under the leadership of the human factors engineer.

UCD is a proven methodology that instills rigor into the user interface design and evaluation process to ensure that applications and technology have high ease of use. This methodology is described by Vredenburg.2 A detailed treatment can be found in Reference 3.

This paper provides an overview of remote usability activities. A case study presented here focuses on how a remote usability program and the UCD process were applied to the Design Solutions application of the Customer Connect portal.

Usability and human-computer interaction This paper focuses on various activities designed to improve the ease of use of the Design Solutions tool. These efforts have involved the scientific discipline of human factors, sometimes referred to as usability. The term usability is synonymous with the phrase ease of use. One can talk about the usability of any system, device, machine, or tool that is intended for human use.

In the case of software human factors engineering, the specialty field known as human-computer interaction (HCI) is most relevant. There are many factors that must be considered in the design of the user interface of a software application or Web site. These include the use of appropriate types of controls (e.g., a drop-down menu versus a list box), the layout and organization of various page/ screen elements, adherence to established user interface guidelines, the sequence of presentation of pages or screens, and clear and consistent terminology that matches the vocabulary of the intended audience. Other significant aspects are immediate and clear feedback from the application, an easily accessed menu with options whose functions are apparent by their terminology, a clear conveyance of what input is required and the form which it should take, and minimizing and recovering easily from errors.

Remote usability activities

The traditional way of conducting most usability activities has been for the human factors specialist, members of the project team, and customers to meet together in the same room. If the activity is a user interface design review, such a meeting might take place in a conference room. If the activity is a usability test, it will often occur in a usability lab that is equipped with audio and video recording capabilities and a computer dedicated to logging the data from the study. The meeting may even take place at the customer's work site if the activity is a field test that is intended to gather data on the usage of the product in the customer's actual work environment. In a remote usability activity, members of the project team and participating customers arc not in the same geographical location. In order for the goals of the remote activities to be accomplished, the sharing of information, files, and so forth, must be feasible over secure, high-bandwidth networks. As such networks have become more commonplace, tools that support remote usability activities have proliferated.

Types of remote usability activities and tools. Remote usability activities can be divided into four distinct types: (1) real-time design walk-throughs and usability tests, (2) surveys, (3) automated usage tracking, and (4) handling of user-reported critical incidents. Of these four methods, three were used for the Design Solutions project. Only the fourth method, having users report critical usability incidents in a systematic way, has not been used for this project to date.

When conducting a design walk-through or usability test in real time, a set of representative tasks is used. Typically, a list of these tasks is constructed based on user input from a previously conducted task analysis. In a design walk-through, a prototype of the user interface is presented via some presentation tool or, perhaps, a semi-functioning interface that has been created using a prototyping tool. A task is defined for the test participant and is accomplished by "walking through" the task with the participant, who provides feedback concerning the interface. Audio communication is typically accomplished through a teleconference. In a usability test, tasks are again attempted but, unlike a design walk-through, functioning code is used, and participants are asked to actually perform tasks on their own while their interaction is observed, their comments are recorded, and usability problems are noted for subsequent causal analysis.

There are several tools available that allow participants to view the product under review in real time and specifically, to allow the test administrator to view the user's interaction with it. Web conferencing software that works over the Internet includes Microsoft NetMeeting**, ShowMe** from Sun Microsystems, and Lotus Sametime**. These tools allow participants to connect to a common "meeting room" interface and run within it applications that are viewable by all who are connected. With a high-speed connection, remote observers' experience of a test participant's use of the product can be as good as that of a local observer. Lotus Sametime, used frequently by IBM, allows a test participant to run the product under evaluation through its application-sharing feature. The test participant runs the application on his or her systemand shares the screen with the observers. The test administrator and other observers can record observations, note usability problems, and record the time required to complete tasks by using a data-logger tool on their own workstations. Verbal feedback from the test participant is usually given by telephone. A walk-through or test session can be recorded on video by using a camera stationed at the test administrator's workstation.

Figure 1 is an example of a test administrator's desktop during a usability test. The application (in this case, Design Solutions) is being displayed to all meeting participants through the Lotus Sametime application-sharing feature. The Design Solutions window is being used and controlled by the test participant. The WinLog data- logger tool partially overlays the Design Solutions window. The WinLog tool is being used by the test administrator to record the test participant's actions and to note problems. Each entry by the test participant is automatically time-stamped. The data-logger window can be moved or minimized at any time by the test administrator. After a set of tasks is attempted, test participants can display a usability survey in the Lotus Sametime window (perhaps by a text editor or by accessing a Web-based survey) and complete it while the other participants watch, allowing for follow-up questions and discussion.

Subjective usability data is typically a useful complement to the more performance-based objective data collected by a usability test that typically includes task completion times, task success/failure rates, the number and severity of errors, and problem descriptions. Subjective usability data is usually collected through surveys that measure opinions regarding various product attributes, such as ease or difficulty of installation, ease or difficulty of accomplishing specific tasks, the ability to locate certain information or menu items, the appropriateness and understandability of terminology, and so forth. Remote surveying can be done using electronic surveys that can be distributed either through e-mail tools or through Web conferencing tools like Lotus Sametime. Various methods are reported by Elgin.4 In the work reported in this paper, end-of-test surveys were administered using Lotus Sametime following a remote usability test. The survey was e-mailed to test participants before the test session. At the appropriate time, the test participant was instructed to complete the survey while sharing the survey application with the project members. This allowed project members to view the test participants' responses in real time and to ask follow-up questions or request clarifications.

Another remote surveying method (also used in the Design Solutions project), is to provide Web-based surveys. Usability surveys can be posted to Web sites with the data being submitted into a reporting tools database for relatively fast and easy analysis. Access to the survey can be allowed via a user ID authentication process and/or by invitation only, using a confidential Internet address. Two survey tools were used for the Design Solutions work. One tool was the GroupWare Systems survey tool that used a local hosting server. The second tool used was WebSurveyor**, with the survey hosted on a server that was administered by IBM User-Centered Design. Both tools contain built- in data analysis and reporting capabilities.

Figure 1 Example of usability test administrator's desktop

A third type of evaluative method that can be employed in remote usability work is the use of automated usage-tracking tools. These tools use software that is installed on the test participants' workstation that automatically tracks their usage of a standalone application or a Web site. A tool designed for Web sites typically records the address of each visited page, the user action that displayed that page (mouse click, function key, etc.), the length of time each page was displayed, and other information useful for analyzing a user's session.

One such tool used extensively in the Design Solutions project was an automated Web site tracking tool called ErgoBrowser by ErgoSoft, Inc. This tool consists of a basic Web browser and logs such information as the address of each visited page, time stamps for each user action and page visited, and the type of action taken by the user (including use of the browser's navigational buttons "Back" and "Forward"). These parameters can be recorded on a per task basis. ErgoBrowser automatically calculates, per task, the time spent on the task, the number of mouse clicks and keyboard presses, page visits, and other information that is useful to the usability analyst. This tool can be used by test participants on their own (with the data files returned to the administrator upon completion) or by a test administrator. Tools of this type are an excellent way to automatically record data like task completion times and can be used for analysis, for example, to recreate a user's path for tasks that were identified as being problematic.

A fourth evaluative method is to have users self-report critical incidents during their evaluation of a product. This method basically involves having test participants use the application on their own for some specified amount of time and report incidents that they believe indicate usability problems or that in some way impeded their use of the application. In this method of evaluation, it is typical to have software installed on the test participants' system that records their actions and provides a view of what the participants have seen. One such tool is IBM's LogCam, which performs real-time audio and video streaming of a user's interaction with an application. This permits a test administrator to view, in real time, the test participant's interaction with the product. As such, it is an alternative to tools like Lotus Sametime. However, LogCam also has data storage capabilities, permitting interactions to be played back, reviewed, and analyzed later.

Another useful tool is AutoLogger, an unobtrusive data-capturing tool that runs on the test participant's workstation and automatically records user interactions (keystrokes, mouse clicks, mouse drags and releases) and window information (titles and locations, windows that have focus), helping the usability analyst to recreate a task scenario that preceded a critical incident. This tool also supports automated usage tracking.

Benefits and challenges of remote usability work. A series of studies have looked at the benefits and problems of using the self- reported, critical incident method of gathering usability data. One such study by Hartson, Castillo, Kelso, Kamler, and Neale5 argued that there are many benefits of using this method, including the performance of real tasks by real users in their actual working environments; very cost-effective long-term data gathering (yielding many data points); no direct interaction needed between the user and the evaluator; and high quality data, which are relatively easy to convert for usability problem evaluation. Another study6 compared several remote methods of usability evaluation and reported that users with no background in software engineering or human-computer interaction, and "with the barest minimum of training in critical incident identification," can effectively use the self-reporting method to report usability problems and can produce severity ratings of problems that closely match the identified problems and severity ratings made by experienced human-computer interaction specialists. However, they also reported that the time when critical incidents were reported by test participants was often significantly delayed from the time when the problem was actually encountered. This might result in some loss of relevant detail, such as the context in which the critical incident occurred.

Perkins7 summarized the benefits and potential problems of conducting remote usability activities. Among the benefits, he noted:

* Users are in their own work environment where they feel comfortable and are more apt to behave normally and demonstrate usage patterns that are typical of them.

* The technical aspects of the environment are ones that are operative during actual usage of the product, such as browser settings, network speeds, and monitor resolution settings.

* As there are no geographical restrictions limiting who can take part in remote activities, a larger number and more diverse group of target users can provide usability data.

* There is also a cost benefit of remote activities, as test participants and project members do not have to be transported, lodged, and fed. The money saved can be used for more testing time, producing more usability data and probably resulting in test results that generalize better to the target population. There is also little or no cost associated with outfitting and maintaining a lab and its equipment.

As also noted by Perkins, there are some questions and challenges associated with remote methods:

* Many of the methods and tools described previously require a high bandwidth network to work effectively, such as T1, DSL (digital subscriber line), or cable modem connections. However, this is probably becoming less of an issue, as high bandwidth connections are becoming more common, even in the consumer sector.

* Getting access to a test participant's computer in his or her own work environment usually requires going through a firewall, something that many companies are reluctant to allow.

* There are technical logistics to deal with. Convincing customers to download and install remote activity software is sometimes problematic. And of course, on occasion technical problems will arise in the installation and/or use of a particular tool.

Krauss and Vigilante8 published setup and usage instructions for using Microsoft NetMeeting and Lotus Sametime web-conferencing software. They also provide\d a checklist of preparatory steps to take before conducting remote design reviews and usability tests, as well as the sequence of steps used during a session. Such items include a pilot session to test all software that will be used, assessment of test participants' computer hardware and software to determine their match to the prerequisites for running the software to be evaluated, timely distribution of test materials (task scenarios, surveys, etc.) to participants, and built-in session time that might be needed for training the test participant in using the remote testing software.

A case study: The IBM Customer Connect portal's Design Solutions application

The IBM Customer Connect portal was created to assist chip designers and semiconductor engineers. Customer Connect is essentially a portal to many Web-based applications, each having a specific function in support of semiconductor design and manufacturing. A sample of tools and services in Customer Connect includes a technical library for standard products and services, information for resellers of IBM storage products, and a tool for managing orders and pricing information, tracking of work-in- progress orders, and requesting authorization to access tools and documentation. Access to Customer Connect tools and documentation is controlled by the IBM identification registration system.

A strategically important tool available on Customer Connect is Design Solutions, a function-rich application specifically intended for both the IBM and non-IBM chip designer who works with various software tools, such as EinsTimer* and ChipBench*, to perform tasks such as power consumption analysis, static timing analysis, clock optimization, and gate-level simulations. Design Solutions provides several categories or main sections of services to assist an ASIC (application-specific integrated circuit) chip designer:

* Libraries and toolkits: Using this section, the engineer can order libraries of software for various ASIC technologies, including Cu-08, Cu-11, and the SA-12 and SA-27 family of cores. Most of these can be ordered as CD-ROMs or downloaded. Delta releases, patches, fixes, and compilers can also be ordered.

* Methodology and tools: This section not only provides technical documentation for various software design and testing tools but also includes methodology documentation, such as the detailed, step-by- step methodology for performing the physical design of chips. A relatively new and important part of Design Solutions is access to a tool called The Guide. The Guide is an application that combines the necessary models, tools, and methodologies in a consistent environment that enables users to more easily process and manage their own designs.

* Design services: This section provides a Web conferencing tool called Web Conferences that permits real-time collaboration between chip design engineers, whether within IBM or between IBM and external customers. It allows data, tools, and documentation to be shared in real time so that design issues can be resolved.

* Education: This section contains a list of courses that would be of interest to those involved in chip design.

* Customer administration: Each external Design Solutions user is assigned an IBM Field Support Engineer (FSE) who assists the customer in gaining access to other tools on Customer Connect, answering questions regarding technologies, products, methodologies, and tools, and keeping customer profile information updated. Using the customer administration function, the FSEs can view and/or update profile information and projects that are associated with customers, assign a new or backup FSE, and order software on behalf of customers.

Table 1 Summary of remote usability activities and tools

Like most of the Customer Connect content, Design Solutions should be considered an application rather than informational Web content. Design Solutions, like the other tools on the Customer Connect portal, is a Web-based application, and has been designed to have the appearance and behavior of more traditional, standalone applications.

The usability work conducted on the Design Solutions application was the first project conducted with Customer Connect that required a great deal of remote usability work. Design Solutions was chosen for the focus of this paper not only because of its importance to the Customer Connect portal but also because it has involved, perhaps more than the other tools on the portal, a more diverse complement of remote usability methods. Table 1 summarizes the various tools that were used to support the different work activities and the type of data that was produced.

Gathering user profiles. To obtain information on the user population for the Design Solutions suite of ASIC libraries, toolkits, documentation, and other functions, an audience definition activity was performed early in the project. The user audience for the Design Solutions application consists primarily of ASIC chip design engineers who are involved in day-to-day design and testing activities, either designing microprocessors for their own specific applications or for those of their customers. They typically have formal educational backgrounds in computer engineering, electrical engineering, or physics. User groups include both IBM engineers, usually from one of the many IBM design centers located around the world, and non-IBM engineers who, in conjunction with an IBM FSE, are entitled to use the tools and services provided by Design Solutions.

Since potential customers, both IBM and non-IBM, were geographically scattered, it was essential to use electronic means to gather the needed user profile information. Two methods were used. A link was distributed via e-mail that pointed to a user profile survey hosted on a Global Systems Decision Support Center (DSC) server. The survey was constructed using the GroupWare Systems software. Additionally, the administration of the same survey was done by telephone with other IBM engineers.

There were many pieces of information that the project team needed to gather about the potential users of this tool. One critical piece of information was the technologies and products that users would be most interested in. This largely determined what cores, toolkits, and libraries were to be made available for ordering. The project team also needed to know the types of computers and network connections that would typically be used to access Design Solutions to order and download these software packages. This type of information was continuously tracked during all of the remote usability work that involved users. Also necessary was a list of users who would be willing to participate in further UCD activities, including task analyses and design evaluations. These surveys enabled a database of such users to be established.

Performance was a key consideration, as was indicated by the user profile surveys. Did the potentially large size of some of the libraries necessitate that the packages be split into multiple downloads, given the types and speeds of network connections that the audience would be using? The answer to this particular question was "yes."

Performing the task analysis. Task analysis also took place remotely and was done in two phases, again using the GroupWare Systems survey tool, e-mail, and telephone interviews. The primary objective of the analysis was to derive a prioritized list of tasks that users would want to perform and to identify key factors that needed to be considered when making functions available through the user interface. This helped to identify specific, required functionality and provided a database of tasks that would be used in subsequent usability activities, such as design walk-throughs and usability tests. The task analysis survey asked participants to list and describe the tasks that they wanted to perform, identify key features of these tasks that the user interface design needed to address, and identify the potential difficulties with these tasks. An example was included that contained a task description and a few details of the task that needed to be considered by the designers of the user interface.

When many activities are done remotely, particularly those that are not administered, instructions must be clear, and examples must be given. An example of the type of information that was derived from the task analysis pertained to the general task of software downloading. For this task, it was important to display information about the file to be downloaded, such as its size and the predicted download time for different connection speeds.

After a list of tasks was compiled, the master list of tasks and their descriptions was presented remotely to participants either by means of the survey tool or by e-mail, and the respondents were asked to rank the tasks in order of importance. These results helped the project team establish the schedule for availability of functions and helped guide subsequent design and review activities. The following list contains a sample of key features and functionality that came from this rank-order analysis.

* A search engine to search the library of technical documentation with specific search attributes and rules

* Ability to save orders as drafts for later editing and submission

* Displays for FSEs, making them aware of what is being sent to customers

* Provision for methodology alerts to all users

* Performance-based features, such as the ability to split large files into multiple files and the display of file sizes and download times for different connection speeds

* For Web Conferences, a means for moving an existing window into the Web Conferences environment for consultation purposes

* A mechanism to display a history of orders by customer and project

Team structure and internal design review methods. The project team was spread over three regions: Research Triangle Park (RTP), North Carol\ina; Fishkill, New York; and Burlington, Vermont. In RTF, the project team consisted of human factors (HF) engineers, a graphic designer, and a technical writer. In Fishkill, the team consisted primarily of those who were responsible for development of the Web Conferences tool. Burlington was the site of the majority of the development work for Design Solutions and also was the home base of the project leader and business analyst (see Figure 2).

Figure 2 Example of geographical distribution of customers and team members

A variety of tools were used to maintain close cooperation among members in these different locations, including Lotus Notes for e- mail, Lotus Sametime Web conferencing, a development server to allow design and function reviews with working code, and a Lotus Notes team database that allowed project members to organize and communicate information and to share files.

An early prototype of the user interface was created through the input of several members from the various groups, including human factors and graphic design groups in RTF. Page mock-ups were usually created first, illustrating some specific function or task. Design reviews of these early mock-ups were conducted among members using Lotus Sametime Web conferencing and teleconferencing. By displaying the mock-ups in Sametime, an editor could highlight desired areas and make impromptu changes during review meetings. Using the Sametime conferencing tool, it was possible to not only share designs with multiple project members but also to make changes quickly and easily.

After a tentative user interface had been mapped out, a computer- created prototype was developed using HTML (HyperText Markup Language) and image files. It realistically depicted the organization and various page elements of the tool without incorporating working links and back-end processing functionality. The project team then conducted several remote design walk-throughs with internal users, which used many of the tasks that were identified during the task analysis. Again, this remote activity, which involved engineers and project members from California, Texas, New York, Vermont, and North Carolina, used the Lotus Sametime Web- conferencing facility in conjunction with teleconferencing. In addition, the WinLog data-logger tool was used to capture user feedback and to associate this feedback with specific pages and their flow. In addition to the data captured by this logging tool, a survey was also administered that assessed opinions of the usefulness of the function and various aspects of the usability of the interface. This survey was e-mailed to users before the Web conference and returned to the human factors engineers upon completion.

Conducting the design walk-through. The next phase of the project involved conducting a formal, face-to-face design walk-through at the RTP Decision Support Center facility. This was the first of several design walk-throughs that were done; it was the only local activity that occurred during the project. Subsequent design walk- throughs were conducted remotely by using Lotus Sametime Web conferencing and included two to three project members and only one user per session. Restricting design reviews to one user per session was found to be more manageable than group design reviews conducted locally and generally allowed more time for in-depth discussion of various design issues and exploration of alternative ways of designing the user interface. In total, in all of the design walk- throughs that were held, eleven IBM engineers (all FSEs for various customers), along with several project team members, participated. These sessions covered the major sections of the Design Solutions application, including the technical library, software ordering and delivery, Web Conferences tool, and customer administration.

The general format was for a member of the project team to make a brief presentation of a section that included its major functionality, followed by a task-based design walk-through led by the DSC facilitator and the human factors engineer. A specific task was defined for the group, and, starting at the Design Solutions main page, the participants were asked to state what steps they thought were needed to complete the task, how each page was to be used, and to provide feedback on the interface. Several different categories of feedback were solicited, including comments on the navigation of pages, use of terminology, layout and grouping of controls, need for and content of cues and page instructions, recognition of icons and other graphics, completeness and clarity of object descriptions (e.g., descriptions of various ASIC cores), and several other categories. Two members of the project team were assigned the role of note takers. Other data were captured by using an e-mailed survey that was completed within the Lotus Sametime sessions while the project members observed the responses.

The totality of the results for these sessions indicated that significant user interface changes were needed for the Web Conferences tool and, to a lesser extent, for customer administration. The identified problems and task errors, as well as negative comments in the subjective data, were much more numerous for these areas than for the software delivery and technical library areas of Design Solutions.

The initial evaluation of Web Conferences found that the user interface was too complex (it involved inviting participants, reassigning various types of meeting privileges, such as who has control of a meeting, and performing the tasks of starting up and integrating an application into the meeting window). Participants spent an inordinate amount of meeting time discussing how to use the tool instead of focusing on the chip design issue that was the impetus for the meeting. Consequently, the project team decided that an early usability test would be beneficial to provide a more complete evaluation and to identify problems early enough to address them.

Conducting a remote usability test. After the last design walk- through session was held and the UI designers and developers had attempted to address the problems, particularly for the Web Conferences function, a remote usability test was conducted that focused on Web Conferences. Technically, this was a particularly complex test environment to set up, due to the nature of the tool and the tasks that needed to be used. Use of the Web Conferences tool requires that participants connect to a separate server that makes the Web Conferences tool available to the client's desktop and performs user ID authentication. The test participants' Customer Connect and Web Conferences sessions also had to be shared by the Lotus Sametime tool so that the test administrator could track their interaction and record data. Data was recorded by the UCD Logger tool on the test administrator's workstation, as well as by the AutoLogger tool running on participants' desktops. Two additional test participants served as meeting attendees to create a more realistic test situation, allowing different attendees to take control of the meeting and to invoke different chip-design and testing applications.

The results showed that while many of the usability problems had been resolved (resulting in almost a 60 percent reduction in the number of identified problems), the reported satisfaction with the usability of the tool was still less than acceptable. The targeted ease-of-use satisfaction level was a 2 (corresponding to "satisfied" on a 5-point scale with 1 being "very satisfied" and 5 being "very dissatisfied"), whereas the obtained average rating was a 3. A few of the more significant usability problems remaining at this point were:

* Users should not have to hold the right mouse button down to select a pop-up menu option.

* There is no clear indication as to (1) how the person currently in control of the meeting can initiate the passing of control to someone else and (2) how the recipient can take control of the meeting once he or she has been given the status of a participant.

* Having to re-enter the names of invitees, either when one is trying to invite them and they aren't logged on or when they return to the meeting, is time-consuming. An address book or list is sorely needed.

* There needs to be more information provided about browser requirements, IDs needed, and so forth, for using Web Conferences, and this information should be displayed in the window that appears when the Design Services link is clicked.

* A function needs to be provided that allows data to be shared during a Web Conferences session (i.e., a cut-and- paste capability).

Redesign and verification. Because of the severity of many of these usability problems and because of the strategic nature of the Web Conferences and customer administration functions and their importance to Design Solutions, further user interface development work on the Web Conferences and customer administration functions received high priority. Human-factors and graphic-design personnel redesigned the user interface in RTP and supplied page mock-ups to development sites as templates for coding. Approximately one month following the remote usability test, new design walk-through sessions, again conducted remotely, were held with the majority of users who had participated in earlier design walk-throughs and usability tests. As part of this design walk-through, participants reviewed previously identified usability issues and discussed how well the project team had addressed them. Participant feedback was captured using both the WinLog tool and a Web-based survey. These sessions produced positive results, indicating that the majority of usability problems had been adequately addressed.

Summary. By applying the UCD principle of an iterative evaluation approach to the Design Solutions interface, the project team was able to refine the interface to a point where most \usability problems were eliminated, producing a tool that permitted engineers to become more productive quickly. It is important to convey how critical an iterative evaluation approach is to ensuring high ease of use. In all likelihood, no initial interface design has satisfactory usability. Only through a design-evaluate-design iterative process of refinement can high ease of use be achieved.

A critical factor that allowed the iterative approach to be followed was the ability to conduct remote usability work. If this work had been performed locally, time and budget constraints would not have permitted the high number of intensive, constructive evaluations that occurred. Many hundreds of person-hours were spent, most of them in remote activities, conducting design walk-throughs and usability tests. If these activities could not have been conducted remotely, only a fraction of this time could have been spent, fewer evaluation sessions would have occurred with fewer users participating, and the usability of the application would undoubtedly have been poorer.

It is also well worth noting the cost savings that occurred by doing most of this work remotely rather than locally. Based upon a conservative estimate of $700 per participant for transportation, lodging, meals, and miscellaneous expenses for each session, the number of remote sessions that were held, and the number of people who participated (customers and project team members), approximately $25 000 to $30 000 was saved by employing remote activities.

Conclusions

In the human factors community, there is a growing realization that remote usability activities have much to offer usability specialists and the product lines they support. The benefits of remote usability work are many. Since remote activities are cost- effective, product managers are more likely to engage human factors experts. Consequently, more products can be made easier to use. By performing activities remotely in their own environments, customers are using the same hardware, software, and network connections when testing as in their use of the actual product, producing more valid test data and ultimately, a better product.

Given the elimination of travel time and the inherent logistics involved in doing on-site activities, remote usability activities enable more time to be made available for evaluations. This was certainly the case with the Design Solutions project in which participating users were scattered throughout the country. This fact made it much more likely that customers would agree to participate in evaluations and consequently, more customers did participate, producing more representative feedback on the design of the tool.

These benefits were not confined to the work done with the target customers. The iterative design process that occurred between human factors personnel and the other project members also benefited. Again, working remotely allowed a more efficient use of time, as non- productive travel time was eliminated. This permitted a more fluid design-review-redesign process to occur in contrast with a situation requiring face-to-face design review meetings.

The increasing awareness of the benefits of working remotely is demonstrated by a growth in the number of papers that have been published on this subject in the past few years, as well as symposiums and presentations that have been featured at various conferences. In addition, this awareness is reflected in the increasing number of software tools that are designed specifically to support remote usability work. IBM's User-Centered Design Workbench,9 with its suite of tools to support remote work, is one example.

Acknowledgments

I would like to thank the members of the Customer Connect project team for their work and support of the UCD activities, including Peter Lyon, Athar Tayyab, and the members of the various development teams, as well as the Field Support Engineers and other users who participated in the sessions and provided so much helpful input. I also wish to thank a human factors colleague, Gae Boyette, who provided able assistance on this project. Finally, a "thank you" is due to another colleague, Tom Toher, for editing this paper and for his invaluable suggestions.

(C)Copyright 2003 by International Business Machines Corporation. Copying in printed form for private use is permitted without payment of royalty provided that (1) each reproduction is done without alteration and (2) the Journal reference and IBM copyright notice are included on the first page. The title and abstract, but no other portions, of this paper may be copied or distributed royalty free without further permission by computer-based and other information- service systems. Permission to republish any other portion of this paper must be obtained from the Editor.

Getting access to a test participant's computer in his or her own work environment usually requires going through a firewall.

When many activities are done remotely, particularly those that are not administered, instructions must be clear and examples must be given.

* Trademark or registered trademark of International Business Machines Corporation.

** Trademark or registered trademark of Microsoft Corp., Sun Microsystems, Inc., Lotus Development Corp., or WebSurveyor Corp.

Cited references

1. See https://www-3.ibm.com/servlet/oem/edge/mdex.jsp.

2. K. Vredenburg, "Building Ease of Use into the IBM User Experience," IBM Systems Journal 42, No. 4, 517-531 (2003, this issue).

3. K. Vredenburg, S. Isensee, and C. Righi, User-Centered Design: An Integrated Approach, Prentice Hall PTR, Upper Saddle River, NJ (2002).

4. B. Elgin, "Subjective Usability Feedback from the Field over a Network," SIGCHI Bulletin (ACM Special Interest Group on Computer- Human Interaction) 27, No. 4, 43-44 (October 1995).

5. H. R. Hartson, J. C. Castillo, J. Kelso, J. Kamler, and W. C. Neale, "Remote Evaluation: The Network as an Extension of the Usability Laboratory," Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI '96) (1996), pp. 228-235.

6. H. R. Hartson, J. C. Castillo, and D. Hix, "Remote Usability Evaluation: Can Users Report Their Own Critical Incidents?" ACM CHI 98 Conference Summary on Human Factors in Computing Systems (1998), pp. 253-254.

7. R. Perkins, "Remote Usability Evaluation over the Internet," in Design by People for People: Essays on Usability, Usability Professionals' Association (2001). See http://www.upassoc.org.

8. H. Krauss and B. Vigilante Jr., A Reference Guide for Remote Usability Activities Using NetMeeting and Sametime. Technical Report 29.3493, IBM Research Triangle Park, NC (2002).

9. See http://ucdwasp1.torolab.ibm.com/pmp/pm.

Accepted for publication May 25, 2003.

Frederick Scott Hunter Krauss IBM Storage Systems Division, P.O. Box 12195, 3039 Cornwallis Road, Research Triangle Park, North Carolina 27709-2196 (hkrauss@us.ibm.com). Dr. Krauss performs research in user interfaces and design and evaluation for several projects, including the IBM Microelectronics Division's Customer Connect portal and Global Services' Server Resource Management system. He received his Ph.D. degree in experimental psychology from North Carolina State University with an emphasis in perception and sensory physiology and with a minor focus in human factors engineering and statistics. He is a member of several professional organizations, including the Usability Professional's Association, Human Factors and Ergonomics, and the Association for Computing Machinery Special Interest Group on Computer-Human Interaction. He has published articles and technical reports on various facets of human-computer interaction and is the holder of several U.S. patents.

Copyright International Business Machines Corporation 2003

More News in this Category