Initially the team developed a base of knowledge among its members by reviewing articles on website redesign and information architecture. The team reviewed writings by Instone, Nielsen, Morville, Rosenfeld, and other respected experts in the field. This brought the entire team up to a base level of knowledge and expertise. The team then proceeded to review, compare, and evaluate similar websites; looking at other comparable university library systems. The team reviewed several other redesign projects as well. A plan was created from this to guide the redesign.

A sub-team of the Web Site Support Team was charged with creating and implementing a heuristic test of the current Libraries' website to compare the existing site to common design elements. The results of the heuristic test showed that the site had not been adhering to many common design elements and constructs.

An online survey was developed to gather basic feedback on user expectations of the Libraries' site and of general user preferences and technical knowledge. Questions were developed by the team and then tested by using student employees within the libraries. The pre-test phase with student employees was extremely beneficial in creating an effective tool. The test phase allowed us to refine the questions to ensure the survey was clear and would provide us with information we could use to help redesign the site. The final survey was then rolled out onto the home page of the Libraries' website. Several of the departmental libraries also included a link to the survey from their home pages. The survey gave us an initial view of our users' concerns with the Libraries' site, a self-perception of their searching abilities, and a scan of their technological position (browser-type, Internet access, etc.)

The online survey, by way of a request for volunteers, also provided the team with a pool of students for the task-oriented testing in the next phase. We also placed several advertisements in the student newspaper requesting volunteers. The incentive was a $10.00 copy card for 20-30 min. of voluntary testing. Our results from the advertisements were very poor; the bulk of our volunteers came from the option to volunteer in the online survey.

The task-oriented phase consisted of a series of tests whereby students and faculty were observed completing a variety of tasks to help us develop prototypes for the Libraries' site. We again used our student employees to test card sets and questions to ensure they were worded properly and that we would elicit useful information. One round of testing consisted of a card sorting test. The terms and sections of the current Libraries' site were individually placed on cards. The deck of cards was given to a participant and asked to sort the cards into 3 piles; very important, moderately important, and less important. Once this was completed the piles were taken away for later analysis. A second matching deck was given to the participants. They were asked to sort the cards into piles of similar resources that made sense to them. They were also asked to create a label for each pile, using blank cards that were provided. The purpose of these two tests was to determine what was important to the students, what they considered to be related, and what they would call a specific group of resources. This method of testing gave us insight into how the libraries were matching the students' and faculties' expectations of what and how they would find information.

A second round of task-oriented testing consisted of a series of questions that the participants would attempt to answer using the current Libraries' site. A team member would observe them attempting to answer each question and write notes on how they accomplished finding the answer. A completely different group of volunteers was used for this round of testing. The participants were expected to think out loud about their impressions and their considerations while trying to answer the questions. The participants were scheduled in small groups and tested simultaneously in our electronic classroom. The purpose of this test was to determine current navigation problems and to shed light on how students and faculty seek information form the Libraries' site.

Throughout both rounds of task-oriented testing a brief focus group discussion was held after each session. Specifically, the participants attending the card sorting and the task-based testing were brought together in a small group where a general discussion was held about the Libraries' site and Web site navigation in general. An interesting development from this discussion was that the participants often would say they preferred one thing but the testing showed they preferred the exact opposite.

The results of the task-based testing focused on several areas: terminology, navigation, and site architecture. Terminology problems consisted of difficulty understanding library jargon as well as problems of inconsistent naming of resources and services. Participants commented on having problems with navigating the site, including dead end and orphaned pages. The site architecture was also clearly a problem. Participants reported in the task-based testing and in the card sort testing desiring better, faster access to resources and services. The site had too many layers, forcing multiple clicks to get to useful information.

The team collated all of the information we had gathered and determined that we needed to focus our energy on several specific design elements. The overall look of the site needed to be refreshed and made more appealing. The navigation needed to be more consistent across the entire site including the departmental libraries; this led us to use Cascading Style Sheets to maintain a consistent look with some flexibility for the departmental webmasters. The site architecture needed to be improved drastically with the creation of a controlled vocabulary to be used across the site. Also it was necessary to create clearer paths between pages and sections so visitors know where they are, where they can go, and how to get back. Finally, the original site was far too deep, with an excessive amount of clicking required to find needed resources. Thus the home page needed to become a more prominent portal for linking to services and sources quickly.

After collecting all of the data from the task-oriented testing and analyzing it for trends and patterns, the Web Site Support Team met with the Libraries' webmasters to discuss the findings and to plan for developing the new site. This provided an opportunity for the webmasters to provide input as well as to create buy-in to the redesign process. The Web Site Support Team presented its findings from the testing and engaged the Webmasters in a discussion of the design ramifications of the findings.

The next phase was to create the underlying infrastructure, the foundation for the redesign of the Libraries' site. A thesaurus was created to resolve issues of inconsistent terminology and excessive jargon. Site architecture was developed to guide the redesign. Visual elements were discussed and samples were created; leading to the development of prototypes.

The Web Site Support Team then broke into two sub-teams, each assigned to create a prototype of the redesigned Libraries' homepage. It was decided that there was enough variation in the information collected that two separate prototypes should be made and then compared and tested by users. The two teams were each given certain design elements to create the homepage and a sample lower-level page. Once the prototypes were made, the team took them back to the users. Participants in the original card sorting and task-oriented testing were contacted and given access to the prototypes. Minor changes were made based on this feedback. The prototypes were then taken to a new group of task oriented testers. Over a 2-3 week period the team set up laptops in a variety of public locations; several of the departmental libraries on campus as well as several high traffic areas. The team solicited volunteers to sit down and complete several tasks with each prototype and provide commentary on which features they found most useful or implemented most effectively. The team alternated which prototype was tested first to avoid any preference based on testing order. An additional function of this public testing was to promote the Libraries' redesign project and to highlight the fact that we were involving users throughout the process.

The final redesign consisted of a merged version of the two prototypes. Elements that showed positive results from the final round of testing along with all of the previous testing and research were brought together for the final redesign. A publicity campaign was developed and the final design was presented to the Libraries' faculty and staff. The design was then rolled out publicly to the faculty and students of the university.

At the conclusion we felt very confident that we had designed a more user-friendly and user-centered site. The site received very positive feedback once it was rolled out. We worked in conjunction with Purdue University Marketing so that the color scheme and some design elements were in line with the eventual redesign of the entire University site. Since we actually rolled ours out first it looked like the University was catching up with the Library; this was great for our own marketing and campus standing.

Throughout the redesign process we focused on involving our users: the students and faculty. We knew there were significant problems but we could not just go and make the necessary changes. This had been tried in the past and was questioned by the Libraries administration. Keeping the redesign user-centered and user-involved throughout allowed us to justify every decision we made so that if it were challenged we could go back to the tests and the data to support the direction we had chosen.

Web sites must remain living, adapting entities; thus there is discussion of conducting a new round of usability testing to determine if there are any design issues with the current site. The Web Site Support Team is in discussion with the new Libraries' administration as to the future development of the Website. Issues being debated include possible migration to a content management system and/or outsourcing the next redesign. It is too soon to tell what path the Purdue Libraries will follow, however we will always strive to keep our users involved in the design process.

In 2001, the University of Virginia Library was evaluating its methods used for internal assessment and process evaluation. The Library's Web sites provide a key entry point for users to many of the Library's services, thus making it the hub of technology infrastructure. Ease of use and consistency in design and functionality are key ingredients to building a customer-oriented system. With this in mind, a Usability Group with representation from Management Information Systems, Digital Access Services, Communications, and other Library and UVa units was created to undertake assessment of the Library's growing web presence. This group set up the initial criteria for selecting sites to be tested, procedures and forms used in testing, and a metric to quantify testing in an effort to measure success.

The criteria for sites to be tested were simple. All new sites and any sites scheduled for design revision would be tested. Certain categories of sites -- primarily temporary pages and web-based incarnations of Library exhibits -- would not be tested. Sites in the UVA Library's web sphere are designed centrally by the Communications web staff, but implementation and maintenance are distributed amongst the numerous site owners throughout the Library staff.

It was agreed that, depending upon the site to be tested, either heuristic testing or full usability testing would be selected. Local heuristic principles were laid out (http://www.lib.virginia.edu/usability/heuristics.html), modeled on Jakob Nielson's "Ten Usability Heuristics" (http://www.useit.com/papers/heuristic/heuristic_list.html). When heuristic testing was called for, members of the Usability Group would perform the tests. As for full usability tests, guidelines were developed for creating site test questions, including the creation of site owner questionnaires and templates for test logs http://www.lib.virginia.edu/usability/tests/. Test administration procedures were documented http://staff.lib.virginia.edu/usability/testing_procedures_original.htm .

The full testing process was a relatively simple one. Sites that met the criteria were identified by the web design staff in the Communications unit. In some cases, site owners of sites not due for revision submitted their sites for testing because they saw room for improvement. The owner of each site was asked to fill out a Site Owner questionnaire, to supply contextual information such as the mission, audience, and functions of the site. A set of between ten and fifteen task-based test questions were developed by the Usability Group, or no more than it would take an hour to test. Smaller sites had as few as seven questions. The questions were framed to require the tester to answer a question or perform a task without cluing them in that something definitely is part of the site. The questions avoided the exact language that is used on the site to label the feature or function. The goal was that all questions should be written as "Is there?" or "Can you?" questions, not "Where is x in the site?". Tests are available for review online http://www.lib.virginia.edu/usability/tests/ .

The goal was to test groups of five categories representing undergraduates, graduates, faculty, Library staff, and the public, if appropriate. Depending upon the site, there might be as few as three categories of groups, as small as three students of mixed type, three teaching faculty, and three Library staff. Testers were solicited from the Library staff through email calls for participation. Faculty were solicited through personal requests from librarians. Students were most often employed by the Library, or solicited by student Library employees. Library employees remained on the clock for their tests. Students were paid $10 for their participation.

Card Sorts were also sometimes used as part of the review of an already existing Library site to suggest revisions to a site's structure and labeling. The UVa Library process is a Closed Card Sort. Groups of no more than 6-8 participants were given stacks of cards, one for each of the existing or suggested set of organizing categories and each single site page. The group was asked to organize cards representing pages under the categories. UVa breaks slightly from the usual practice by additionally allowing users to suggest new categories or new wording for category labeling. One or more groups might be tested, representing multiple user constituencies.

Members of the Usability Group administered all tests. Tests were administered with one tester, one facilitator, and one observer to record results on a paper log, logging steps taken by the tester with time elapsed in 15-second intervals. Tester names were never recorded, only the category, such as faculty or graduate student. Test results were summarized and a report written by a member of the Communications web design staff for presentation to the site owner. The recommendations in that report were implemented whenever possible, but time and resources were not always available for site revisions in an environment where site maintenance is distributed. Sites were only on rare occasions re-tested after revisions were made.

The University of Virginia Library developed a set of Balanced Scorecard metrics in 2001 for assessment purposes. A usability metric was put in place with the target that eighty percent of sites that met the criteria should be tested each year. For the first three years the group met or narrowly missed its target.

In mid-2005 the Usability group reviewed its procedures. This review was initiated as part of an overall review of procedures for all activities related to internal processes with corresponding Balanced Scorecard metrics. The metric had previously been grouped with a category of measurable activities called "Learning & Growth," looking at how well the library was positioned to ensure that goals are met in the future. Usability testing and its metric are now reviewed in the context of how well the library's internal processes function to efficiently deliver library collections and services.

Purpose - To provide an overview of the process involved in usability testing with a case study about offering library services through a campus portal.

Design/methodology/approach - To gather feedback from potential users of a customizable electronic library product, participants were selected from different library patron groups: undergraduate, graduate students, faculty, and library reference staff. The tests were conducted as one-on-one structured interviews. Participants were asked to perform selected tasks by using our campus portal (we call it WINGS) and another University's portal. Responses were noted and tallied manually rather than through use of computer software because of the relatively small number of people involved.

Findings - Responses about the potential usefulness of a customizable library were mixed. Participants indicated that having a customizable library in the campus portal may be useful. A majority of the participants were using WINGS for one or two functions like webmail, calendar, Course Studio, etc. but were not utilizing all of the portal's features.

Keywords: Library portals, library services, case studies

Wright State University implemented SunGard SCT's Luminis as our campus portal in 2004. The portal, which we call WINGS, provides single sign-on access to University services like course registration, grades, accounts, campus activities, and more. We see tremendous potential for WINGS to make the Wright State University community's work easier.

The University Libraries created several optional channels for WINGS, including channels for the catalog, course reserves, dictionary searching, and library news. These channels link to static pages on our library website. The channels do not automatically display personalized information upon portal login. We would like for users to be able to customize lists of resources (such as databases and journals) that they use most frequently, and ultimately we would like library services such as requesting books, checking account status, and interlibrary loan to work under the single sign on protocol for WINGS. Since the portal is designed to manage multiple accounts, we are investigating different ways to create a customizable electronic library within WINGS.

Before implementing a customizable library, we conducted a small usability study in order to gauge our patrons' response to the product. Our goals were to gather opinions from users about the library channels we currently offer through WINGS, and the potential usefulness of a customizable electronic library product.

In this essay, we will describe the procedure we used, the conclusions we reached, and how the results will inform our future decisions about customizable library implementation.

According to Jakob Nielsen's popular rule of five, 85% of a site's problems can be found with as few as five participants. It may be true that you can learn a lot from just a few interviews, but when the audience for your product is composed of distinct groups of people, it is good idea to select at least two participants from each group. In our study, the groups were undergraduate students, graduate students, faculty and staff of Wright State University. We selected two undergraduate students and two graduate students. Due to availability constraints, we were only able to test one faculty member and one library reference staff member. We chose participants based on our personal contacts, attempting to find representatives from multiple fields of study. Tests were scheduled by e-mail or by phone. We made note of the demographics (status, field of study) for each confirmed participant.

We used the same set of open-ended questions for all tests in order to clearly capture the different perspectives of our different audience groups. Our script consisted of three sections. In the first section, we asked participants to characterize their use of our library website:

The test monitor plays a crucial role in the success of any usability test. His or her role is to lead the participant through the test questions while being careful to not skew the results in any way. We begin each usability test by thanking the participant for volunteering, then reassuring the participant that the test is of our site, not of them.

The usability tests described here were conducted in a much less formal fashion than usability tests we have conducted on our library website. The current project (implementation of a customizable library) is still in the initial planning phases. Comparatively few participants were involved, and we just wanted to get a general feel for what they had to say.

The testing locations were different for each participant and were selected by the participants themselves for comfort, convenience, and so as to take a minimum of their time. The tests could be conducted from any computer with Internet access. With a more formal test, it is more important to standardize the experience, conducting all tests from the same location. When we conducted tests of our library website, we tested from computers that had Camtasia installed for recording mouse clicks and vocal comments.

Our subjects were two undergraduate students, two graduate students, one faculty member, and one library reference staff member. Participants' levels of experience with the Libraries' website ranged from basic to advanced. The undergraduates we surveyed had used the site for course reserves and catalog searching but had little experience with article databases. Graduate students and faculty were more sophisticated in their use of the website. They generally reported bookmarking frequently used pages such as the electronic journals list. The librarian did not bookmark pages because he was familiar enough with the site to navigate through it quickly as needed.

The second part of our survey addressed participants' use of the WINGS portal. Of the six people we surveyed, only one could be considered a regular WINGS user. This person, a graduate student, reported using WINGS about once a week, mainly for email, the calendar, and the academics tab. She had done some customization of her WINGS portal and was the only participant who knew about the library channels we created under the Academics tab. She supported leaving the tab as it was, with the library integrated with Academics. One undergraduate student had used the Course Studio function within WINGS. Most participants used WINGS very occasionally for e-mail, preferring an alternative URL for accessing Wright State e-mail via the Web. The library reference staff member was required to use WINGS to maintain his professional calendar and to access group messages and files, but was dissatisfied with the efficiency of those processes.

The third part of our survey involved a demonstration of the MyLibrary system as implemented at another university. Reaction to MyLibrary was mixed. One graduate student was very enthusiastic about the customization options and said that MyLibrary would increase her use of WINGS. Most other participants acknowledged that it was nice that customization options existed, but could not commit to saying MyLibrary was something they would use personally. The mechanics of customization seemed straightforward to the participants. However, the faculty member expressed a lack of confidence in his ability to choose the best customization options for himself. He worried about "missing something" and was pleased to hear that resources could be recommended for him by the subject liaison for his department. He also expressed reluctance to change his research habits unless MyLibrary made his work significantly easier.

The library reference staff member did not think students would be likely to use MyLibrary. The faculty member, on the other hand, commented that MyLibrary would probably get most of its use from people without ingrained research habits.

This case study illustrates that usability testing can and should be done early on in the web development process. By conducting just a few informal interviews, we gained a better understanding of how our patrons use the WINGS portal and what customized library services they would like to see in it.

Students, faculty, and staff have different library use patterns. As such, we are thinking about developing a customizable library system in which the default display varies based on patron type. Since this system would be homegrown, it would be easier for us to maintain and troubleshoot. The WINGS portal is still in its introduction phase on our campus, but we anticipate that more people will use it regularly in the future. We are continuing with our plans to implement the customizable library within WINGS. As the project continues, we will conduct additional, more formal usability tests.