GALILEO User Survey 2008 Report

Measuring GALILEO Performance

The annual GALILEO survey is conducted to ascertain user satisfaction and usage habits regarding GALILEO resources. The 2008 survey was open from November 3 to November 9, 2008. All surveys were gathered via the GALILEO website. The survey included questions asked every year, as well as questions asked for the first time about GALILEO content for the e-collections development committee and questions about the new user views that were implemented earlier in the year. No questions on the survey required a response, so the total number of responses to each question varied.

There were 827 total responses after duplicate surveys were removed, down 15% from last year’s total responses (971). The highest percentage of responses came from the K-12 public school consortium (31%), followed by state four-year universities and colleges (20%). The same groups were the highest responders in the previous year.

Respondents were asked to self-identify their roles as GALILEO users. Undergraduates comprised the largest population of respondents (25%), followed by librarians and library staff (20%). In an effort to better gauge end user habits and satisfaction, totals for all the questions asked on the survey have been tallied both with librarians’ and media specialists’ responses included and with them removed. Charts that show counts tallied both ways are included in this report.

On the 2008 survey, as in previous annual surveys, respondents were asked to specify their level of agreement to six statements about their satisfaction with GALILEO. These statements pertained to the reliability, responsiveness, and perceived value of GALILEO. Levels of satisfaction have stayed high consistently over the years, and dissatisfaction levels, which have always been low, have been trending even lower. For the statement, “I found it easy to use GALILEO,” more than 80% have responded that they agree or strongly agree with this statement over the past six years. The mean of the responses has trended up since 2003 from 3.18 (out of 4) in 2003 to 3.34 in 2008. Response to the statement that “GALILEO is a valuable service” has also been consistently rated high with a mean of 3.70 plus since 2003. Agreement with the statement, “GALILEO met my information needs” has been trending up from a mean of 3.28 in 2004 to 3.45 in 2008 (slightly lower than the 3.48 mean in 2007). While agreement that “GALILEO response time is acceptable” has remained consistent over time with a mean between 3.38 and 3.55, changes made in the GALILEO Upgrade to the way pages are loaded have made response time in the GALILEO environment faster; however, anecdotal evidence in the narrative responses of the 2008 survey suggests that perceptions of slowness now relate to services by external vendors, mainly the federated search tool, which encountered performance issues during 2008.

In addition to this set of questions posed annually, a separate set of statements this year addressed user response to the new GALILEO interfaces implemented in March 2008. The new GALILEO environment features user views designed with user-centered design processes and customized for different GALILEO communities (Scholar, High School, Teen, and Library). Respondents were asked to specify their level of agreement to statements specifically pertaining to the new look and functionality of the website. Satisfaction ratings comparing the new user views with the previous GALILEO interface were high with 78% agreeing or strongly agreeing that “GALILEO is easier to use in general with the new look and functionality.” In spite of performance issues with the federated search tool, less than 6% of respondents disagreed or strongly disagreed to the statement that “(t)echnical problems with the new functionality have been kept to an acceptable level.” Charts in the survey report show responses in total, as well as broken out by user and institution types.

The percentage of survey respondents who took the time to offer narrative comments to the “I would improve” question rose to 70%, the highest percentage since the question was first asked in 1998. A summary of responses to the question is included.

In 2008, the GALILEO steering committee charged the e-collection development committee with evaluating the resource collections, both core and consortial collections. The 2008 annual survey provided an opportunity to gather input from GALILEO users about areas where they think resources are missing and which resources or subject areas in the current collections are important to retain. One or both the questions were answered by 62% of the respondents.

Of the 827 respondents to the 2008 survey, 582, or 70%, took the time to offer narrative comments to the “I would improve” question that is asked on the annual survey every year. Of the 582 comments this year, 12% felt that GALILEO is fine as it is and needs no improvement. Some representative positive comments are below.

“I actually enjoy using galileo when finding resources for all my literature papers. I think that the content is very easy to navigate and use. I do not have any complaints.”

“As an educator, I really like Galileo and encourage our students to use it. I think for some students it can be overwhelming, so I engage them with one database and then, once they are comfortable with that introduce them to others. Galileo is a great time saving device and provides students with scholarly information.”

“I find that Galileo meets the information needs of our students after they learn how to use the many, many databases available.“

“I really don't know how you could better Galileo. Galileo is very simple and easy to understand. I believe a first time computer user could easily navigate through the different tabs and such that Galileo has. It has plenty of information that one would need to complete a project, essay, etc.”

"i thnk galileo iz vry uzefol b/c i uze it at mi houze!!
galileo iz a gret wbste 2 uze evn if u r at school!!"

"Continue to enhance the student areas with more sites for them to search for information.
(Thanks for the improvements that were made last year. They're great!)"

While the positive narrative responses and the high ranking users gave to GALILEO in other parts of the survey show an overall satisfaction with GALILEO, this question asks users to offer opinions about shortcomings or areas that need improvement, so it is understandable that the number of responses coded as negative or neutral will be higher than the number of strictly positive responses. Suggestions made in response to this question are valuable sources for informing modifications and future development of GALILEO and have been referred to the change review process.

Over one third of the comments (37%) related to a suggestion to add content, either by general suggestions for more databases (18%) or for specific subject areas or products (11%). Later in the survey, two other questions addressed specific resources or subject areas that would enhance the GALILEO collection and resources or areas currently in the collections that should be retained, and responses to those questions have been coded separately than the responses to the “I would improve” question.

Many comments relate to the new community-customized interfaces that were implemented in March 2008, refering to the look and feel (“I would like a search box on the first screen”), to the organization and navigation (“I have some trouble navigating the site and finding the resources I need”), and to anticipated updates (“I look forward to the new interface for elementary level users”). Some comments referred to performance issues of the federated search product, WebFeat, including problems encountered with server errors and other error messages. Performance issues have been reported to the vendor, and most have been resolved since the survey responses were collected.

Some comments reflected expectations that have been molded by users’ experiences with the Web, such as optimization for mobile formats and “Did you mean?” suggestions for spell checking and search term refinement. Customization options, such as persistent folders and customized search sets created by individual users, were also suggested.

Appendix