GALILEO Assessment Overview
Contents
Executive Summary | Introduction | Methodology | Usage Statistics | Summary of Data | Future Assessment
Executive Summary
GALILEO is dynamic, and for proper interpretation of data, the following must always be considered for the timeframe of interest: (1) number of institutional members and type of users, (2) number and type of databases available, and (3) current design and options of the web-based delivery system. To aid in assessment customized, sophisticated GALILEO usage data may be calculated online.
Guided by the Assessment Committee, four formal assessments have been conducted to date, with differing survey populations. Each provided valuable data and feedback for improvement. Assessment criteria are based on model assessment components from the University System of Georgia, "Assessment Model for Libraries." Eight variables were identified specifically for assessment of GALILEO: reliability, access, responsiveness, communication, understanding the customer, credibility, security, and cost benefit. Assessment data identified a number of concerns and recommended improvements, which have been addressed and are being monitored.
The most recent GALILEO assessment employed an online survey to solicit feedback from end users. Overall the positive results were very high across all six assessment criteria used, ranging from 89% (acceptable response time) to 96% (GALILEO is an effective service). At the time of these assessments, GALILEO was primarily used by academic libraries and a scattering of public libraries. With the explosion in the use of GALILEO among all types of Georgia libraries, future assessments will include a wide diversity of academic, public, K-12, and technical institution library users.
Future assessment plans are:
- Refine the online user survey;
- Administer the online survey at least annually;
- Encourage assessment by individual GALILEO sub-consortia groups, and by individual researchers under Assessment Committee guidance; and
- Expand and refine collection of data on the cost benefit of GALILEO.
[ Return to Top ]
I. Introduction
Background
The GALILEO Project (Georgia Library Learning Online), also known as, "A Vision for One Statewide Library," was initiated by the University System of Georgia (USG) librarians. There were eight components in the original project proposal, of which four relate specifically to the online system. For the purpose of this report, "GALILEO" will refer to the online system; "GALILEO Project" will refer to all eight components of the original proposal.
To understand the context for GALILEO assessment, one must be aware that GALILEO was implemented in the following phases:
Phase I (September 21, 1995 - June 1996)
- 34 University System of Georgia institutions
Phase II (July 1996 - June 1997)
- 56 public libraries (regional headquarters)
- 4 technical institutes
- 41 private college/university institutions
Phase III (July 1997 - date)
- 01 additional public libraries
- 80 public school districts (representing 1,800+ K-12 schools)
Assessment
Usage statistics have been available for GALILEO from the beginning. Important and valuable data are available online where users can request number of searches and full text retrievals by date range and by institution.
Formal GALILEO assessments have been conducted to date, as follows:
| 1. | USG library directors baseline survey | October 1995 |
| 2. | Re-administered initial survey | October 1996 |
| 3. | USG library staff survey | February 1997 |
| 4. | Online survey of GALILEO users | April 1997 |
The baseline and followup surveys (administered in 1995 and 1996) provided valuable data and feedback for improvement of GALILEO. The USG library directors were very pleased with GALILEO overall, but identified issues of concern and specific areas for improvement.
A slight modification was made to the original survey instrument, and questionnaires were distributed (in 1997) to libraries of the USG for distribution to those staff who had the most contact with GALILEO. Credible search results was ranked as the most essential variable, and was the second highest rated variable.
The most recent component of GALILEO assessment includes soliciting feedback from the ultimate customers: the students, faculty and staff of the USG, as well as non-USG users who have access to core GALILEO databases. This was the first time GALILEO mounted an online survey. The number of voluntary responses was gratifying, with 543 usable replies received during the six-day survey period in April 1997. Responses were designed to be passed to a database program; preliminary analysis of responses indicate satisfaction with GALILEO, as 85 percent would recommend it to a friend. The online survey instrument asked users to identify themselves by user type, institutional affiliation, and frequency of GALILEO use. The respondents were asked to reply "yes," "no," or "usually" to six statements concerning GALILEO. Given the opportunity to suggest improvements, 180 provided open ended comments; most often mentioned were: make it easier to use, provide more full text, and improve response time.
[ Return to Top ]
II. Usage Statistics
Sophisticated usage statistics are available for GALILEO, providing valuable information in the aggregate and by institution. These data are freely available from the GALILEO homepage, under "About GALILEO" [http://www.galileo.peachnet.edu/stats/html/stats.html]. Here any user may request a report be calculated by selected timeframe and specific institution(s) and selected type of search(es). Summary data demonstrate a dramatic increase in usage:
| Timeframe | Searches | Full Text |
|---|---|---|
| September 1995 - June 1996 | 1,362,952 | 853,985 |
| July 1996 - June 1997 | 3,523,269 | 1,481,153 |
| July 1997 - June 1998 | 6,442,241 | 2,186,532 |
[ Return to Top ]
III. Methodology
Assessment criteria are based on model assessment components from the "Assessment Model for Libraries," adopted by the USG library directors in 1994. The criteria incorporate variables, use data, and critical incident methodology. For the GALILEO Project assessment, the Model includes:
- mission
- system planning
- GALILEO planning
- collections
- organization
- staff
- services
- facilities
- administration
- fiscal planning
Variables identified specifically for assessment of the GALILEO online component include:
- reliability
- access
- responsiveness
- communication
- understanding/knowing the customer
- credibility
- security
- cost benefit
[ Return to Top ]
IV. Summary of the Data
Surveys 1 & 2: USG Directors
- Return rate
- 1995 (21 out of 34) 62%
- 1996 (33 out of 34) 97%
- Online GALILEO system does a good job
- 1996 (33 out of 33 = yes) 100%
Essential Scores
To determine which elements were considered essential to the success of GALILEO, eleven elements were surveyed; ten were considered essential. The respondents then rated the ten essential GALILEO elements on how well they were being accomplished.
- Reliability was ranked the most essential variable (a 4.94 mean on a scale of 1-5), but had the lowest rating (a 3.39 on a scale of 1-5)
- Cost/value benefit was tied for the most essential variable (a 4.94 mean), and had the highest rating (4.82)
Asked to identify the most positive outcomes and most critical problems for three separate user groups, the library directors submitted free text comments; categorized below are the most cited, in rank order:
| 1995 | 1996 |
|---|---|
|
|
| 1995 | 1996 |
|---|---|
|
|
| 1995 | 1996 |
|---|---|
|
|
| 1995 | 1996 |
|---|---|
|
|
When asked to identify one addition or change they would recommend for GALILEO, the following responses were received from the USG library directors:
| 1995 | 1996 |
|---|---|
|
|
Survey 3: USG Library Faculty/Staff
- Return rate = Not Applicable (distributed to all library faculty and staff "who had the most contact with GALILEO")
- 237 usable surveys received, from library and classified staff (35.98% in the reference area)
- library types were representative of the USG:
- 34.29% Research/Regional universities
- 30.00% 4-year institutions
- 25.71% 2-year institutions
- 10.00% not identifiable
- Online GALILEO system does a good job
- (208 of 211 = yes) 98.58%
Essential Scores
- Credible search results (96.92%) and reliability (96.04%) were ranked as the most essential variables. Credible search results had the second-highest rating (84.00%), but reliability had the second-lowest rating (62.00%).
Survey 4: Online Survey of GALILEO Users (USG +)
- Return rate = Not Applicable (User option: survey available for each user connecting April 9-14, 1997) (543 usable responses received)
Overall the positive responses are very high across all six assessment criteria (ranging from 89 to 96%). The percentage of negative responses is small (ranging from 4 to 11%). Analysis of data seems to indicate that regardless of some dissatisfaction with system response time, the vast majority of GALILEO users (85%) would recommend the system to a friend.
Respondents were definitely pleased with GALILEO's effectiveness (77%) and that it always saved the respondent time (64%). They were less satisfied with response time (41%) and its ease of use (58%). Respondents said that GALILEO results usually or always met the respondent's information needs 93% of the time. Analysis of survey open-ended comments reveals that most often mentioned were: make it easier to use, provide more full text, and improve response time.
[ Return to Top ]
V. Future Assessment Plans
The Assessment Committee is refining the online user survey to adapt it to the expanded GALILEO user base and to allow for more specificity in responses by use of a Likert scale. It is proposed that data be collected annually or semiannually using the online survey.
Efforts are underway to expand and refine collection of data on the cost benefit of GALILEO. The Assessment Committee is designing a form for collecting information to compare costs and benefits with and without GALILEO. The aim will be to collect as much of the data as possible centrally without asking individual libraries to supply it.
Assessment efforts by individual sub-consortia groups within the GALILEO consortium will be encouraged. By use of common data elements and assessment strategies, data can be easily aggregated and combined for assessment input from the entire user base. Individual researchers will be encouraged to conduct assessment projects, under Committee guidance.