Table of Contents
To successfully implement digital library collections and services, you'll need to get to know your users: their attitudes, behaviors, wishes, needs and desires. The more you know about them, the better you'll be able to provide services and collections that effectively meet their needs.
There are several ways to gather information about your users, including one-on-one conversations, focus group interviews (described elsewhere in this manual) and surveys. This chapter provides a basic overview of surveys.
In general, surveys are carried out to gather information needed to support decision making. There are many reasons a library might want to conduct a survey:
Determine the demand for a planned product or service
Improve an existing service
Find how users rate the library's current products or services
Identify unmet user needs
Discover problems with existing products or services
Use library resources more effectively
Gather input for strategic planning
Surveys, because they gather comparable information from relatively large numbers of people, are good for gathering quantitative information. They can tell you about characteristics of your users: demographics, how often they use library services, their attitudes towards library services. They are less useful at answering "why" sorts of questions; for example, why your users like or dislike a particular library service. Focus group interviews are better at gathering qualitative information, as they allow follow up questions and probing into responses.
Effective survey design involves the following steps:
Articulate survey goals
Decide who to survey
Decide how to administer the survey
Develop the survey questions
Pretest the survey
Conduct the survey
Analyze the results and document the findings
We'll discuss each of these in turn.
This is an important step, frequently overlooked. Drafting and writing down a concise statement of the survey goals will be invaluable as you proceed with the process. It will help you as you draft your questions, select who you will survey, and decide how you will conduct the survey and analyze and use the results.
The group of people you want information on is known as the population of interest, or target population. In a library, that might be the users of the library's services, or those the library is meant to serve. In other contexts, it might be a smaller group, such as employees of an organization. If the population of interest is small enough and the members known, every member of the target population can be surveyed. This is known as a census survey. In cases where the target population is to large for a census survey to be practical, or the identity of some of the members is unknown, only a portion, or sample, can be surveyed.
If the sample is properly selected, survey results can fairly accurately represent the responses you would have gotten from the target population as a whole.
There are two issues to address with survey samples: sample selection and sample size.
There are several ways to select a survey sample:
Random sample: A random sample is one where the probability of a given member of the target population being surveyed is equal to that of every other member, ensuring the people surveyed are representative of the target population. If those selected to be in the sample are not representative of the population as a whole, bias can occur and the results may be inaccurate. [1]
Systematic sample: A systematic sample, selects every nth person in a list to be surveyed. This technique can approach random results unless the list is clustered or structured in some way.
Stratified sample: A stratified sample is used when the target population is composed of several subgroups with differing characteristics, such as faculty and students, to ensure an adequate number of responses from each group. Stratified sampling selects from each of the subgroups a number of individuals to be surveyed, in proportion to their numbers in the target population.
Convenience sample: A convenience sample is one that surveys whoever is easily available, for example, by putting a link to the survey on the library website, or handing out surveys to persons present in the library. This type of survey can provide useful information, but the results should not be viewed as being representative or statistically valid, as there is no way of knowing how closely those who took the survey resemble the target population.
How many people should one survey? Alreck and Settle (1985) [2] recommend a minimum of 30 and a maximum of 10% or 1,000, whichever is less. Factors indicating the need for a larger sample include: 1) The decision that needs to be made is important and costly; 2) The target population is diverse. Conversely, if the decision is less important, the population is homogeneous, or only rough estimates are needed, a smaller sample size may be adequate.
It is true that the greater the number of responses, the more useful information one is likely to get, particularly if the survey has a section for comments. However, going from 250 responses to 1,000 responses will only double the validity of the results. [3]
There are several ways surveys can be conducted.
One on one: An interviewer conducts the survey, either in person or on the phone. Advantages are that one is more likely to get an answer to every question (people are less likely to skip questions) and the interviewer can explain questions that aren't clear to the person being surveyed. A disadvantage is that people may not answer sensitive questions truthfully, or give answers they think will please the interviewer or that are more socially acceptable. Also, one on one surveys are time consuming to conduct, and to compile the results.
By mail or e-mail: Advantages of mail surveys are that people are more likely to answer sensitive questions truthfully, particularly if the survey is confidential, and that they can be conducted more quickly. Disadvantages include the inability to clarify questions, and the fact that mail surveys typically have low response rates.
Internet: Internet surveys have a number of advantages: they are fast and easy to conduct. People are more likely to answer sensitive questions truthfully (as long as they believe their answers will be kept confidential). People tend to provide lengthier comments on Internet surveys than in mail surveys. Internet surveys allow linking to a page or service one is asking about. Software is available that makes creating Internet surveys easy, and compiles the results automatically. A disadvantage is that using Internet surveys could introduce bias in that the persons taking the survey will be computer users and may not be representative of the target population.
Questions are the most important part of a survey. To get the right information you need to ask the right questions. Good survey questions have three attributes: focus, brevity, and clarity.
Every question should focus on a single issue or topic and should be designed to gather a single piece of information. Questions should be as brief as possible. Long questions are harder for respondents to process, and more likely to be understand. Questions should be as clear as it is possible to make them.
Questions should be reviewed by persons other than the ones who drafted them to see if there is any way to make them clearer or briefer.
Keep the survey as brief as possible. Aim for a survey that takes no more than 10-15 minutes to complete. You will lose people with long surveys--they may start skipping questions or answer them without really thinking through their answer.
Avoid jargon: ILS, database, serial. Unless you're doing person to person surveys, there won't be an opportunity to clarify questions. A survey respondent who answers without understanding the question won't provide usable information.
Begin with an introduction stating the purpose of the survey, what you hope to learn from it, and how the information gathered will be used. Ask those being surveyed for their help and thank them.
Questions should move from general to specific, and from easy to hard. Group questions in sections, with transitions.
Provide contact information at the end. This allows survey respondents to clarify questions they might have, or report problems with the survey.
There are two basic types of survey questions: structured and unstructured. Structured questions set the format of the response, such as Yes or No, multiple choice, or asking for a number ("How many years have you worked in the library?").
Unstructured questions are useful where responses would be difficult to categorize (e.g. "If you could change one thing about the library, what would it be?" "Is there anything else you'd like to tell us?").
Most survey questions should be structured, for several reasons:
the information they gather can be more easily analyzed, as responses are comparable
they make the survey taker's task easier by showing the range of responses being sought
responses to unstructured questions must be grouped and categorized, requiring analysis and interpretation
Structured questions fall into several categories:
| Examples: | "What is your status? [ ] Student; [ ] Faculty; [ ] Staff" |
| "Do you have Internet access to the library at home? [ ]Yes; [ ] No" |
| Examples: | "How useful is the library's 'Ask a Librarian' service to you? [ ] Not useful at all; [ ] Minimally useful; [ ] Moderately useful; [ ] Very useful" |
| "On a scale of 1 to 5, with 1 being "Don't use" and 5 being "Use regularly," rate how much you use the following library services: [ ] ILL; [ ] Ask-a-Librarian; [ ] WorldCat; [ ] The library catalog" |
Example:
| Strongly disagree | Disagree | Agree | Strongly agree |
|---|---|---|---|---|
| Library collections in my area of study are adequate | [ ] | [ ] | [ ] | [ ] |
| I prefer to access library materials online rather than in print | [ ] | [ ] | [ ] | [ ] |
| The number of hours the library is open meets my needs | [ ] | [ ] | [ ] | [ ] |
The above example illustrates a question presented in a matrix format. These present information compactly, which can be helpful when used sparingly, but can also be difficult for survey respondents in large doses.
Responses provided to questions should be mutually exclusive.
Responses provided should cover all possible cases. Questions that don't allow persons to give exact responses will frustrate the respondent. One way to cover all possible cases is to provide a "Don't know" or "No opinion" option.
Demographic questions should be put questions at the end, unless you are only looking for particular type of respondent. They are more likely to be answered at the end.
If doing an Internet survey, allow adequate space for comments. A web survey form with inadequate space for comments will frustrate respondents whose comments are cut short, and prevent you from getting useful information. Unsolicited comments on surveys can be golden.
Some articles on surveys advocate giving an even number of choices. The theory is that with an odd number of choices, people are more likely to choose the middle option (e.g. 3 on a 1-5 scale).
One source of survey error is habituation. If the survey has a series of question whose answers are structured identically, users may select the same option for multiple questions. One way to address this is to change the answer format or direction of the scale (1 is best vs 1 is worst). This will make it more likely respondents will address the questions independently.
Pretest the survey with member of survey's target population. Testing will likely reveal potential problems that would keep the survey from yielding useful information, such as unclear questions or instructions.
If doing a Web survey, test it on a variety of browsers.
If you are surveying a predetermined sample via mail, e-mail or Internet, you will want to send out pre-notification letters to let the people know the survey is coming. The letter should ask the individual to help by taking the survey, explain the reason for the survey, when it is being done, who is sponsoring the survey, what the benefit will be, and the confidentiality policy (if applicable). Research shows that letters coming from the top person in the organization, e.g. library director, result in higher participation rates.
Several days after the survey has been delivered, a letter thanking the survey participants and reminding them to complete and return the survey should be sent out.
A week or so any after the survey was sent out, any non-respondents should be sent a follow-up letter reminding them to complete the survey.
In analyzing survey results it is useful to describe the central tendency for numeric data. To do so, the terms mean, median, and mode will be needed.
Mean (also known as "average") is the total of the values divided by the number of values. If Joe scored 87% on a test, Marty scored 95% and Fred scored 72%, the mean test score would be 254 (total of the scores), divided by 3 (number of values), or roughly 86.7%.
Sometimes the mean can be misleading, and other measures of the central tendency are better. If the incomes of the top five officers of a company were:
$2,500,000
$137,000
$107,000
$88,000
$74,000
the mean salary would be $581,200. A more useful statistic in this case is the median salary. The median is the middle case, where half are above and half below. Here the median salary would be $107,000.
To describe the central tendency for category data, the mode is sometimes used. The mode is the category with the highest total. If the enrollment at a university is as follows:
| College of Arts and Letters: | 2,874 |
| College of Business Administration: | 1,595 |
| College of Science: | 953 |
the mode would indicate that a randomly selected student would be most likely to be enrolled in the College of Arts and Letters.
The most commonly-used tools for analyzing survey results are frequency tables and cross-tabulation tables.
A simple example showing analysis of category data using a frequency table:
| Category | Frequency | Percent |
|---|---|---|
| Undergraduate student | 87 | 77.0% |
| Graduate student | 13 | 11.5% |
| Faculty | 9 | 8.0% |
| Other | 4 | 3.5% |
| Total | 113 | 100.0% |
Cross-tabulation tables is another commonly used technique showing the relationship of two categories.
| College | Undergrads | Grad students | Faculty | Row total |
|---|---|---|---|---|
Arts and Letters Row % Col % | 29 70.7% 33.3% | 7 17.0% 53.8% | 5 12.1% 55.5% | 41 100% 37.6% |
Business Row % Col % | 38 90.4% 43.6% | 2 4.7% 15.3% | 2 4.7% 22.2% | 42 100% 38.5% |
Science Row % Col % | 20 76.9% 22.9% | 4 15.3% 30.7% | 2 7.6% 22.2% | 26 100% 23.8% |
Total Row % | 87 79.8% | 13 11.9% | 9 8.2% | 109 |
Other statistical tools are available for more complex analyses of survey results consisting of large sets of continuous numeric data. If interested in more complex analyses of survey data, consult those with expertise in this area or advanced textbooks.
[1] A well-know example of survey bias is that of the Literary Digest. Prior to the 1936 presidential election it mailed out millions of postcards asking people's preference. "Its 2.3 million "voters" constituted a huge sample; however they were generally more affluent Americans who tended to have Republican sympathies. The Literary Digest did nothing to correct this bias. The week before election day, it reported that Alf Landon was far more popular than Franklin D. Roosevelt. At the same time, George Gallup conducted a far smaller, but more scientifically-based survey, in which he polled a demographically representative sample. Gallup correctly predicted Roosevelt's landslide victory." http://en.wikipedia.org/wiki/Opinion_poll
[2] The Survey Research Handbook, by P. Alreck and R. Settle. R.D. Irwin, 1985. 400 p.
[3] The Survey System: http://www.surveysystem.com/sdesign.htm
[4] Survival Statistics: Designing and Using Surveys, by David S. Walonick, c1997-2004 http://www.statpac.com/surveys/surveys.doc