Seeking Synchronicity: Evaluating Virtual Reference Services from User, Non-User, and Librarian Perspectives

Background

The time is right to evaluate virtual reference services (VRS):

Web-based library reference services have emerged as vital alternatives to the traditional face-to-face (FtF) or telephone reference encounter.
Synchronous, (i.e., chat reference or Ask a Librarian services) and asynchronous (i.e., email) virtual reference services have grown in number and become common features of both public and academic library home pages.
Many virtual reference services were initially supported by grant money, but support is running out for many and sustainability is a critical issue in these times of extremely tight budgets, rapidly changing technology, and continually evolving service models.

Beyond sporadic—usually quantitative—data, little is known about the participants in VRS, including characteristics such as:

how users determine service excellence
how users rate satisfaction
how and why users choose to use VRS
patterns of user behavior
librarians' satisfaction with their own performance
how librarians determine success and satisfaction

Even less, indeed a negligible amount, is known about non-users of these services and the reasons they do not choose VRS.

See the project proposal (PDF:237K/24pp.) for a detailed discussion of the current research context.

Primary research concept

This international study will:

investigate factors that influence the selection and use of synchronous (chat) VRS
study user and staff perceptions of satisfaction.

It also seeks to develop research-based recommendations for VRS staff to increase satisfaction.

Research objectives

This study addresses the following research questions that are derived from gaps uncovered in the review of the literature.

What are the critical factors that influence users' decisions to select and use VRS? Why do non-users opt to use other means?
What are the critical factors that determine users' perceptions of success and satisfaction in VRS?
What is the relationship between information delivered/received (task/content) and interpersonal (relational) dimensions of VRS in determining perceptions of satisfaction/success?
How do users and librarians differ in their perception of factors critical to their perceptions of success and satisfaction?
What is the impact of the use of prepared scripted messages on satisfaction/success (e.g., "Welcome to our service, a librarian will be with you in a few minutes.")? Do impersonal scripted messages impact user behavior (e.g., promote rude behavior)?
How does users' satisfaction with face-to-face reference encounters compare to satisfaction with reference encounters in virtual environments (including chat and email)?
How do users express satisfaction? Do overt "thank you" messages equal satisfaction/success?

Methodology

This study involves a combination of data collection techniques, including:

transcript analysis
focus group interviews
online surveys
individual interviews with users, non-users, and librarians

Both quantitative and qualitative approaches to data analysis will be employed.

Importance and impact of this work

This project will innovatively address current issues concerning the evaluation, sustainability, and, ultimately, the relevance of VRS for libraries.

The study also will identify ways to increase the visibility and use of VRS, and to improve service.

Improving virtual reference services and making them more attractive to the public should result in increased use and increased recognition that VRS fills an important niche not served by other, more traditional library services.

This, in turn, could help secure funding allocations, and the growth and improvement of services.

Generalizability and Usefulness of Results

The study methodology has been designed to ensure that results will be generalizable through large random sample sizes and through the use of multiple methods of data collection and triangulation of results.

There are preliminary indications that this research will be immediately relevant and useful to the LIS community.

In addition, as each of the proposed research phases are concluded, results will be disseminated, thus enabling immediate impact.

Results

The proposed research has the following intended results:

To address the research questions detailed above.
To identify research-based practices for attracting additional users to VRS.
To understand what users want from VRS in order to develop more effective services that meet the users' information needs and ensure their satisfaction.
To collect information from individuals from diverse cohort groups who are infrequently sampled in LIS research (including non-users, international users, etc.).
To provide research-based guidelines to inform VRS practice and policy.
To refine Radford's recommendations for improving interpersonal communication in VRS for both librarians and users.
To identify factors critical to successful VRS interactions and to develop guidelines and recommendations for evaluation of VRS.
To provide a snapshot of VRS and users in a time of rapid change and ongoing development.
To inform software development and interface design.
To develop a research agenda and to serve as a foundation for future research projects in user-centered VRS.
To develop a theoretical model for VRS that incorporates interpersonal (relational) aspects as well as information (content) aspects.
To provide opportunities to develop research skills and research agendas for masters and doctoral students.

Evaluation Plan

The specific products planned as outcomes for this project include:

Final reports from the four research phases
A series of six papers, each designed to serve both as national conference papers and as journal article submissions. See Dissemination section.
Refinement of recommendations/guidelines for VRS practice that will be made available to VRS providers and system designers for more effective VRS.
A theoretical model of VRS and a research agenda for user-centered inquiry in VRS.
Results from this research will be widely disseminated to the global LIS community.

Schedule

This project will be divided into four phases:

Phase I – focus group interviews
Phase II – transcript analysis
Phase III – online survey
Phase IV – individual interviews.

The project proposal (PDF:237K/24pp.) includes more in-depth descriptions of each phase; the project timeline (PDF:27K/1sheet) provides a schedule of these phases.

Findings / Interim reports / Results

Results from this research will be widely disseminated to the global LIS community.

Resources

All project resources (presentations, reports, conference handouts, etc.) are available from the project home page and the Web site's navigation menu. Key items include:.

Project proposal (PDF:237K/24pg.)
- an in-depth discussion of the project, including background and research objectives, methodology, impact, and expected results.
Project timeline (PDF:27K/1pg.)
- schedule of project phases
Bibliography (PDF:60K/5pp.)

Support

The Seeking Synchronicity project is funded by a grant from the Institute of Museum and Library Services (IMLS) and in-kind contributions from Rutgers, The State University of New Jersey, and OCLC.