Agora Release 2 - UEA Case Study
Vers. 1.4 - DRAFT
5 October 2001
David Palmer & G. Newton-Ingham
Table of Contents
EXECUTIVE SUMMARY
The Agora Release 2 case study at UEA was intended to explore issues relating to the role and use of interlending within the hybrid library from both a users and staff perspective. A group of users was identified based on their extensive use of interlending services to act as user input and two staff members from the UEA Library Interlending Office staff were selected to provide staff assessment.
Both groups received training and were told to use the system for their normal interlending use. Twenty-one high interlending volume users participated, with 18 receiving training and 11 actually using the system to order documents.
Assessment/evaluation was by way of questionnaire at the end of the evaluation period. The questionnaires were reviewed by the Project evaluators and evaluation advisors, CERLIM, located at Manchester Metropolitan University. Functional effectiveness and impact of the system on interlending use/process were the foci of the questionnaires. The financial accounting functionality of the system was not activated nor evaluated.
The evaluation period ran from late April to the first week of June, 2001. During the evaluation period, eighty-one requests were received. Thirteen user questionnaires were completed and returned.
Users were very positive about the system. Eighty percent of respondents found the presentation and general use of the system acceptable. Functionality was also popular with over three-quarters of respondents having a positive view of requesting, reporting and delivery as offered by the system. Critically, a majority felt that the system functionality would work well with the regular level of interlending use and was an improvement over the present system. Further, they felt that the system would save them time and make their teaching and/or research more effective. Respondents did, however, wish to see the blank request form made more easily accessible and found signing and returning a copyright declaration to be difficult to remember. Respondents were most impressed with the requesting functionality and least impressed with the reporting functionality.
All users their usage of interlending services would either increase or stay the same using Agora – none felt it would decrease. They felt that an increase in quality, if not quantity of resources within the system, would also increase their level of interlending.
Users split on how they used the system with half using just the blank request form to place requests and the other half using the search facility as well It was clear that they wished to be informed of the progress of requests and a slight preference for the web over email was expressed..
Library staff were less positive about the system with noting that only in the receiving and transmitting of requests was it superior to the present system. The automation of the requesting & transmitting process was seen positively and it appeared easy to staff who also felt it would save them time and would work well with their usual level of work.
Tracking of requests and communication with both users and suppliers was regarded less positively. Staff found it difficult to locate requests at different stages of a transaction, had problems understanding ‘filtered searches’ and felt that the system did not communicate with them particularly well, especially in regards future actions to be taken. Little time saving was seen by use of the system, and staff were not confident that it would work well with their usual level of requests,. It should be noted that mis-configuration by Project staff caused some of these difficulties as did the unfamiliarity of staff with the system. Staff did acknowledge that further training and familiarity with the system would asuage many of their concerns.
Staff liked the way the system communicated with both users and suppliers for usual straightforward transactions delivering useful information. However, the inability of the system to communicate outside the range of usual messages was a concern.
Some opinions cut across all functions. Staff felt the system was more complex than the present system, would involve more or the same amount of work and, not surprisingly, would also involve changes to policies and the way in which they work. The complexity of the system, both in its appearance and operation, was remarked upon often by staff who often qualified answers with the comment that they had little experience with the system, felt the system to be ‘unfinished’ and this might effect the validity of their responses.
As a general point, configuration was an issue, particularly for the staff assessment. Poor configuration or lack of it in some cases, did negatively impact the effectiveness of the system to both users and staff and pointed out how critical proper system configuration is in a highly automated environment such as a HLMS.
As an overall assessment, the case study showed that users readily accept the HLMS concept as it applies to interlending and it will find an enthusiastic audience. Staff were more reserved but did could see merit in the concept, if not in some of the ways Agora chose to implement the concept. Planning and thinking will need to be done by any institution intending on installing a HLMS but properly configured and supported, it will provide interlending services that are superior to what is available presently.
INTRODUCTION The Agora Project is one of the five eLib Phase 3 hybrid library projects funded by JISC to explore issues concerning the creation and use of ‘hybrid libraries’. The stated purpose of the Agora Project is " to explore issues of distributed, mixed media information management based on an open standards-based platform. This objective includes developing the scalability, enabling infrastructure and change-management tools for successful widespread dissemination and implementation throughout the community." [Newton-Ingham, Greg. Agora Project Plan. Vers. 1.1, 24/05/99, p. 1-1.] In reality, this has meant the definition, construction, and evaluation of a hybrid library management system (hereafter ‘HLMS’) as ‘testbed’ for exploring the concept of the hybrid library. The HLMS essentially attempts to combine, or synthesize various previously separate tasks. It also combines access to an array of resources that would otherwise be quite separate, including both electronic and non-electronic resources. In essence, the Agora HLMS attempts to provide one interface and one system for the discovery, searching, location and delivery of items from disparate resources. The present case study is part of the evaluation process; an attempt to ‘expose’ the Agora HLMS to ‘real’ users and obtain feedback. Each Library Associate participating in the Project has conducted at least one case study which, in total, cover a range of issues regarding the HLMS concept and provide a basis for assessment and further development of the concept. The University of East Anglia, as the lead institution within the Agora Project, has a great interest in exploring the possibilities of the Agora HLMS concept, and in creating ‘change management tools’ that will be of use to the wider library community. UEA also has an interest in exploring ways in which the Agora HLMS concept can be ‘transitioned’ into a potential service for the University. Thus, UEA has decided to examine the question of interlending within the Agora HLMS. This will examine an issues not addressed by other Library Associates. It benefits the University by offering an opportunity to examine not only interlending within the Agora HLMS, but by extension, an opportunity to review and assess current practices in that functional area.
BACKGROUND & CONTEXT
The University of East Anglia is located in Norwich and is a multidisciplinary institution serving a total population of approximately 9000 students, of which 7500 are undergraduates and 2000 postgraduates.
The University offers more than 300 courses in a wide array of areas that cover Arts & Humanities, Social Sciences, Physical Sciences and the Professions. It also offers programs for distance learners and courses for day and evening learners within the larger community.
The UEA Library itself is located in one building, central to the campus, and houses the vast majority of the book and journal stock held by the University. The University holds 800,000 monographs and subscribes to 2,100 current journal titles in print. It also is aggressively incorporating electronic resources with a special emphasis on enhancing holdings in fulltext. The Library has explicitly stated that its strategy is to maximize the use of electronic resources and thus sees the HLMS as a means of furthering this strategic goal.
Within the Library, organizational structure mirrors discrete functional areas with the two major divisions being Services and Resources. The former includes functions such as acquisitions, cataloguing and circulation. The latter is responsible for the content of the Library’s resources, interlending services and for liaison with the schools and departments regarding the development of the Library’s services & resources.
Interlending services form part of the ‘Information and Document Supply Services’ which also includes reference desk services and user education. This combination of services might not seem the most ‘natural’ of mixes and it is the strategic aim of the unit to further integrate the two ‘halves’ of the unit.
Interlending services are an integral part of the service offer of the Library. Servicing, for the vast majority of transactions, faculty and postgraduate students, the Library has averaged 15,000 borrowing transactions over the last 3 years. Of these transactions, 50% were on behalf of postgraduates, 45% on behalf of faculty and 5% on behalf of undergraduates. Over 50% of borrowing has been done by the science schools themselves with approximately 75% of the total borrowed by the Library in the form of photocopies. Approximately 95% of all material is borrowed from the British Library Document Supply Centre. It should be noted that all requests are, by practice, sent to the BLDSC first.
The service is run by 4 staff members who contribute approximately 2.5 fte to interlending as they all have other duties within the Library. The service is a mixture of manual and automated processes. Entry, tracking and transmission of requests to the BLDSC and communication with the BLDSC are fully automated through the use of the ILLOS interlending software package. This software has been in the Library for over 10 years and, having been designed expressly by, and for, interlending services, has been entirely satisfactory.
ILLOS however, is a standalone system and does not interact with any of our other Library systems. User records are entered manually, and until recently, all requests had to be keyed in by hand as well.
It is the user request element of the UEA process that is most ‘manual’. Forms are provided for each media type with the periodical form incorporating a copyright declaration. These forms are available both in the Library and in each School. All the information necessary to process a request and assign it to the proper fund can be entered on the form. Users must either mail the request to the Interlending Office via internal mail, or to walk over to the Library to submit their forms.
METHODOLOGY
The methodology chosen was trial use of the system followed by questionnaire evaluation targeted upon two subject groups, the users of interlending services, and the administrators and operators of interlending services within the Library. A six week period was chosen to allow for a full cycle of requests to be completed and to accommodate the various schedules of the user participants.
The participants were identified by Project Staff for possible inclusion in the case study based on their past use of interlending and the likelihood of participation in the case study itself. An initial group of 40 possible participants were identified, and of these, a total of 24 persons chose to participate in the case study. A list of the participants is appended as Appendix A to this case study.
There are 8 schools represented, including all the schools which are the heaviest users of interlending services. The user group is heavily weighted towards staff (faculty & researchers) with 19 of the 24 in this category and only 5 postgraduates participating. No undergraduates are represented as they comprise less than 10% of overall interlending usage.
Training was offered to all participants prior to the cast study. This took the form of a participative session in the Learning and Resources Centre in the Library of 1.5 hours in duration. The sessions consisted of a MS Powerpoint presentation introducing the system followed by an introduction to the system allowing for each attendee to work on a PC concurrently with the instruction. Help text from the system was also presented in bound, print form to participants. Eighteen (18) of the participants underwent training.
While this was a small group, it was felt that their experience in using interlending services would outweigh their relatively small number. And, as a heavy users of interlending services, it was hoped that a sufficient number of requests would be generated to provide useful scaleability feedback from both the users and administrator’s perspective
There was also the factor that the case study work had to be integrated into the existing work of the Interlending Office and therefore, only a limited number of participants were sustainable. The questionnaire itself isolates three major areas for examination; presentation and general use of the system, functionality, and use of interlending. The intent was to place an assessment of interlending within Agora itself within a context of their expectations for interlending services and separate issues of perception and functionality. The questionnaire was in the form of statements with tick boxes on a discriminatory scale.
To obtain staff input into the case study, a similar approach was taken. Two members of staff within the Interlending Office were trained on the system, asked to adminster the test transactions, and to answer a questionnaire thereafter on their experiences with, and opinion of, the Agora system. One staff member had already participated in the earlier case study and therefore could offer insight on the progress of the system. The questionnaire addressed areas of presentation & general use, receiving & transmission of requests, tracking of requests, and communication with both users and suppliers. The system did not contain, and no evaluation was made of, the financial system with Agora/VDX.
The official evaluation period commenced on 23 April and continued to 9 June 2001. Unfortunately, several late ‘bugs’ and configuration errors needed to be corrected so the ‘effective’ evaluation period for interlending staff ran from 9 May to 9 June 2001.
Questionnaires were distributed to the participants on 25 May with a return date of 4 June requested. Of the 24 participants, 13 returned questionnaires, a response rate of 54%. No questionnaires were returned from any of the persons who did not attend training session. It was later learned that two of these participants were away from the University during the evaluation period.
It should be noted that the questionnaires were anonymised.
The staff questionnaire was distributed on 8 June and returned by the end of the following week.
RESULTS AND ANALYSIS
User Questionnaire
Introduction / General Overview
As noted above, 13 of the 24 participants returned questionnaires and a collation of the questionnaire responses and comments are available as separated documents within Appendix B.
The response rate was good given that the evaluation period coincided with marking of examinations. Indeed, 72% of the participants who attended training returned questionnaires and 100% of those who submitted interlending requests returned completed questionnaires.
The usage of the system varied widely as Appendix A shows. Eleven, or 46%, of the 24 participants used the system for interlending. Of the 81 requests received, just under 35% came from one participant, with two others accounting for a further 30%.
As between schools represented in the study, only one did not submit either a request or a questionnaire. At the other end of the spectrum, 100% of another school’s participants both returned questionnaires and requested items. Additionally, whilst CHE accounted for only 12% of the participants, it accounted for 40% of the use of the system.
For clarity, the description of the results and analysis along the lines of the questionnaire itself; i.e. along functional lines, looking at presentation & general use, requesting, reporting, delivery and a general summary of the system as a whole.
Presentation & General Use - Results & Analysis
In general, users liked the appearance of the system as 75% of the responders to the question (all 13 questionnaires had an answers in this area) agreed that the the appearance of the system was pleasant, legible and that the balance of text and images was appropriate. The one main area of concern in regards appearance was the use of black text on a blue background which was widely criticized in written comments. One user also noted that the extensive use of a blue background ‘wastes’ ink when printing.
Terminology and help text within the system was also viewed positively with over 80% of respondents satisfied with both. It was felt that the language clearly and accurately described the function, and that the help text was both helpful and easily found.
Navigation was generally felt to be acceptable as almost 80% of respondents found it logical, consistent and easy to use. However, over 30% (4 of 12) respondents disagreed that they knew where to go in the system to do what they wished to. This could be due to the general dissatisfaction with access to the blank request form.
Throughout this section, participants noted that the blank form was hard to find, ‘hidden’ from the user, and needed to be made much more apparent to the user. To quote one user “Blank request form would be best as a single icon and not hidden in with User Details. This seems to be a strange place for it to be.” Indeed, 80% of responders found that the form was not easy to find.
There also seemed to be some suggestion that less sophisticated users might have problems with language and navigation. There are several references to potential problems for new users understanding instructions and navigating with one user stating “Had I not had lots of experience with databases I may not have found navigation to be easy”
The major points that emerge from the questions on appearance and general use are that colour combinations need to be carefully thought through, that navigation & terminology needs to be simplified, and access to the blank request form needs to be made easier.
As to colour, given the amount of text on each screen, particularly the hit list, ensuring a pleasant combination of text and background colour is essential. Whilst there was no indication that this was critical to the success of the system, it is a simple ‘fix’ that may have an positive influence out of proportion to the effort to effect the change.
Whilst all participants were sophisticated users of both systems and interlending, it is interesting to note that they obviously saw potential problems with navigation and terminology for ‘simple users’. They see this system as having a wider application, and two, that it needs to accommodate a variety of users. While the focus of this study was not on such users, the comments here are an indication that the system will need to ‘de-jargon-ize’ terminology and simplify navigation. It is interesting to note that about one-third of these ‘sophisticated’ users were unsure of where to go within the system to do what they wished to do – one could expect greater problems with ‘simpler’ users.
Interestingly, in comparison to other studies, the use of the word ‘landscape’ did not attract any negative comment.
Clearly, there is a significant proportion of users who may wish to by-pass the integration of searching and requesting completely and simply use Agora as a route to basic interlending services. The system will have to recognize this use by making access to a blank interlending form clearer and easier. Several users noted the need to ‘distinguish’ the use of a blank form from the search & request functionality – obviously, the integration that the designers of Agora envisaged of searching and requesting is not as well accepted, or perhaps known, within the user community. Another explanation is simply that an increasing amount of information sought is in fulltext format so that the likelihood of moving from a citation to request is decreasing – what is being ordered is so obscure as to be unavailable electronically in any form.
Requesting - Results & Analysis
In looking at any of the three functional areas, requesting, reporting and delivery, there are three major issues to be addressed. One, does it work the way it should; two, does it work in a way that is desirable for the user; and three, what effect will it have on how users conduct interlending?
With one notable exception already highlighted above, respondents were positive about the requesting functionality. Fully 7 of 12 respondents found it to be the most impressive feature of the system and none found it least impressive as compared with other functional areas of the system. In addition, 8 of 11 respondents found it to be better than requesting within the current interlending system, with only 3 finding it to be the same or worse. It also seemed to meet user needs as 75% (9 of 12) respondents felt requesting would work well with their usual level of requesting.
Once again, the difficulty in finding the blank request page was noted. This is of particular importance as 5 of 11 responders primarily used the request form to generate requests, ignoring the search capabilities of the system.
Once found, the request form was easy to use and responders had no problems transferring results from a search to the form. All agreed that requesting was reliable in operation. Some concern was raised in regards knowing when a request had actually been made with one user having a particular problem with requesting from the Shopping List page where no acknowledgement appeared on the screen. This problem was not widespread but it illustrates the importance of immediate confirmation of process success, even where (as with the ‘Requests’ screen within Agora) evidence of proper processing exists elsewhere in the system.
The ability to check potential requests against home location holdings was area of some concern as one-third (4 of 11) of responders did not find it easy to see if the item desired was in the UEA Library. Several users specifically noted the inability to see if individual articles were held as disincentive to requesting. As virtually all (12 of 13) respondents felt it important to know if UEA held an item prior to requesting (6 strongly so), this functionality would appear to be key to acceptance of interlending within the hybrid library concept.
The interaction between the content of the system and interlending was interesting to note with opinion mixed as to the effect of content on the use of interlending
Obviously one might expect that those respondents who primarily used the blank request form would not be as interested in, or effected by, the number and nature of resources within the system. This appeared to be borne out as roughly half of the respondents did not feel an increase in the number of resources would increase their interlending usage. A smaller number felt the same about an increase in the quality of resources but still 30% felt it would not increase their level of interlending. Interestingly, none of the respondents who failed to submit a request cited lack of appropriate content/resources as the reason for their non-use.
None of the respondents felt that content was solely the most important factor in their use of interlending. However, a majority (7 of 11) did feel that a mix of content and ILL functionality was the most important factor with the remaining four stating that the ILL functionality alone was most important. Seemingly, even for those respondents who used the search functionality to generate requests, content was seen only as one factor in interlending use.
However, as noted above, a majority felt an increase in the quality of resources would increase interlending usage. It would seem then, that a concentration on a small number of high quality resources within an HLMS combined with good ILL functionality will prove most popular with users.
Another way of looking at requesting functionality is to ask why it wasn’t used. The only reasons chosen were no time and/or other priorities (4), poor system reliability (2), and satisfaction with the existing system (1). The poor system reliability is a concern but it should be seen in light of 10 of 10 responders stating that requesting documents works reliably and the lack of comment elsewhere citing issues of reliability.
Conversely, other potential reasons for non-use such as lack of appropriate content, poor functionality, poor training and lack of interest were explicitly rejected. It would seem that there was very little inherent in the system itself that inhibited use of interlending within the Project.
In returning to our initial questions, it would appear that users were extremely satisfied that requesting documents via interlending worked well as presented within Agora, but that it did not completely meet their needs. The inaccessibility of the blank request form was a major irritant as was the inability to check items of interest, specifically articles, against UEA holdings. Apart from that, users felt it was better than the present system, met their needs, and was the most impressive feature of the system. It was clear that users differed in their use of the system and this was reflected in their assessment of the relative importance of content and functionality to their interlending use. Use of Agora would certainly not decrease use of interlending with a substantial minority increasing their use. This increase seems to be based on a mix of better quality content and improved interlending functionality.
Reporting – Results & Analysis
The second major area of functionality addressed was how the system reported to users. The same issues of effectiveness, meeting user needs, and the effect on interlending use can be seen here as with requesting.
In regards the basic functional effectiveness of the reporting function, opinion was extremely positive with over 90% of respondents satisfied with the overall performance of the reporting functionality. It should be noted that response in this area was lower than for requesting, mainly because the reporting functionality is a direct outcome of requesting – if one didn’t place any requests, the reporting functionality was mostly unseen.
The system provides two modes of reporting to the user; on the Agora web interface itself and by email. Both methods worked well for users. All nine respondents found it easy to locate the status of their requests on the web interface and 8 of the 9 understood the information presented there. The amount of information presented was also unanimously found to be adequate. Email reporting also worked well with all respondents understanding the emails and finding them sufficiently informative.
For all forms of reporting, users overwhelmingly stated that it would work well with their usual volume of requesting.
Whether reporting actually performed a function that was valuable to the users is a moot question based on respondents’ input. Sixty-six percent (6 of 9) of respondents felt that the reporting function did NOT assist them in managing their interlending transactions. Further, 5 of 7 respondents found reporting to be the least impressive function of the system and reporting was held to be the least important function in determining use of interlending.
However, six of 8 respondents found, email reporting to be useful, and 7 of 8 respondents stated that the Agora reporting mechanism would; (a) be better than the current system which relies on surface mail notification, and (b) work well with their usual level of interlending use.
Strangely, despite finding email notification useful, fully 50% of respondents wanted information about their requests ONLY by way of the web interface with a further 30% wanting both. Only 20% wanted email notification only. It is speculated that users want web reporting and that , given their low opinion of reporting in general within Agora and the apparent approval of email reporting within the system, Agora’s web reporting was a problem.
Some things however, are clear: users prefer a web environment for reporting to directed mail, either electronic or surface; reporting is not terribly important to their use of interlending and is not seen as a factor in their management of their transactions; and, it is preferable to the current system.
Delivery – Results & Analysis
Delivery is the way in which the Agora system delivers documents to the user. In an interlending context, it is tied up with existing institutional policies in regards document delivery. For example, at UEA, we ask the BLDSC to deliver photocopies directly to the user, whilst books are picked up from the Interlending Office. Thus, any assessment of this functionality will necessarily be effected by the ability of both the UEA Interlending Office and the BLDSC to deliver items with existing practices. For the case study, we did not alter our policies or procedures.
As with responding, the response rate was lower relating to delivery as requesting is an obvious pre-condition to delivery and several respondents noted that they had not had the opportunity to assess the delivery functionality. Thus, the overall response rate was only 55%.
Overall opinion of functionality was positive with 75% of respondents agreeing with positive statements. There were, however, some exceptions and one respondent who was decidedly negative.
Overall opinion of the delivery functionality was much more mixed than with requesting for example. Fifty percent thought it no better or worse than the current system of delivery and it received the fewest votes (2) as the function that most impressed the
respondents. However, only 2 of 7 responders also stated that it was the least impressive feature of the system and a majority felt it would work well with their usual level of requesting. All requesters also felt that they were adequately informed of the delivery of an item.
Ease of delivery and timeframe for delivery for both photocopies and monographs was generally acceptable with only one respondent finding it unacceptable in each case. Timeframe is heavily influenced by BLDSC response time. However, it should be noted that a number of monograph requests were marginally delayed due to a search configuration error by Project staff which effectively ‘hid’ requests that the BLDSC had responded to other than ‘Shipped’. This highlights, as further detailed later in this report, the importance of proper system configuration within a highly automated HLMS. The one respondent with a highly negative view of the delivery functionality was one of the respondents worst effected by this error. The respondent noted “What is gained in the request mode is lost in delivery mode”.
Another area of great concern is copyright. In the case study, it was decided to ask for copyright signatures at the time that requesters received their photocopies, rather than as a prerequisite to processing the request. This was done to speed the request process and to test the willingness of requesters to return copyright declarations. In the present manual system, declarations are part of the request form and signed when placing the request.
All copyright declarations were returned but there was no consensus as to when the declaration is to be signed, with virtually an even split between those who prefer it prior to placing a request and those who wish to sign upon receipt of the request. It is clear, though, that regardless of when submitted, requesters almost universally found this to be annoying process and a disincentive to automated interlending use.
A majority of users did not find it easy to sign and return the declaration and a number of requesters noted their displeasure in comments remarking, for example, that “...needing to return individual copyright forms is a serious disincentive” and “Do not like the signing of copyright declaration process”. Given that this will be a feature of any HLMS within the current legal regime in the UK, significant attention will have to be paid to how this process can be made easier for, or more acceptable to, users.
Users also felt that alternative delivery methods were required with online full-text desired. Eight of 11 respondents wanted additional delivery mechanisms within Agora with most comments requesting greater access to fulltext in various formats. One user was disappointed in his inability to download into EndNote, whilst another wanted “...everything fulltext and printable/downloadable” and another stated that “ultimately, downloaded pdf’s from all scientific journal would be my main hope!”
Cost was seen as a factor tempering the desire for online fulltext as only 5 of 11 responders would prefer online fulltext to interlending if it cost more. It would appear that if an acceptable timeframe and ease of interlending delivery can be offered, users are willing to sacrifice some speed of delivery to save money.
In sum, most of the delivery functionality worked reasonably well for what it did, but users want more options, albeit within financial limits. Copyright is a significant issue that needs to be addressed for any HLMS to prove successful. BLDSC and institutional policies and practices will have to take into account the increased expectations of users and adapt, where necessary, to the demands that come with a HLMS.
Interlending staff questionnaire results
Introduction and General Overview
Like some of the users, interlending staff viewed Agora as a good concept but one that requires further thought and work to become operationally effective.
The responses of staff are subject to a number of caveats that they themselves would admit to:
· some questions were not answered as staff did not understand what was being asked due to lack of clarity in the question;
·some questions were based upon anticipated situations that simply did not arise and therefore staff were not able to be commented upon;
·Staff had differing experiences with the system and differing levels of knowledge and use of the system;
·there was a realization by staff that the system was not yet fully configured and therefore all functionality could not be assessed (eg. financial accounting);
·ongoing development of configurations by Project staff during the case study produced a feeling amongst staff that they were not dealing with a ‘stable’ system as yet;
·each staff member was learning the system as they were using it and therefore were making comparisons with an existing system with which they were very familiar; and,
It was also impossible in the time frame allowed to train staff as one would with the implementation of a new system so some answers reflect shortcomings in the training, not the system itself.
Despite the above limitations, the assessment provided by staff is valuable, as it highlights issues that will need to be addressed in any HLMS and provides a useful assessment of system functionality by staff who will deal with it most often and have the greatest knowledge of their role within the interlending process workflow.
Presentation and General Use
Staff were generally negative regarding the presentation and general use of the system. The appearance of the system and navigation within it attracted the most negative comments.
Despite splitting over whether they liked the appearance of the system, neither staff members found the appearance logical, nor did they find icons helpful, or text legible and easy to read. Both felt the appearance of the system was worse than the current DOS-based system. However, they did split on whether there was too much or just enough visual and textual information on each screen.
Confusion caused by the appearance of the system will no doubt decrease with increasing familiarity but simplification of the working screens for staff would appear to be essential as would either better explanation of, or fewer icons. The appearance of the system is not instinctively intuitive so substantial training and documentation will be needed to ameliorate the complexity of the system’s appearance.
Training and documentation were slightly better received but a need for both was clearly expressed. Whilst both staff members felt training was adequate, it would appear that the training provided was the bare minimum required. One member did qualify their response by noting it was adequate for the ‘basic’ level at which they had been using the system. Further, both strongly agreed that extensive training is required to use the system effectively although they split on whether it was easy to learn the basics of the system.
Documentation was not commented on by one staff member who did comment however, that the “documentation did not always seem to relate”. The other staff member found the documentation helpful “to some extent” and did feel that further help text and documentation was required. A ancilliary effect of the relative lack of documentation within and external to the system is that it made it more difficult to diagnose and fix ongoing problems or mis-configurations within the system. This increased the level of confusion and frustration felt by staff members.
Given comments elsewhere by staff about the complexity of the system, implementation of any HLMS along the lines of Agora will require more system help, better documentation and more extensive and better training than what was possible within the context of this case study.
Navigation within the system was, for the most part, seen as a problem. Neither staff member found it easy to know where they were, or where to go to perform a task within the system. However, they were split on the ease of navigation, once they knew where they needed to go. Some features were seen as helpful, particularly the Selection Manager and the Worksheet tabs at the bottom of each worksheet. Conversely, the icons were not seen as useful for navigation purposes and one staff member felt that the amount of information on each screen hindered navigation. Overall, both felt navigation was inferior to the system currently in place.
Strangely, one staff member found navigation to be logical but neither intuitive nor instinctive. However, the same person qualified their negative assessment with the observation that ease of navigation would improve with regular use of the system.
Overall, in looking at issue of presentation and general use, improvement is needed, particularly in the area of navigation and the appearance of the system. Ease of use is critical to the acceptance of any system by staff and improvements in this area will enhance the acceptance of the enhanced functionality of other aspects of Agora.
Requesting/Transmitting Requests – Results & Analysis
Requesting & transmitting requests scores most highly with staff members amongst the functionality of Agora on offer. Staff felt that locating an incoming request, checking it, and transmitting it was straightforward, easily understood and that there are not too many steps in the process. Both staff members also felt that the auto-mediation was easy to understand. The elimination of hand-written requests, batch processing and the auto-mediation of requests were favourably commented upon, with one staff member stating,
“I think that the functions for sending off requests and notifying readers by email are really good and area an improvement on our present system. The information entered by users is clearer and the transmitting process is very simple to use and quick”
Both staff members agreed that requesting and transmitting requests with Agora was superior to the Library’s present system, noting that they felt they would save time entering & transmitting requests, and that it would work with the usual number of requests processed by the Interlending Office. In short, they liked it.
On the negative side, both felt that not enough information about what was occurring, or needed to be done was given by the system. This would seem to emphasize the earlier point about the need for further internal and external documentation with Agora.
Another area of concern for staff was the ease of changing information within incoming requests. This is an area where further work with the system would undoubtedly assist staff members but the method of altering requests could be made clearer in documentation and system help. It is also interesting to note that the method of modifying a request before and after transmission is different and this might have caused some confusion.
Interestingly, given their endorsement of requesting within Agora, both staff also felt that use of Agora would increase both the amount and complexity of their work in this area. The only explanation would seem to be that the system is better, regardless of the of complexity and work level.
As well, both felt that use of the system would entail changes to the policies and working habits of interlending at UEA. This certainly isn’t necessarily a negative as, for example, the work of manual inputting of requests would be eliminated, freeing time for other tasks. Nonetheless, these are changes that would have to be managed.
An issue is the degree to which the quality of the request itself effects the auto-mediation process. In a fully auto-mediated environment, a request with incomplete or incorrect information can cause false results, and unnecessary work.
For examples, we did encounter several instances of ‘false’ hits where the system ‘thought’ the item was held at UEA but in fact was not. In one case, this was due to the item being missing and in others, particularly with journals, it was because we did not hold the requested issue.
Elimination of false hits on location searching depends upon better bibliographic data within incoming requests, and on an improvement in the sophistication and depth of Z39.50 searching. In the case of journals, provision of a link from article hits to a search of the local catalogue would be very useful.
An important issue is how auto-mediation within Agora ‘maps’ to the interlending practices of UEA and other institutions. The case study system only contained UEA and the BLDSC as locations within the locate & request rota. This was certainly acceptable in the context of a case study with limited number of requests where, in any case, over 90% of all requests are satisfied from the BLDSC. The problem lies in the fact that the remaining 10% of requests, which can be substantial in number, come from a very wide array of sources, all of which need to be within the rota for the system to handle them. Significant configuration time will be required to set up these locations for any site implementing a HLMS – perhaps this is an area where libraries and HLMS designers can work to together to agree a set of locations available in every system.
Staff approve of both the concept and implementation of receiving and transmitting requests within Agora. Further familiarity with the system and technical improvements should only increase the acceptance of the part of the Agora functionality.
Tracking Requests – Results & Analysis
Tracking requests was, unfortunately, less popular with staff than entering and transmitting requests. Staff was less confident in the system in this area than with requesting & transmitting. As a functional area, tracking has far more variables, and because of the extensive configuration needed, perhaps also more prone to error or malfunction than requesting.
The very first problem was identifying requests at a particular stage of their transactional history to allow for tracking to occur. ‘Filtered’ searches are an extremely important factor in enabling tracking and are a new concept to staff, whose present system, in effect, as ‘pre-set’ filters that present transaction ‘groups’ to users. The filtered searches are much more flexible and customizable than the current system but also more prone to error or misunderstanding for that reason.
The identification of individual requests by request number posed few problems although the addition/use of audit numbers caused some confusion initially as did the proliferation of available ‘indexes’ within which to search for the request. The concept of filtering was certainly accepted but it’s implementation was less successful as neither staff member found it either easy or intuitive. As one staff member stated, “Using filtered searches is logical when you know what you need but it is not intuitive”
Problems with filtered searches were due to several factors. One was the sheer number of attributes that one could filter on and the confusion as to what they meant within the system. As one staff member remarked: “Whilst the concept of ‘filtering’ is excellent, having looked at it on a number of occasions, it at times seemed near impossible to locate a ‘term’ you require, & even then could take a long time to find. Some of the terms mean nothing.”
Another factor was mis-configuration of filtered searches on the part of Project staff. This occurred where the configuration did not produce the intended results due to mis-setting of attributes, or, where necessary attributes were missing altogether. This produced searches that, on occasion, ‘hid’ requests (eg. where the BLDSC had emailed a response other than ‘shipped’), or did not meet the needs of staff. Significant knowledge of both institutional interlending practices and search configurations are needed to ensure that this problem is avoided.
Apart from the issue of filtered searching, staff unfortunately did not find it easy to locate a request, identify it any stage of the interlending process nor be informed of the changes in status of requests (see related comments in ‘Communicating with Suppliers’ section). Nor did they find it easy to have the system inform them of actions to be taken in future.
In fact, the entire area of system communication with staff itself was an area of great concern. There are many instances where staff must carry out an action some time in future on a transaction and it was found that this was difficult to do within Agora. This functionality is particularly important when following up on ‘chasers’ with the BLDSC. Project staff attempted to work with the system to provide a ‘bring forward’ or ‘file tidy’ facility within Agora but none of the attempted solutions proved successful. Resolution of this problem is very important to enhancing Agora’s functionality in tracking 7 monitoring requests.
There was a split on whether there was enough information presented about individual requests. It might be, with the staff who felt there wasn’t enough information, that it was more a case of inability to have all the information in front of them at one time. On the other hand, some information was found in odd places. For example, explanatory notes clarifying messages from the BLDSC were found in a field labelled Errors. Rationalization of status information into meaningful fields and a consolidation of information into fewer screens would help staff in this regard.
Staff felt there would be no time savings tracking requests using Agora, and were not confident that they could adequately track requests with the usual interlending volume. They felt the complexity and amount of work would increase using Agora and agree that significant changes in work patterns would result from adopting it. Interestingly, they split on whether changes in policy would be required.
Management of copyright was an area of interest for staff as well as users. In particular, the need to match paper signatures with an electronic record was seen as mildly retrogressive to present manual process where the signature accompanies the request. Regardless of when a signature is asked for, there will be this additional step in a mixed manual-electronic environment mandated by current statute.
In sum, staff did not like the tracking functionality and agreed that reporting on the status of requests is worse than in the present system.
In order to improve the performance of the system in this area, and the perception of the system, an effort will need to be made to ‘de-mystify’ functionality. More extensive training, documentation and use will no doubt help but simplification of screens, less complexity in the choices available for filtered searching, and a greater emphasis on proper initial configuration will also be needed. The power and flexibility of the system offers many possibilities unavailable at the moment, but complexity that accompanies the power and flexibility must be managed carefully in order to allow staff to properly exploit the advantages of Agora.
Communicating with Users – Results & Analysis
Communicating with users is essential to any interlending system and Agora has some interesting and useful features in this regard. Integration of email communication with users is an improvement over the present situation of surface mail communication, both in terms of speed and workload.
Staff felt the system delivered needed information, and enough of it. In addition, they agreed that they would save time using it, and liked the way in which the system communicated with users. There was also satisfaction with the elimination of the present paper process of notification, which is dreary for staff and labour-intensive.
However, they also felt little confidence that the system could handle the usual volume of requests, and thought the complexity and amount of work would increase with use of Agora. They also felt changes would need to occur in their work habits and interlending policy. Both felt Agora was worse than the present system in communicating with users.
It seems clear that staff liked the concept of user alerting within Agora. As one said, “...the theory behind communicating with readers by preset text emails is good...” and “I think that the functions for ... notifying readers by email are really good and are an improvement on our present system”.
Due to some minor configuration problems early in the case study, one participant did not receive notification of receipt of items. There were also configuration errors in generating letters to users upon the basis of an unfilled request message of some sort. Once discovered, the problem was easily rectified but it did demonstrate again the importance of proper configuration within a highly automated system. Once the proper parameters for the generation of a user alert are determined, the production of an alert is relatively easy and was found to be reliable and consistent.
The latter point is supported by concerns about how to communicate with the reader in a situation not covered by the system user alerts. Due to either unfamiliarity with the system functionality, or because of a lack of functionality, communicating with users outside the normal range of alerts was found to be impossible within Agora. This does create a problem in ensuring that all actions taken within a particular transaction are recorded within the system. Notes can, and were, added to the system but absent a note, the audit trail within the system would have no record of the interaction with the user outside the alerting process. As one staff member stated, “ I think we would need far greater access for creating individual letters etc. to readers. Many requests have a need for unique letters [to be] created.”
Although a majority of transactions can be dealt with using the user alerts, there are sufficient numbers of requests requiring ‘non-standard’ communication between the Interlending Office and the user to justify a means of incorporating ‘non-standard’ communication within Agora. This would lessen confusion and incorporate all actions relating to a request within the ‘umbrella’ of the Agora system.
It would then seem clear that the user alerting functionality within Agora is a welcome improvement over present practice, and, given proper configuration, will be able to handle much of the communication with users that is currently paper-based. It is equally clear, however, that the difficulty communicating with users on an ad hoc basis causes concern to staff and lessens the acceptance of the system.
Communicating with Suppliers – Results & Analysis
Communicating with suppliers is the lifeblood of any interlending process. In the UK context, this invariably means communication with the BLDSC which is the overwhelming supplier of choice by most academic libraries. Communicating the right information, at the right time, with the least effort should be the aim of any interlending system.
Within the context of the case study, it was decided that, given time and manpower limitations, only the BLDSC would be configured for communication. This was deemed acceptable as it supplies almost 95% of UEA’s interlending materials.
Agora passes the initial test of being able to communicate with the BLDSC in both directions which, in itself, is an improvement over prior releases of Agora. Using Artemail protocol, Agora was able to both send requests to the BLDSC and receive the daily ‘intray’ from the BLDSC. Records were updated dynamically to reflect changes in transaction and authorisation status, and can also receive & integrate a variety of BLDSC codes. This is a major improvement over the present system which relies on a manual update of transaction records of the intray contents.
Communication was reliable with no known instances of there being a failure to either send, or receive, requests & responses with/from the BLDSC once configuration was correct. There was some initial problems with the configuration of the account with the BLDSC but this was resolved relatively speedily with consultations between Project staff, the BLDSC and the Project technical suppliers.
Communication with other suppliers was not tested but would appear possible. A policy issue, however, for interlending operations, however, is whether the effort of configuring every possible source is worth the effort given the relatively low number of requests satisfied from those locations. If not, such transactions/communications will be outside the system and will have to rely on manual notes on the Agora record to maintain an accurate transaction record within Agora. It should also be noted that if other locations do not employ the same protocols as within Agora, manual updating will also have to be performed. The eventual adoption of ISO-ILL will hopefully ease this problem.
The staff reaction to communication with suppliers was mixed. Both staff felt that their relative lack of understanding of Agora may have influenced their responses. It would seem that they disliked the functionality but were more positive in assessing Agora’s impact on their work.
Dissatisfaction was expressed with the ease of communication for both regular and special case tasks/events and both staff felt that it was worse than the current system. Opinion was split on whether the system provided needed and enough information to the suppliers but it should be noted that we did not have any technical problem with the BLDSC receiving and processing our requests.
As with communication with users, concern was expressed regarding communicating with the BLDSC outside the usual expected messages within a transaction. One comment was that “Supplying info. to the supplier can often need changing (via editing the req.) several times before the DSC can supply/(or not). This does not seem particularly easy to do” and another noted “The capability of communicating special instructions to supplierssounds more complex than the current system”.
Neither staff member was confident that they could communicate adequately with suppliers under normal workloads. Both were united in their opinion that Agora would change the way they work in this area. The elimination of the manual transfer of the intray contents and the need to use filtered searches to ‘discover’ communications from the BLDSC would be the major changes envisaged.
One notable problem in this area was the inability of the system to properly email the ‘copyperson’ within the system with a copy of the daily intray when in contained messages from the BLDSC. This proved to be quite important as a mis-configuration by Project staff of a filtered search meant the such requests/messages were temporarily ‘lost’ within the system. Lacking the back-up of the intray to reveal this error, staff remained unaware of the changes in the requests for some time. Given proper configuration and training, the need for this back-up will no doubt lessen, but a highly automated system creates a dependency on the automated processes and a back-up record of communication with suppliers will be a useful counterbalance to this dependency.
Both staff liked the way the system communicated with suppliers. Neither staff member felt their work level would increase but they were split on whether the complexity of their work would increase and whether they would save time communicating with suppliers. It would seem, however, that if the problem of communicating with the BLDSC for non-standard requests is resolved, this, combined with proper & comprehensive configuration of filtered searches, should allow the advantages of the system in regards standard requests to come forth into greater prominence.
It should also be noted that staff recognized that the dynamic nature of communication between the BLDSC and Agora, particularly in sending information to the BLDSC, would result in quicker response time to users as compared to the once a day communication using Artemail in our present system.
In sum, Agora handles the basics of communication with suppliers well and the dynamic interaction between the requesting institution and the lending institution is a real step forward. As with all other parts of the Agora system, the highly automated nature of the functionality makes proper and complete configuration of automated processes absolutely critical.
CONCLUSION
Acceptance of the hybrid library concept and it’s manifestation in a HLMS is dependent upon staff and user groups both finding value in the system and being convinced that it is superior to whatever the current system provides.
The most positive conclusion from this case study is that the concept of the HLMS is accepted by both users and staff. Users may well have other uses for an HLMS that need to be accommodated, but the functionality is certainly seen as valuable and an improvement on present practices. From the staff perspective, the concept of integration and automation of process was accepted, particularly with receipt & transmission of requests, but staff were less sure of the operational implementation of the concepts in the Agora format and still preferred, in some cases, the existing system. With greater support and familiarity, most of these reservations can be eliminated as the staff themselves admitted.
It is also clear is that each technical advance raises the ‘bar’ of expectations by both users and staff.. Users quickly accept and indeed expect technical innovations such as the use of an online form, cross-domain searching and the integration of the two. However, they also voice a strong demand for checking of local holdings prior to submission of a request, particularly for articles, and would like to see the blank form much more apparent to the user, just not there somewhere.
Likewise the staff readily greatly appreciate the value of full system interconnectivity with the BLDSC resident systems and the dynamic updating of requests but also see a need for the system to integrate non-standard communication with both users and suppliers.
One issue that cut across either group was the importance of system configuration. User dissatisfaction in several cases could be directly linked to a configuration error and staff perception of the system was highly dependant upon the quality of the configuration. In functional areas where few problems were encountered, staff responded to the ease and speed that automation provided; where configuration was incomplete or not understood, confusion and uncertainty resulted. The highly automated nature of the system necessitates accurate and complete configuration, and . any implementation of a HLMS will require considerable thinking and effort by the institution about it’s processes and policies and how that ‘maps’ onto an automated environment. With such forward thinking, the undeniable benefits of a highly automated and integrated interlending functionality within an HLMS will be more readily apparent and appreciated.
The other important finding coming out of the case study that is the importance of ease of use. Functionality that is masked will not been seen or appreciated. Users were emphatic in wanting the online request form to be more visible regardless of its functionality and disliked the copyright process, seeing it as an ‘additional’ functional burden. Likewise, staff often mentioned the complexity of the system as a negative factor, lessening their ease and acceptance of the system. Lessening the system visual ‘overload’ for staff, providing better training & support to staff, and simplifying/clarifying navigation paths for users would help in this regard.
The hybrid library is simple in its conception but complex in its implementation. If the simplicity of the implementation can match the simplicity of the concept, both users and staff will gladly accept and use the hybrid library.