61st IFLA General Conference - Conference Proceedings - August 20-25, 1995

The Next Generation of Performance Indicators

John Sumsion, Library and Information Statistics Unit,Loughborough University of Technology, Loughborough, Leicestershire, England
and
Suzanne Ward, De Montfort University, Leicester, England

ABSTRACT

This paper derives from research commissioned by the European Commission and undertaken in 1994 by De Montfort University in partnership with Essex County Libraries and the Library & Information Stat istics Unit at Loughborough University. Publication of the full report of that work by the E.C. is scheduled for Spring/Summer 1995. Informally it was known as the Toolbox study. Publication of the findings here is by agreement of the E.C. The very substantial contributions of colleagues to the text of this paper is acknowledged - particularly that of Suzanne Ward, who will be named as joint a uthor in any published version of this paper.

PAPER

This paper is in three parts. After the Introduction Part I considers shortcomings in current performance indicators - as revealed by the recent study - and proposes a general strategy to make best use of present computer power. Particular attention is drawn to the weak area of Cost:Output ratios. Part II outlines proposals to improve and initiate particular performance measures. Part III con tains some observations on the applicability of these techniques to libraries of different type and size.

INTRODUCTION

Although many libraries in Europe and throughout the world still operate from closed access stocks, the more typical situation is one of open access for most materials and this is the situation addre ssed here. While the principles of performance assessment apply to any size and type of library, in practice the opportunities provided by computerised data are so much greater, and the problems of l arge libraries so much more complex, that the performance indicators we consider are appropriate mainly to libraries that run a computerised circulation system.

Performance Indicators are considered in a broad context. As an important management technique there is not intended to be any rigid distinction between them and the I.T. framework for management pr ovided by Management Information Systems (MIS) or Decision Support Systems (DSS) or later developments. Their purpose covers:

monitoring performance
monitoring the results of innovation
identifying problems and opportunities
evaluation of alternative options
planning

The aim of the Toolbox study has been to act as a stimulus to demonstrate not only how performance measurement can be carried out, but how it can be executed by taking advantage of the advanced autom ated environment which will soon be commonplace in all but the smallest libraries.

Having examined the extent of Performance Indicators in practice - and their shortcomings - the study then proceeded to put together a Toolbox of performance indicators and measures. This was design ed both to be comprehensive and innovative. All the main indicators already developed were intended for inclusion, with reference to existing descriptions as appropriate. New and extended indicator s were added to meet general and particular shortcomings identified - and to employ latest computing facilities. This article considers only the most significant and innovative features.

I. SHORTCOMINGS OF TRADITIONAL PERFORMANCE INDICATORS

(a) The current situation in Europe

While the impact of information technology in many libraries in Europe is still limited, a substantial amount of work and experimentation in performance measurement and decision support techniques ha s been revealed. The relevance of performance measurement in libraries is unquestionable given the financial and technological developments of recent decades.

There are two levels of development called for. The first - and the longest - is to raise the overall standard of performance measurement to that already practised by pre-eminent library managers. In each country, and between countries, there are large and crucial gaps between the average and the exemplary.

The second level involves work at the leading edge to produce more effective and flexible systems for the leading institutions. There is substantial scope here to utilise the power of contemporary so ftware.

(b) Shortfall features

If progress has been slow and patchy, this is due to some degree to important deficiencies in the established methodology. This in turn reflect the limitations of systems software and hardware in th e 1980s. Shortcomings include:

Early performance measures concentrated on Resources and Output totals - on how well resources were used rather than how ell users’ demand was met. This perspective is now being corrected.
Outputs typically not related to user features. For example: public library in house use analysable by age and intent of users: academic library journal use analysable by users’ status and depar tment.
Analysis at top level only
Historical data not available for trends to be studied
Cumbersome nature of complete print-outs; inflexible or inadequate query facilities as alternative

Most of these weaknesses can potentially be overcome; the ways in which this can be achieved are discussed in the rest of this paper. However there is one shortfall relating to Cost:Benefit ratios t hat is so significant that it calls for special explanation.

(c) Costs and Performance

The problems relating Costs to Performance are difficult but should not be avoided. The Toolbox differs from a number of previously published guidelines in advocating that cost analysis has to be an essential part of performance measurement.

The main problem can be presented quite simply. The obvious indicator is the Cost per Loan calculated as Total (Net) Expenditure divided by Total Issues. This is extremely c rude and is mostly to be avoided - since it makes no allowance for the extent of non-lending use of the library. Increasingly this is significant where there are:

large reference and information functions
services provided off the premises and through networks
well developed special services to particular groups of users
community services, educational programmes and exhibitions

In broad terms there are two approaches to resolve this problem - both in theory and as evidenced in practice.

Solution 1 concentrates on the output measure by replacing ‘loan’ in the ‘Cost per loan’ formula with a composite measure totalling either all the principal uses provided or all types of document delivery.

A ‘total activity’ example from a Swedish public library comprises:

In UK university libraries a prototype ‘document delivery’ measure is made up of:

Such broad brush formulae represent non-lending activities in a crude but feasible approach. Emphasis on diverse activity can have publicity advantages - although the measures are not readily unders tandable by the public. Less work is involved than in Solution 2 - but the answers are also less satisfactory.

Solution 2 is the Functional Cost Analysis approach where costs involved in every principal service are estimated to produce a separate Cost per loan calculation for each main service provided.

The main feature is the analysis of staff costs according to staff time spent on main functions in each location. The main functions are:

Lending & circulation (adult/children’s combined or separate)
Mobiles
Information & reference
Domiciliary service
Specialist services, eg. archives, local studies
Promotion
Study spaces
User training & instruction
Acquisitions (Incl. Cataloguing)
Regular children’s activities
Special events
Preservation

It should not be necessary to install any sophisticated or routine time clocking procedures. Annually each member of staff should be asked to apportion the percentage of time spent on principal func tions - this to be checked and agreed by management. These percentages are then applied to annual staff cost figures and grossed up by an allowance for indirect and overhead charges.

It is more straightforward to allocate other costs. Materials are conventionally charged to adult lending, adult reference, children’s audio/visual, etc. and/or analysed as books, periodicals, CD-ROM s, music, maps, audio/visual, etc. Premises costs are allocated according to space used for different functions. Automation and other costs are allocated as appropriate - typically in proportion eith er to space or to staff time employed on each function.

The results of this approach are cost:output ratios which are more exact and meaningful than any large composite measure can be. In a large library they are vital for informed allocation of resources and to inform decisions on service expansion.

Why has this not happened already? Factors that have inhibited such a functional cost analysis in the past include:

no help with detailed procedures and inadequate definitions
no obvious application to problems requiring an answer
no requirement formulated for systems work.
lack of appropriate computer hardware and software
fear of accountancy precision requirement

The fourth factor should be overcome by modern relational database software on platforms now within a feasible price range. The fifth factor has been considered above: it is essential not to confuse this system with that required to monitor and control spending levels. The first three factors are - in effect - the subject of this study: there are many decision areas where managers are currently navigating in financial shadows.

Large organisations need to estimate costs in four dimensions:

by type of expenditure: staff, materials, premises, automation (income), etc.
by location: site, service point, information unit
by function: adult lending, children’s lending, information, storytelling, events, reading places, etc.
by subject category and, for some purposes, academic department.

The requirement (for large units) is not to provide extensive weekly or monthly reports covering all levels of measure and indicator. The requirement is to provide facilities for cost:output calculat ions, or cost allocation ratios, on an infrequent enquiry basis as and when these are required to support particular investigations or decision steps. For this purpose the requirement is for data to be readily accessible and conveniently manipulated for special reports and for ad hoc enquiries.

(d) Harnessing computer power to overcome shortcomings

In principle what is required is the ability to access datasets, in convenient enquiry mode, on a much larger scale than has hitherto been envisaged. It should now be within the capacity of the prese nt generation of computer hardware and software to provide what is required.

The requirements derive from the expressed needs of library managers - and from the logic of the situation. It is not a case of putting new technology to use because it is there, but rather that tech nology will now permit what has been a requirement for decades. The library manager needs to be empowered, not inhibited or impoverished. This is the strategic approach that underlies the innovative aspects of the Toolbox.

Working this out represents a major change in approach. Generally systems analysts aim to hold and present only data which is specifically known to be in demand. The proposal here is for data to be h eld for those occasions when it might be required - often in several years’ time.

In this concept data are entered and held with regard to their ultimate not their immediate use. For example to hold date of birth, date of registration and post code area will allow not only correc t calculation of fines and privileges but also analyses of lapsed users and comparisons of users:age/sex profile with residents in areas served. Studies can be made of library use by students by yea r and by faculty/department - to compare with previous years’ experience.

(e) The multi-dimensional Database for Enquiry interrogation

The use of established performance indicators as well as more sophisticated features that include:

functionally analysed expenditure/costs
several years data held and analysed
separate units within the library
separate analyses according to subject disciplines
“real” data corrected for inflation indexes
quantitative measurement of electronic services

The amount of data to be held - by modern industry standards - is not large. However the organisation of such data is a major undertaking relying entirely on Relational database and SQL software. Pr ocedures to maintain a readily accessible historical data file, and to relate this to the next year or two, need to be worked out and maintained with care and adequate documentation. A main objective must be to make the data collection process as easy as possible for the librarian, that is, requiring minimum effort and resource.

(f) Data for other years

To establish trends data are required from recent years. Practitioners feel this is important. However in existing computer systems there have been severe feasibility problems in holding and accessin g historical management data . Such a demand has not been clearly enunciated; neither, until recently, has it been practicable in terms of the hardware and software available in stand-alone systems.

How long is “the recent past”? Practitioners consulted in the course of our study focused on the last ten years rather than the last five. For such a database the “future” should at least cover the budget for the present year and a forecast for the year following. Beyond that lies strategic or medium/long-term planning. There will be important connections between actual performance indicators a nd longer-term plans.

For how many indicators and measures should historic data be held? This is not easy to decide, but the following guidelines should be helpful:

there is no need to hold more than annual results: data for intermediate periods will not be required.
detailed analysis needs to be held either in percentage form or in absolute numbers, not both
at least the first level of breakdown will be required for expenditure, materials, subject categories, user categories etc.
whether to hold data for each site or for the whole library system will depend on the size and structure of the library authority. For all but the smallest units, it will normally be valuable to hold data separately for the central library and for the whole system. Much data should also be held for each service point.
since organisational arrangements and provision of buildings and technological features change unpredictably over time, the holding of disaggregated data is to be encouraged.

II. PARTICULAR PROPOSALS

(Items in this section can mostly be used as measures or converted to give ‘per capita’ ratios of performance)

(a) Analysis of Staff Time

This is to provide data (1) for monitoring, review and decisions on optimum allocation of staff, (2) to publicise to outsiders the variety of tasks undertaken, and (3) to calculate cost:benefit (cost :output) ratios (as described at I(c) above). Sophisticated or routine time-clocking procedures should not be necessary to estimate the percentage of staff time spent on principal functions.

b) Speed of Supply/Reaction by retrospective sampling

Measuring this is properly done by sampling. However it is not everywhere realised that there are advantages in taking the sample at the end of the process rather than identifying sample items at the start. Clerically much less work is involved; it is also less susceptible to manipulation in that it removes the opportunity for preferential treatment for items earmarked as part of a sample. In pr esenting average results the ‘median’ is preferred to the ‘arithmetic mean’. This technique only takes account of items that have been supplied, so the number or proportion of failed actions has to b e a separate measure.

We give here examples to illustrate the technique rather than the comprehensive set of applications.

To determine the time taken to acquire items and to make them available to users the following steps are undertaken:

Take a sample of items that are newly available on the shelves.
Obtain data on the date each item was ordered by the library.
Subtract the order date (or publication date where books were ordered prior to publication) from the current date.
Calculate the median number of days taken for all items in the sample - or the proportion of items made available within n days.

For more detailed analysis each individual stage through ordering, checking, cataloguing and processing can be analysed separately, if data is collected at each point. This is an important measure to identify bottlenecks and delays at different stages in the supply process.

This sampling technique should also be applied to the Overall speed in satisfying requests - the time taken to satisfy all types of request for material not immediately available on site. This covers time taken for delivery of material to be supplied through acquisition , inter-library loan, fetching from other sites/service points , and material recalled where it is already on loan. Each of these is also an indicator in its own right.

(c) User analysis by type of user

Categorisation of users will often be demanded for these measures. In academic libraries this will typically be by faculty/department and by status, in public libraries by age, sex, status and reside nce - such as:

undergraduate students
postgraduate students
academic staff
external visitors
children
young adults
adults
pensioners
non-residents
unemployed
disabled
others

(d) Active Users

To determine the number of active borrowers (those who have had at least one item issued to them in the last 12 months) is possible with most automated circulation systems. If the system, or a signi ficant part of it, is not automated, this can be difficult or impossible. But to limit the count to borrowers is not ideal, so there are ways to extend this to all active users.

To establish the proportion of the population using the library we need to know the number of people from the target population who have used the library during the last year. This can be obt ained by one of two methods: either (1) In a survey of the target population, people are asked whether or not they have used the library during the last year; or (2) Establish from a questionn aire survey the proportion of users who never borrow material and apply this percentage to the number of active borrowers .

Example:: A users’ survey establishes that 14 per cent of users have not borrowed material in the previous twelve months. The number of borrowers who have taken out at least one book or A/V item in the pr evious twelve months from that service point is 12,260. The number of active library users is therefore:

                      12260
                   ------------  =  14,256
                    (1-0=0.86)

(e) Use of Information Services

Counting the number of Reference Transactions handled is not without its problems. It is difficult to achieve consistency when transactions are counted by library staff either on a sample or an ongoi ng basis. However, a major drawback is that success in getting users to help themselves to information - by well signposted stock, for instance, has a negative effect of the Transaction count.

To overcome this weakness a modification to the A.L.A. Needs Fill Rate questions is proposed by including a question:

If you were looking for information (i.e. to find something out), were you successful? YES / NO / PARTIALLY

Did you consult a member of staff?	YES / NO

The results assess:

the success of users in obtaining information on their own,
their success in queries addressed to staff, and
the proportion of users requiring information. The data relate to specific needs on a particular day, and they rely on the users’ immediate assessment of satisfaction.

(f) Space provision

This can be categorised according to purpose:

(i) ‘Library Services’ includes space used for reading, studying, information delivery, computers and any other services delivered to users by library staff.

(ii) Library operations includes receipt of materials, bindery, acquisitions, cataloguing, computing, and management.

(iii) Materials storage includes all areas principally devoted to materials whether open access, closed access, special collections or reserve stock.

(iv) ‘Special events’ includes seminar and meeting rooms, space for groups to meet, and exhibition space.

(v) Miscellaneous includes cafe’s, toilets and staff recreation areas.

(vi) Access measures space required only for access to other areas, ie. corridors and gangways.

(v) and (vi) need not be separated; in small libraries categories (i), (ii) and (iii) will be sufficient.

Measurement of space is useful to review allocation of the space provision and to establish or avoid the need for new building. A further analysis by principal services delivered will also be valuabl e in many situations, eg. space taken up by special collections, local history, special information services, etc. To optimise space use is an important objective.

Measurement of floor areas can often be taken from data used for other purposes, eg. cleaning contracts, insurance quotations. Space measurements should be checked for an annual review, but there is no need for frequent recalculation. Precision in the allocation of space between categories is not essential. This measure should not include space occupied by theatres, museums, concert halls where these are not used for ‘library’ purposes but may be physically on the same premises.

(g) Stock Quality analysed by age and by title counts

The Age of items in stock is a simple calculation showing the proportion of stock more than n, n2, n3, n4..... nx years old. Items are counted relative to year of acquisition or year of publication.

There is more scope for Title counts in sets of Performance Indicators than is indicated in recent literature. To assess the staleness/appropriateness of stock a count of titles that have issued duri ng the year (or quarter) can be compared with the number that have stayed on the shelves. This can be particularly revealing for separate sections of stock, although it does not, of course, cover use on the premises.

Titles added per capita, Copies added per title added, and Titles issued per capita are all indicators revealing the extent to which multiple copies of bestsellers or textbooks conflict with objectives of stock variety and depth of interest.

(h) Service Points and Opening Hours

Service points that open for only a few hours each week, and small informal collections of material, can be excluded from the count where there is a minimum qualification, for example: “Service points open 10 hours/week and more”. Service points can be tabulated by Hours Open. As a base for performance measures such tabulations are of limited value.

Multiplying Service Points by Hours Open (to give Service point hours open) is an improvement, but Total service point hours can give a very misleading picture where there are many service poi nts of variable size open for different total hours per week.

To overcome these problems a new measure is proposed:

Weighted average hours open per week

Definition: Average hours per week library services are available. A weighting factor relating to the size or use of service points is used in calculating the average.

Method: Opening hours for each service point are weighted according to the size/use of that service point to produce an average figure. Weighting factor can be either Issues, Floor Space, St aff numbers, Stock, items On Loan, Readers' Seats or Visits - whichever is most appropriate or critical. For public libraries Issues or items On Loan will be favoured. For academic libraries it may w ell be Readers' Seats (student bias) or Stock (research bias).

Example:

For the purpose of illustration we choose bookstock for the weighted average calculation, which then gives these results for 1993 and 1994.

	                    1993					1994				
	         Bookstock     OpeningHours   Mult'n	Bookstock   OpeningHours        Mult'n
                   '000s				'000s
Central Library    50     x      60    =        3000	 50       x      48      =         2400
Large Branch       30     x      50    =        1500	 32       x      48      =         1536
Medium Branch      20     x      44    =         880	 18       x      42      =          756
Small Branch        6     x      24    =         144	  5       x      24      =          120
New Small Branch    -             -                       4       x      24      =           96
                   106		178		5524    109             186                4908
			
							
Weighted Average   5524                                     4908
                   ----  = 52.1                             ----  = 45.0
Opening Hours       106                                      109

In this example the Service Point Hours count would have shown an increase of 186/178 = +4 per cent, whereas the more realistic Weighted Hours Open shows a reduction of 45.0/52.1 = 14 per cent!

This measure refers to hours services are physically available to users on library premises. Distinction can be made between (a) full services, and (b) partial services. In academic libraries separat e counts are needed for term time and vacations.

III. DIFFERENCES BETWEEN TYPES OF LIBRARY

During the study consultations surprise was frequently expressed that we were covering, in one project, both academic and public libraries. The tradition in the literature of performance indicators i s for separate treatment. We believe it was good to have the combined approach - partly because computer systems generally have to cater for all types. However this does mean that the Toolbox must be viewed sensitively. Many of the indicators and proposals will apply to some types of library only.

Some of the most significant variations are these:

large university library with central and departmental libraries on several sites
university library on one central site
special libraries employing 2 - 5 staff with heavy on-line information load
large public library authority employing hundreds of staff in 30 - 80 service points grouped in areas
substantial public library authority with large central library and several branch service points
city public library
small town public library

The principles underlying the toolbox apply in all these situations, but the appropriateness of particular indicators will vary greatly.

School libraries were not considered as part of this project; special libraries and college libraries have received less consideration than university and public libraries. National libraries or regi onal co-operatives have also not been considered.

CONCLUSION

In practice local and national definitions will be required. It was not an objective of this study to standardise detail - but rather to outline information requirements for future computer systems development. This work is being undertaken in four successor projects launched and funded by the European Commission - DECIDE, DECIMAL, EQLIPSE and MINSTREL - for which this Toolbox study is a base.