OSI for Libraries: Standards to Services (1992)

4. CONFORMANCE TESTING

4.1 Basic Principles

ISO has developed standardized procedures and a methodology for conformance testing (OSI Conformance Testing Methodology and Framework - ISO DIS 9646). The methodology involves: (1) the development of a set of test cases to exercise the implementation and (2) the development of software to execute the tests and generate test results. The set of test cases is referred to as a test suite which is derived from an abstract test suite that contains a superset of standard tests. The supporting software is termed a est tool.

The actual testing activities, however, are not conducted by ISO. Organizations such as the Corporation for Open Systems (COS) in the U.S., the National Computing Centre (NCC) in the U.K., and the CTS program of the Commission of the European Communities, have developed testing programs and established test centers or laboratories.

To conduct OSI testing, test centers or laboratories offer services which include development testing, conformance testing, interoperability testing, and arbitration testing, explained below.

4.1.1 Development Testing

4.1.2 Conformance Testing

4.1.3 Interoperability Testing

The relative priority and advantages of conformance testing and interoperability testing is a controversial issue. For some developers, conformance testing takes a backseat to interoperability testing because the market, particularly the end-user market, places a higher value on demonstrable interoperability than on conformance. In addition, some developers would prefer to avoid conformance testing because the cost can be prohibitive. However, a consensus appears to be emerging that sees both types of testing as complementary activities: conformance testing demonstrates that an implementation meets the fundamental requirement of conformance to an OSI standard or profile and in so doing simplifies the subsequent interoperability, or "inter-product" testing required between vendors' products.

4.1.4 Arbitration Testing

4.2 Bibliographic Protocol Testing

The National Library of Canada, as the driving force behind the development of the ILL protocol, is committed to its use as the standard for ILL communications in Canada. Because the National Library of Canada works closely with the private sector to develop ILL systems that incorporate the ILL protocol, it was compelled to also develop the test tools and service which would ensure that the products developed conform to the ILL standard. The National Library therefore created an ILL testing facility that adopts the general concepts and components of the OSI conformance testing standards, tools, and services.

The development of the test facility also necessitated the development of a test suite for the ILL protocol that is now being reviewed for standardization by ISO. Test cases selected from the test suite are used by the ILL testing facility to evaluate how accurately ILL systems developed by Canadian vendors meet the requirements of the ILL protocol.

Interoperability testing of Canadian ILL products are also coordinated by the National Library of Canada to determine if systems tested for conformance can interwork in an operational environment. Since conformance testing is relatively exhaustive, interoperability testing is not as thorough and therefore only a small interoperability test suite is needed. This approach reduces the duration of testing and the time involved in coordinating several participants.

The development of testing services for SR and Z39.50 protocols has not evolved to the same degree as it has for the ILL protocol. Neither a test suite nor a test system exists for these standards. One reason is the cost and commitment required to develop an SR test suite and test system. Another reason is that the search and retrieve protocols are not nearly as complex as the ILL protocol. Complexity of implementation lies not with the protocol module but with the development of interfaces to the client and server systems. In the U.S., where there are a few implementations of Z39.50, implementors have chosen to test products among themselves and forego any formal conformance testing.

For the most part, libraries will not become directly involved in either conformance testing or interoperability testing. As purchasers of OSI-based products though, they should be aware of the importance of conformance and interoperability testing and seek evidence from suppliers that the product they are purchasing has been properly tested. This applies to products that support bibliographic protocols as well as to those that support protocols that are more generic such as X.400 and FTAM.