GRIDS deployment for NMI releases has demonstrated the benefits of such an integrated middleware package and the need for advanced testing capabilities. The center's leadership and staff are dedicated to developing, implementing, and deploying a comprehensive testing framework and to offer its services to the Grid community.

This transparent and transferable testing framework will address  the increasing demand for timely distribution of robust Grid middleware. Our emphasis is on processes and tools that achieve quality, reliability and stability. The approach allows us to: 1) dramatically shorten the period from component update to release (speed of integration), 2) enhance integration of test modules (quality of integration), 3) validate middleware installations (certification), and 4) determine if an application failure is to blamed on the application, middleware, or fabric (fault isolation). GRIDS has already improved the quality and accessibility of its integrated Grid stack, and plans are underway to extend this capability and transfer it to customized software for the scientific and engineering disciplines funded by the National Science Foundation.

The complete testing framework embodies three classes of activity:

• The harness itself which includes, common presentation and reporting of results, scheduling, grouping/classification, managing tests that cross integrative domains, and more.

• Unit test modules that operate under the harness and are most often provided by component providers.

• Integrative test modules that exercise and test inter-module functionality.

Clearly, all-to-all component testing is infeasible, indicating that selecting and building integrative tests will require experimentation as new software components are added and experience from field deployment is gathered and analyzed. The harness itself has to be scalable and extensible to support thousands of test modules across a variety of heterogeneous architectures and environments. The GRIDS testing framework is moving toward an Open Grid Services Architecture (OGSA) emphasis, with focus on correctness of functionality.

The expanded GRIDS testing framework will incorporate the following principles:

• Close integration of build process and testing process. A smooth transition from the build to the testing phase is essential to provide timely information to both component developers and package integrators. Giving the potentially large and diverse group of component providers, tight coupling between the build and testing processes require common and portable tools and procedures.

• Automatic installation and configuration of middleware on testing facility. All components must support automatic installation /de-installation so that the management software can swap old and new components.

• Modular, extensible and scalable test modules. The complexity and heterogeneity of the software that performs and evaluates tests requires that the testing software be implemented as a well-structured distributed application with well-defined inter-module protocols, so that the Grid community can expand the coverage and functionality of the suite. As the size of the testing facility expands and the duration of functionality tests increases, the software that implements the testing must also scale.

• Flexible management tests and data. By their very nature, testing facilities for Grid middleware are large, dynamic, heterogeneous and may include elements with unpredictable behavior. Each middleware component and test module is likely to generate an independent stream of data. Scheduling the test and recording the data requires scalable, powerful and reliable resource and data management tool.

• Resilience to failures and unexpected behavior. One of the most challenging requirements of Grid middleware is dealing with hardware and or software failures. Therefore, all the software elements of the testing facility must be fault tolerant as well.

• Effective utilization of all available resources. We will never have enough resources to perform all the required tests. Thus, we must make effective use of the resources at our disposal.