- What are the attributes of sustainable digital initiatives and to what extent did the NSDL program demonstrate these attributes?
- What lessons from the NSDL program could be useful to other programs, both within and outside NSF?
The National Science Foundation (NSF) created the National Science Digital Library/Distributed Learning (NSDL) program in 2000. Between 2000 and 2011, NSDL awarded more than 250 grants to support the development of resources, collections, and technical tools and services for teachers and learners in science, technology, engineering, and math (STEM) education. NSDL's mission was to assemble these elements into a distributed digital library that would be widely available to the public through websites. In 2011, NSF discontinued grant funding for projects in order to build on the results of the NSDL program and to establish the next generation of cyberlearning programs.
This report documents the third and final phase in a series of evaluations of the NSDL program. The broad goal of the evaluation was to assess the sustainability of NSDL resources, tools, and services over time and across the changing technology landscape. The report also identifies lessons that could be useful to other programs, both within and outside NSF — specifically, the attributes of sustainable digital initiatives and the extent to which the NSDL program demonstrated these attributes.
- NSDL and the pathways have shown considerable improvement in identifying the audience served.
- The NSDL portal shows strong audience engagement in terms providing capabilities for users to organize, comment on, contribute, and share resources.
- NSDL exceled in development of paradata, which document resource use in the aggregate, and in the collection and harvesting of annotation metadata, which consist of users' comments and ratings for resources.
- NSDL has a strong technical infrastructure, with a robust platform that is well suited to a metadata-based collection portal.
- The NSDL portal and pathways have visionary leadership, but evidence was less clear for staff in operational roles to support a transition from research to sustainable business organizations.
- Broad and diverse parent organizations and other stakeholders are a potential asset with respect to revenue for sustainability.
- The NSDL portal and pathways showed improvements in the authoritativeness of their collections, professional oversight of resource accessions, and basic search services.
- Accurate and complete metadata, alignment of educational standards to resources in metadata, and the expansion of usage metadata are significant contributions of the NSDL portal and some pathways, but few projects have shown how materials can be tailed to meet specific educational requirements or audiences.
Tools and Services
- Funding supported the development of a diverse array of tools and services.
- Although many NSDL projects created innovative products, there were redundant efforts across some projects, as well as gaps in the provision of needed tools and services.
- The degree to which products were integrated with the NSDL portal was variable.
- Improve audience identification by enabling users to customize resources to their workflows, providing a compelling rationale for users to select each portal over competing alternatives, providing stronger evidence of accountability to the audience, and broadening opportunities for audience engagement.
- Improve organization by allowing for integrated security beyond basic user identification and password protection, providing support for users in terms of training and personalized services, and staffing roles to support the transition from research projects to nonprofit businesses.
- Improve collections by publishing a sound copyright strategy, documenting copyright and permitted uses in resource metadata, and enabling searching from the front page of each site.
- Improve tools and services by fostering collaboration among projects to avoid redundancy in development and fill gaps in services and ensuring that projects are integrated with the NSDL portal.
- Future evaluation efforts of the health and sustainability of digital initiatives should use a broader range of methods and data sources, such as data from clients and customers and findings about impact on learning outcomes.
- Cyberlearning initiatives, particularly multiyear, collaborative initiatives, need a much greater focus on sustainability from the outset to balance the emphasis that NSF has traditionally placed on innovation.
- Cyberlearning initiatives should use a combination of initial grant funding, followed by contracts or cooperative agreements, to foster a balance between innovation and sustainability.
- NSF should incorporate concrete sustainability requirements into solicitations and annual reporting requirements — in particular, to target funding directly to products that have been identified as sustainable and worthy of ongoing development and support.
Table of Contents
Overview of the NSDL Program
Sustainability of Digital Collection Initiatives: A Literature Review
Health and Sustainability of Collected Resources
Digital Tools and Services
Usability of NSDL for Lesson Planning
Full Instantiation of the NSDL Logic Model: Replicated Case Studies
Conclusions and Recommendations
NSDL Organization Leadership
Evaluation of the NSDL Technical Platform
Key NSDL Tools and Services
CPALMS and iCPALMS Funding Sources
Proposed Teachers' Survey
The research reported here was conducted by the Science, Technology and Policy Program within RAND Justice, Infrastructure, and Environment.
This report is part of the RAND Corporation Research report series. RAND reports present research findings and objective analysis that address the challenges facing the public and private sectors. All RAND reports undergo rigorous peer review to ensure high standards for research quality and objectivity.
Permission is given to duplicate this electronic document for personal use only, as long as it is unaltered and complete. Copies may not be duplicated for commercial purposes. Unauthorized posting of RAND PDFs to a non-RAND Web site is prohibited. RAND PDFs are protected under copyright law. For information on reprint and linking permissions, please visit the RAND Permissions page.
The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.