Report of the Sun-Earth Connection Study Team

Report of the Sun-Earth Connection Study Team of the Space Science Data Systems Technical Working Group - July 1998, River Bend Workshop

Richard Bogart (Stanford University), Robert Hanisch (Space Telescope Science Institute), Joseph King (National Space Science Data Center), Roger Pyle (Bartol Research Institute/University of Delaware), David Sibeck (The Johns Hopkins University Applied Physics Laboratory), Ray Walker (University of California Los Angeles), David Winningham (Southwest Research Institute)

Executive Summary

This report describes the system architecture and organization of a data service to manage the data associated with the Sun-Earth Connection theme within the context of the NASA Space Science Data System. We propose a distributed service in which the data sets are managed at a number of sites by scientists actively involved in data analysis. These data providers will be grouped under three service groups: Solar Physics, Terrestrial Environment Imaging, and In Situ Space Physics. The data providers will also be organized into three cross-cutting user views: The Sun as a Star, The Sun in Space, and The Earth in Space. The organization of the proposed data service will consist of a “thin” management layer that will provide a unified budget and reporting path, and a governing council that will ensure interoperability and sensitivity to interdisciplinary needs. The overall budget for the fully organized and operational data service is estimated at $5 million per year.

1 Introduction

Scientific discoveries provide the true return from NASA’s investment in space exploration. To maximize this return, researchers must be able to exploit the rich database of observations from NASA’s past, present, and future missions. Successful data use requires easy location of and access to active and historical archives, as well as the information and tools necessary to interpret the observations. Functions are being incorporated within the NASA Space Science Data System (SSDS) designed to provide common search and access tools across the range of space science. An essential component of the SSDS will be a system designed to manage the abundance of data associated with the Sun-Earth Connection (SEC) theme, with an emphasis on enabling correlative studies both within the theme and within the SSDS. This report recommends a structure for such a service and describes the relation of that service to broader level SSDS activities.

It is both natural (in view of limited budgets) and desirable (for cross-discipline compatibility) that the nascent Sun-Earth Connection Data Service (SECDS) will make use of, and build upon the tools and organizational structure developed by the more mature astrophysics data environment, Planetary Data System (PDS), and other existing systems. These systems have demonstrated that there are many benefits to improving community organization to a greater degree than has been the case within the Sun-Earth Connection theme. Their use of common data systems and formats greatly facilitates the delivery of data. Nevertheless, it is clear that any successful data system must evolve from existing SEC functions and services, rather than being imposed upon the research community. It is also a central tenet of the SSDS that data are best managed by scientists actively engaged in their analysis.

There are some problems in the SEC data environment that are serious impediments to scientific research. These problems include: 1) publicly inaccessible data sets; 2) data sets that are only available in formats which are ineffective for scientific analysis; 3) data set documentation that does not support independent data use; 4) the excessively wide range of data formats in use; and 5) difficulty of locating data in today’s distributed data environment. While some important data sets and services are currently conveniently accessible to the SEC community from a number of sites, we believe the establishment of an SECDS is important. The SECDS will be established to: 1) ensure accessibility to a broad suite of data sets; 2) promote interoperable search and use of data and other services across multiple SEC disciplines and sites; 3) improve the interface through which mission data flow to public-access sites; and 4) promote interoperability across the entire space science domain through participation in and adherence to the standards of SSDS.

These recommendations are an outgrowth of the Community Wide Workshop on NASA’s Space Physics Data System held at Rice University in June 1993, and are guided by the recommendations of the Task Group on Science Data Management. These recommendations are also responsive to NASA’s Science Information Services Study Team preliminary report. They are a direct result of the efforts of the SSDS Technical Working Group (SSDS TWG) to increase interoperability between the various space science disciplines.

1.1 General Requirements

The SECDS will serve the needs of scientists in and across the traditional disciplines of Solar Physics, Cosmic and Heliospheric Physics, and Magnetosphere, Ionosphere, Thermosphere, and Mesosphere Physics. From a user viewpoint, it will be capable of supporting research along three major scientific themes: The Sun as A Star (including helio- and asteroseismology, solar and stellar activity, and luminosity variations); The Sun in Space (including studies of coronal heating, the solar wind, the heliosphere, interplanetary energetic particle populations, and the interplanetary magnetic field); and The Earth in Space (including the Earth’s magnetosphere and upper atmosphere and their response to the changing space environment). From a data management viewpoint, the SECDS will be organized by general data classification: Solar Physics, Terrestrial Environment Imaging, and In Situ Space Physics.

The SECDS is also designed to assist scientists in more diverse fields who need to analyze or correlate data arising from research in different traditional disciplines. It must assume responsibility for all data sets of scientific interest resulting from NASA missions and investigations within the Sun-Earth Connection Theme and the former Space Physics Division, including data from such projects as the International Solar Terrestrial Physics Program (SOHO, Polar, Wind, Equator-S, Geotail), Yohkoh, TRACE, Ulysses, ACE, FAST, TIMED, SMM, UARS, KPVT, IMP8, and Voyager 1 & 2 missions. The SECDS should also strive to provide full access to and interoperability with other relevant data archives, including both non-NASA space missions and ground-based observatories. In cases where such data are important to Office of Space Science (OSS) research and are at risk of becoming inaccessible, SECDS should endeavor to preserve, curate, and archive the data in accessible form.

The SECDS will function as an integral component of the SSDS with the aim of developing a level of connection and interoperability useful to scientists involved in cross-disciplinary research. Particular attention should be paid to coordination with closely related disciplines in other fields, such as stellar activity, stellar winds, asteroseismology, particle physics, planetary magnetospheres and atmospheres, and cosmic ray sources.

1.2 Services

The SECDS will provide the following primary services through a distributed data system architecture (described in Section 3):

1. Direct and rapid electronic access to data sets from all existing and future missions. Since the data sets are currently held at a large number of locations, the SECDS must provide a catalog of data set holdings within the community. It is expected that data distribution via the Internet will continue to increase in popularity; however, provisions must also be made for delivery of substantial data sets on off-line media as appropriate.
2. Arrangements for the permanent (deep) archiving and suitable curation of all scientifically valuable data sets obtained by NASA missions and NASA-funded investigations. As resources permit, the SECDS will also incorporate data sets obtained by non-NASA funded domestic and foreign agencies.
3. Coordination of the development and review of mission data management plans for NASA missions, taking account of community needs. SECDS will identify problems with these plans and will report on the status of these plans to NASA Headquarters. SECDS will provide guidelines to data management plans, documentation, formats, media, and archiving that support standardization within SEC and, where possible, across the solar-terrestrial, astrophysics, and planetary science disciplines.
4. Development of standards related to data formats, media, and documentation through work with community members. Information and expertise on data standards will be provided to both data providers and users of archived data.
5. Assistance in the development of search tools and browse-level data products for access to data catalogs and relevant information, such as spacecraft trajectories, observing times, and object locations, within the framework of the SSDS.
6. Restoration of previously obtained data sets relevant to the SEC theme and support for the development of value-added products. SECDS will identify data sets requiring restoration and/or enhanced levels of accessibility on the basis of provider and user interest.
7. Coordination of activities involving public outreach and education with the relevant data centers.

2 Scientific User View of the System

The scientific user view of SECDS reflects scientific interests that may lie in accessing cross-disciplinary data sets. This suggests that a user interface presenting three cross-cutting thematic categories would be useful. We propose that the thematic categories The Sun as Star, The Sun in Space, and The Earth in Space be implemented at the SECDS coordination level and overseen by the Project Scientist and the Science Members of the Coordinating Council. Each category will consist of a mapping of relevant data sets from areas covered by the three Service Groups (see sections 3 and 4 for details concerning these entities).

As seen in Figure 1, each Service Group is responsible for providing a portion of each thematic view of the data. The exact mechanism by which each thematic view is presented will be determined by the Service Groups.

3 Data Management System Architecture

The overall system architecture of the SECDS is designed to accomplish two goals: 1) to provide users with rapid access to well-documented, Sun-Earth Connection data and 2) to provide efficient management of the SECDS. To this end, the SECDS will consist of three levels that represent different management, user service, and data provider responsibilities (see).

The first level in the SECDS architecture will consist of a “thin” Management Office, an Advisory Committee, and a Coordinating Council. These three bodies are described in detail in section 4.0.

The next level of the SECDS will be composed of service groups organized by scientific discipline or data type. We propose that there be three groups at this level that are data-oriented and reflect similar collection procedures and data structures. These groups will be organized around Solar Physics, Terrestrial Environment Imagery (auroral and energetic neutral imaging, for example), and In Situ Space Physics (e.g., interplanetary measurements and magnetospheric, ionospheric, and atmospheric data sets). The responsibilities of these three Service Groups are given in section 3.1.

The third level in this system architecture will consist of a dynamically evolving set of Data Providers that will accomplish specified tasks including data set management and the development of software tools. These Providers may supply data to one or more Service Groups, depending on their function. There may also be Support Providers at this level, which will be chartered to provide specified software tools or support functions to a particular Service Group. The Providers? responsibilities are discussed in section 3.2.

3.1 Level II Service Groups

The primary role of each Level II Service Group is to identify, acquire, validate, and deliver scientific data sets in its area of responsibility to scientific users and to provide for their permanent archive in a timely and cost-effective manner.

3.1.1 Data Identification and Acquisition

An integral part of the data identification and acquisition process will be the pre- and post-launch interactions between the Level II Service Groups and spaceflight project personnel. These groups will work to ensure the orderly and timely flow of well-documented and standardized data into SECDS archives. To accomplish this task, scientific and technical experts within the Level II Service Groups will interface with project personnel starting early in the project data management planning phases. Data identification and acquisition topics that will be discussed include 1) a Project Data Management Plan (PDMP), 2) relevant standards and guidelines for the preparation and archiving of data and supporting material, and 3) tools and services available through SECDS and elsewhere that will be useful in data preparation and archiving and also for reaching the project’s science objectives.

PDMPs will be reviewed by Service Group personnel for adherence to SECDS guidelines and standards. These personnel will also review (and arrange for reviews of) data and supporting material to be archived as those products are first created and judged ready for archiving by Data Providers (project or PI level) and iterate with Data Providers as needed to ensure that the products are correct, complete, comprehensible, and standardized. Some current or older projects may be too budget-constrained to provide the best organized and annotated still-reversible data set and supporting material. In these cases, Service Group personnel will consult with project-level and/or instrument PI personnel to define a data preparation/archiving activity that represents an affordable effort and the best benefit-to-cost ratio data products in light of anticipated future demand. Data prepared for archiving must be documented sufficiently to support independent use.

Service Group and project personnel will explore the feasibility and cost effectiveness of providing public access to SECDS-adherent data and supporting material from project facilities while those facilities exist. In many cases, Service Group personnel will also work directly with instrument Principal Investigators (PIs) concerning the archiving and public accessibility of data sets and supporting material created at PI sites rather than at central project facilities. Such PI-specific efforts (data products, supporting materials, schedules and pathways for archiving, etc.) should also be addressed in the PDMP.

In interactions with both project-level and PI-level personnel, a key role of Service Group personnel will be to explain the available tools and services. These tools and services will aid in satisfying requirements and may be useful for project and PI data management and analysis.

3.1.2 Data Preparation

Level II Service Groups identify potential Level III Data Providers, work with them to develop mutually agreeable data formats and media, and sponsor (either funded or unfunded) the preparation by the Providers of data sets in these formats and media, along with documentation sufficient to interpret the observations. The organization and requirements of the Level II Service Groups must be sufficiently flexible to be responsive both to the needs of new missions and to evolving trends in information technology as they affect the scientific community.

Service Groups validate data sets and accompanying metadata as they are produced and ingested.

3.1.3 Data Access

Rapid access to well-documented data (including the results of relevant models) is the ultimate goal of the SECDS. With advice from the community and in response to user requests, the SECDS will provide for the most rapid possible access to the digital data within the constraint of available resources. The locations of the data repositories and the means of access may vary with both data set and time, in response to community needs.

3.1.4 Value-Added Services

A distributed, dynamically updated, on-line, searchable catalog of data sets, metadata, software, and relevant models will be an essential feature of the SECDS. The catalog must be consistent with those maintained by the Planetary and Astrophysics communities (i.e., for integration into the SSDS) and must be compatible with the search engines that these communities employ. The catalogs will be automatically updated by the Service Groups and Data Providers using distributed data base technologies, reflecting newly available resources and changes in the status of existing resources. Service Groups, in consultation with each other, the SECDS management, and SSDS representatives, will identify keywords suitable for describing SECDS resources to members of both the SECDS and broader SSDS communities. In particular, these catalogs must be developed in parallel with the ‘search’ and ‘browse’ functions described below, so as to provide a comprehensive view of the data holdings in the system. The catalog must provide a short but complete description including keywords, information concerning the time period covered, and pointers to contact persons and bibliographies concerning the resources. Finally, the catalogs (or subsections of them) must be downloadable by interested users or service providers.

Service Groups, and others, will provide search interfaces which serve as primary entry points for potential data users. By querying the search interfaces with appropriate keywords, users obtain lists of all links to resources matching the query. Possible search criteria include, but are not limited to, time interval, spacecraft, instrument/data type, and region of space. It must be possible to use the interface iteratively to isolate the data set of interest. Lower level search engines must be able to forward unsuccessful or incomplete requests to other elements of the distributed search facilities transparently. The search interface should also provide information concerning available tools, value-added products, contact points, and bibliographies for selected data sets.

Service Groups may produce value-added products and tools such as browse (low resolution) parameters and on-line plotting routines suitable for both standard and interdisciplinary scientific research, or they may commission Data Providers to undertake these tasks. The browser should be able to view both the entire data set and selected short intervals when appropriate. These tools should be applicable across the browseable data sets held by the SECDS. Service Groups must ensure that efforts to develop tools do not duplicate those already being used in the community. Tools should be written as system-independent software to facilitate maximum portability. Such software products must be registered in the searchable catalog.

Service Groups will identify general-purpose tools and disseminate knowledge about them to the community and to the other Service Groups. Under some circumstances, with the approval of the Coordinating Council, they may establish Support Providers to develop general-purpose software.

3.1.5 Standards

Service Groups will develop and/or apply standards for user interfaces, directories, data formats, documentation, node connectivity, distribution and archive media. They will set expectations for Data Providers and develop criteria for evaluating their success. Proposed standards should be considered and approved by the Coordinating Council.

3.1.6 Compliance

The Level II Service Groups will primarily be responsible for facilitation of data access to the scientific community. This will involve encouragement and facilitation of adherence to data preparation and archiving standards and schedules. SECDS Service Groups will periodically report to the Coordinating Council (and, through it, to NASA Headquarters) on the level of compliance to expected standards and schedules by spaceflight projects and other NASA-funded Data Providers. Ultimately, it is NASA’s responsibility to provide incentives for compliance for both NASA and non-NASA official Data Providers.

3.1.7 Communications

Service Groups maintain links to each other and a distributed system of multiple access points to the SECDS.

Service Groups may keep records of user names and data set requests in order to establish usage rates for metric reporting, contact users concerning improvements in service of updates to databases, and identify needs for improvements in service.

3.2 Level III Data and Support Providers

Level III Data Providers will provide data and technological expertise to the SECDS. Typical Data Provider tasks include ingesting data from both active and past missions into the SECDS, entering and updating the characteristics of these data sets into the distributed data base, and providing verified data to users.

Under certain circumstances, Support Providers may be established to prepare tools for the SECDS. Existing data centers and archives will be assimilated into the SECDS as appropriate, and new Providers may be created to respond to specific needs. Level III Data Providers are best viewed as typically transient extensions of Level II Service Groups. Data Providers receive their charter and funding through Service Groups.

All Data Providers will be required to hand the data off to the Service Group prior to the conclusion of the Data Provider?s activities. Upon selection each Provider will prepare a data transfer plan. The data must adhere to SECDS standards for data documentation and be on archivable media. The Service Group will make sure that the data transfer plan adheres to SECDS standards and that the data are in a form that the Service Group can make accessible to the community after the relationship with the Provider formally ends. The Service Group has the responsibility for data access after the Data Provider?s activities end. This does not necessarily mean that the Service Group will provide access to the data directly. Its job is to make sure the data are accessible.

Not all Data Providers need to receive funding from NASA or the SECDS. Other US government agencies or foreign entities may sponsor the formation of Data Providers. If these Data Providers subscribe to SECDS and SSDS data policies, they may associate with Service Groups to become part of the SECDS.

4 Management Structure

4.1 Overview

The management of the SECDS will reflect the system?s distributed architecture. The Management Office, a small, efficient organization consisting of the Project Scientist, the Project Manager, and clerical support, will have overall responsibility for the SECDS. The Management Office will be responsible for the fiscal management of the project and coordination of the science activities. A separate Coordinating Council will set SECDS policy and an Advisory Committee will provide oversight of the SECDS activities.

4.2 Governance

4.2.1 Project Scientist

The SECDS Project Scientist, an individual cognizant and experienced in data management and use, will have overall management responsibility for the SECDS. The SECDS Project Scientist will work to facilitate communication between the scientific community and the SECDS, will represent SECDS at scientific and project meetings, and will chair the Coordinating Council. This will be a full time position. The Project Scientist should be an active research scientist in a field related to the Sun-Earth Connections. It is anticipated that the Project Scientist will spend ~50% time on SECDS management and facilitation activities and ~50% on scientific research.

4.2.2 Project Manager

The Project Manager will have the day-to-day management responsibility for the SECDS. Under the direction of the Project Scientist and with the assistance of the Service Groups, the Project Manager will prepare the annual budget for the project. He or she will oversee contract negotiations with the Service Groups and Data Providers. The Project Manager will be responsible for coordinating the activities of the separate SECDS bodies and for the day to day interaction with NASA Headquarters. He or she will complete and submit project financial reports and will coordinate the submission of financial reports from the Groups or Providers. This will be a full-time position.

4.2.3 Clerical Support

A half-time clerical position will be provided to support the Project Scientist and Project Manager.

4.2.4 Coordinating Council

The Coordinating Council will consist of the Project Scientist (Chair), the Chief Scientists of the Service Groups, the Project Manager, and three members of the scientific community. These community members will represent the scientific themes (The Sun as a Star, The Sun in Space and The Earth in Space) in Sun-Earth Connection research. They will have the important task of assuring that the needs of their disciplines are met by SECDS. They will be appointed by the Project Scientist and will be compensated for their work on the Coordinating Council. The Council will be responsible for setting SECDS policy within the overall guidelines of NASA and SSDS policy. They will make financial decisions based on the budget submitted by the Project Manager. They will also set priorities for data ingestion (mission activities and data restoration projects) and for development of standards and tools for data management and access. The Coordinating Council must approve Data Provider funding. It is anticipated that the Coordinating Council will meet 3-4 times per year.

4.2.5 SECDS Advisory Committee

The SECDS Advisory Committee will provide high-level oversight of the activities of the Data Service. It will regularly review the performance of SECDS and report to the Program Manager, Mr. J. Bredekamp. This committee will consist of members of the scientific community appointed through the Program Manager. It will meet twice per year.

4.2.6 Service Group and DataProvider Management

The chief scientists of the SECDS Service Groups will participate in the overall management of SECDS by their membership on the SECDS Coordinating Council. They will define and manage the ongoing activities of the Groups. In addition, the Group scientists will be responsible for the performance of the associated Data Providers. They will define upcoming work and prepare and budgets for the Management Office. The Group scientists will prepare bimonthly reports of the Groups? activities and accomplishments and deliver them to the Management Office. It is anticipated that each of the Service Groups will establish its own advisory structure.

4.3 Selection of Service Groups and Data Providers

The Management Office shall be competitively selected via the standard NASA peer review process of proposals submitted in response to a Request for Proposal (RFP). We strongly recommend that the Management Office be at a site within the active SEC science community recognized for its scientific, technical and management capabilities.

The Service Groups and Data Providers (many of which are expected to reside in universities and other institutions outside NASA) will be competitively selected. An initial SECDS RFP will be issued specifying that three Service Groups be selected (Solar Physics, Terrestrial Environment Imagery, and In Situ Space Physics). The responding proposals can specify Data Providers needed to acquire the data, models, and necessary software. Individual proposals may be for the Service Groups only or for combinations of Service Group and Data Providers. The participants will be selected by using the standard NASA peer review process. They will be recompeted every 5 years. The specific relationship between the Service Groups and Data Providers will be determined by individual contracts.

Service Groups may solicit proposals from time to time for specified purposes. They will also annually review unsolicited proposals for Data Providers supporting specific data sets or tools. The SECDS Coordinating Council will have final authority to approve or disapprove the Data Providers.

4.4 Interface to NASA

The SECDS will be funded solely by NASA OSS for the benefit of the OSS research endeavor and to meet the OSS requirement to ensure the public availability of SEC mission data. The SECDS is responsible to NASA for the effective use of public funds in pursuit of its goals. The SECDS Project Manager will submit to NASA bimonthly reports on SECDS activities and accomplishments. Under the direction of the Project Scientist, he or she will develop and provide to NASA budgets for future years. The SECDS Project Scientist will arrange for Headquarters-requested reviews of any parts of the SECDS (in addition to hosting regular SECDS Advisory Committee reviews).

Responsibility of the SECDS to the broader Space Science community and the SSDS is ensured via the oversight of the ISSOMOWG and SSDS TWG.

5 Funding Level

Given the breadth of activities and scope of data holdings associated with the SEC theme, and based on a comparison with the other NASA OSS data systems, the full cost of SECDS activities is estimated to be approximately $5 million per year. This includes the costs associated with data services currently funded through active missions. An initial funding level of $1.5 million/year should suffice for the organization and operations of the SECDS Service Groups and Data Providers and the Coordinating Council, ramping up to the $5 million/year level as Data Providers are incorporated into the system.

Acknowledgments

The participants gratefully acknowledge the logistical support of Southwest Research Institute, particularly the input and support of Richard Murphy and Cyndi Farmer.

List of Acronyms

ACE — Advance Composition Explorer
FAST — Fast Auroral Snapshot
IMP — Interplanetary Monitoring Platform
ISSOMOWG — Integrated Space Science Operations Mission Operations Working Group
KPVT — Kitt Peak Vacuum Telescope
NASA — National Aeronautics and Space Administration
OSS — Office of Space Science
PDS — Planetary Data System
RFP — Request for Proposal
SEC — Sun-Earth Connection
SECDS — Sun-Earth Connection Data Service
SMM — Solar Maximum Mission
SOHO — Solar and Heliospheric Observatory
SSDS — Space Science Data System
TIMED — Thermosphere, Ionosphere, Mesosphere Energetics and Dynamics
TRACE — Transition Region and Coronal Explorer
TWG — Technical Working Group
UARS — Upper Atmosphere Research Satellite