SPDF - History

The Space Physics Data Facility (SPDF) was established in 1992 as one of the main archive offices at the Goddard Space Flight Center (GSFC) within the Heliophysics Science Division (HSD) of GSFC’s Sciences & Exploration Directorate. It started as one of the data archive offices within the Space Science Data Operations Office (SSDOO), along with Astrophysics, Earth Sciences, and the National Space Science Data Center (NSSDC). The NSSDC was established in 1966 at GSFC to make space exploration information available to the public. SPDF and NSSDC were partners for many years, collaborating and sharing common resources of both infrastructure and staff. Eventually, SPDF took over full responsibility for archiving heliophysics data from NSSDC, in addition to its work as a multi-mission support project. SPDF and NSSDC work together with the Heliophysics Data and Modeling Consortium (HDMC) project to ensure all online heliophysics datasets have Space Physics Archive Search and Extract (SPASE) descriptions. NSSDC changed its name to the NASA Space Science Data Coordinated Archive (NSSDCA) in 2015. In addition, various project, mission, and investigator web sites provide access to current data, some of which are not yet available through the SPDF archive.

The SPDF leads in the design and implementation of unique multi-mission and multi-disciplinary data services and software. The goal of these services is to strategically advance NASA’s solar-terrestrial program and extend our science understanding of the structure, physics, and dynamics of the Heliosphere of the Sun and to support the science missions of NASA’s Heliophysics Great Observatory. The SPDF works with the missions, with other elements in the Heliophysics Data Environment (HPDE), and with NASA HQ to ensure that the data and supporting material are effectively findable, accessible and correctly usable by potential users from the NASA and international research communities, according to the requirements in the HPDE policy.

One early SPDF project, the Space Physics Data System (SPDS), was born from the recognition that the datasets derived from past, present and future Space Physics Division (SPD) (before adding solar physics to create the field of heliophysics) missions and from related activities of other national and international organizations are a capital asset, and that it is imperative to preserve, distribute and support the synergistic analysis of these data in concert with the overall NASA/SPD program. As knowledge of the Sun, Earth, planets, and the space that surrounds them and their governing physical laws increases, the space physics community requires increasingly sophisticated measurements and combinations of observations in order to continue its advance in understanding the wide range of interactions between regions as well as the collective roles of physics and chemistry in these regions. Therefore, it became clear that new tools and approaches for managing relevant observational data and enabling the correlative analysis of SPD data are required in conjunction with relevant observations from non-SPD sources and physical models. The SPDS served all four science disciplines: cosmic and heliospheric physics, ionospheric, thermospheric, and mesospheric physics, magnetospheric physics, and solar physics. Later, the functions of SPDS were reconstituted into the SPDF holdings, CDAWeb, and the SPASE resource description projects.

The SPDF includes web-based services for browse survey and higher-level experiment data and trajectories to promote correlative and collaborative research across discipline and mission boundaries, in addition to its primary role as the active final archive for most NASA Heliophysics missions, except for solar imaging. The Solar Data Analysis Center (SDAC) is the other active final archive at GSFC, storing NASA heliophysics solar data products.

Major SPDF efforts include multi-mission data services such as Coordinated Data Analysis Web (CDAWeb) and the SPASE-based Heliophysics Data Portal (HDP) (originally called Virtual Space Physics Observatory (VSPO)), science planning and orbit services such as the Satellite Situation Center (SSCWeb), data tools such as the Common Data Format (CDF) software and tools, and a range of other science and technology research efforts. The staff supporting SPDF includes scientists and information technology experts.

The Coordinated Data Analysis Workshop (CDAW) was developed in the late 1970s to enhance the study of collaborative data sets by individuals, large groups meeting in a workshop setting, and small groups communicating via a network. Later CDAWs added interactive plotting and other analysis using X-terminals and IDL data software. When the World-Wide-Web with readily available web browsers in 1993, SPDF converted the X-terminal software to a web browser-based system, called CDAWeb. CDAWeb was implemented with a Perl CGI/HTML UI and IDL backend for the data manipulation, subset/superset CDF creation, listing and plotting. Many data sets are available through the CDAWeb service and the data coverage continues to grow. CDAWeb started with magnetospheric data and nearby solar wind data of the ISTP era (1992-present) at time resolutions of approximately a minute.

The Satellite Situation Center (SSC) was established in the mid-1970s, and also was converted to a browser-based system. It provides orbits from multiple spacecraft in listings and plots, and interactively with the 4D orbit display. It also enables complex queries based on geophysical regions, magnetic field tracing, and ground stations.

The Common Data Format (CDF) was developed in 1985. It provides a conceptual data abstraction for storing, manipulating, and accessing multidimensional data sets. The origins of CDF date back to the development of the NASA Climate Data System at the NSSDC. It originally had three main requirements driving its development:

Facilitate ingestion of data sets and data products into CDF Utilize standard common terminology (metadata) to describe the data sets Development of higher-level applications (e.g., NSSDC Graphics System [NGS]) The CDF library allows developers of CDF-based systems to easily create applications that permit users to slice data across multidimensional subspaces, access entire structures of data, perform subsampling of data, and access one data element independently regardless of its relationship to any other data element. CDF is supported by commercial and open source data analysis/visualization software such as IDL, MATLAB, and IBM’s Data Explorer (XP). Those who are familiar with a language like IDL or MATLAB can easily create sophisticated plots from CDF files instead of writing a lengthy program in C, Fortran, or Java.

The OMNIWeb was created in 2003 at NSSDC. OMNI started as an hourly-resolution 1963-near_current data set with near-Earth solar wind magnetic field and plasma data, solar and geomagnetic activity indices, and energetic particle flux data. The data set was created at NSSDC by interspersing, after cross-normalizing, field and plasma data from each of several spacecraft, and mapping to the nose of the bow shock. The OMNIWeb interface provides access to this data set with graphical browse and subsetting capability, and to higher resolution data sets contributing to OMNI. Additionally, this interface provides solar wind input data for studies of solar wind - magnetosphere coupling.

COHOWeb is another graphical browse and data subset/retrieval tool. Established in 2004, it provides access to a series of spacecraft-specific hourly resolution data sets wherein all records are of a common format and content, with magnetic field, plasma and spacecraft position data. There are data sets for virtually all deep-space spacecraft (Voyager, Ulysses, Pioneer, Helios, etc.)

The SPDF supports direct access to all data files in the archive via HTTPS/FTPS. A subset of these have dataset-specific browse/subset/retrieve interfaces and they can be found in the FTPBrowser. The subset of FTP/FTPBrowser-accessible data that relate to the atmosphere and ionosphere are also accessible through ATMOWeb.

In 1991, the International Solar Terrestrial Physics (ISTP) collaboration was established as the flagship program for the space physics scientific community. The ISTP Metadata Guidelines were created as a convention for describing and naming the datasets. A data set using ISTP guidelines, by definition, forms a logically complete and self-sufficient whole (data and descriptions). The goal is to make the resulting data set correctly and independently usable by the science community and accessible through the CDAWeb Display and Retrieval system. The objective of ISTP is to promote further understanding of the Earth’s complex plasma environment and the Sun-Earth connection. The underlying data standard of the CDF (and later, netCDFs) using the ISTP Guidelines has allowed effective data sharing particularly through the CDAWeb System. The data standard and the growing amount of data available in this standard contribute to the growing success of cross-mission data analysis.

ISTP metadata is independent of CDF and easily used in other self-describing science formats like CEFs, NetCDFs and HDFs, FITS, and ASDF. Global attributes are used to provide information about the data set as an entity. Together with variables and variable attributes, the global attributes make the data correctly and independently usable by someone not connected with the instrument team, and hence, a good archive product. The global attributes are also used by the CDAWeb Display and Retrieval system. There are three types of variables included in ISTP CDF files: data variables of primary importance, support_data variables of secondary importance, and metadata variables. Variable attributes are linked with each individual variable, and provide additional information about each variable. A standard set of these attributes is very important; this is where the information can be stored in a commonly defined manner. Variables are defined with CDF specifications and required attributes. Data variables also have attached variables for time and dependencies (support_data) and labels (metadata). The support_data variables and metadata variables can be attached to data variables.