We have identified three types of variables to be included in ISTP CDF files: data variables of primary importance (e.g., density, magnetic_field, particle_flux), support_data variables of secondary importance (e.g., time, energy_bands associated with particle_flux) and metadata variables (e.g., a variable holding “Bx,By,Bz” to label magnetic field). Variables are defined with CDF specifications and required attributes. Data variables also have attached variables for time and dependencies (support_data) and labels (metadata). The support_data variables can be attached to data variables via DEPEND_i variable attributes. Metadata variables can be attached to data variables via LABL_PTR_i variable attributes (see below).
Examples of data and support_data variables commonly found in ISTP investigations are shown below. They are mapped to their corresponding dimensions and sizes in CDF.
Data is always either Real or Integer type. Data is always time (record) varying, but can be of any dimensionality. Real or Integer data are always defined as having one element.
The following variable attributes are required.
Example of a simple scalar data variable.
We show here the variable, Ion number density, as it would appear in a CDF Skeleton table. We include all required variable attributes. Some
recommended variable attributes are also shown. See the Display of this variable.
! Variable Data Number Record Dimension ! Name Type Elements Dims Sizes Variance Variances ! -------- ---- -------- ---- ----- -------- --------- !"SW_P_Den" CDF_REAL4 1 0 T
! Attribute Data ! Name Type Value ! -------- ---- ----- "CATDESC" CDF_CHAR { "Ion number density (Solar Wind " - "Analyzer), scalar" } "DEPEND_0" CDF_CHAR { "Epoch" } "DICT_KEY" CDF_CHAR { "density>ion_number" } "DISPLAY_TYPE" CDF_CHAR { "time_series" } "FIELDNAM" CDF_CHAR { "Ion Number Density (CPI/SWA)" } "FILLVAL" CDF_REAL4 { -1.0e+31 } "FORMAT" CDF_CHAR { "f8.3" } "LABLAXIS" CDF_CHAR { "Ion N" } "UNITS" CDF_CHAR { "cm^3" } "VALIDMIN" CDF_REAL4 { 0.01 } "VALIDMAX" CDF_REAL4 { 1000.0 } "VAR_NOTES" CDF_CHAR { "Assuming no helium (0.3 - several " - "hundred) if the density is less than " - "0.3/cc the higher moments (VEL,TEMP) " - "shall not be used because of the poor " - "counting statistics." } "VAR_TYPE" CDF_CHAR { "data" }.
Example of a vector magnetic field data variable. We show here the variable, Vector Magnetic Field, as it would appear in a CDF Skeleton table. We include all required variable attributes. Some recommended variable attributes are also shown. See the Display of this variable.
(Magnetic Field does not need a DEPEND_1 because it does not depend on any support_data. In past versions of the ISTP Guidelines we tied vectors to their coordinate system, but this is not really needed and so we have dropped the requirement. It is still allowable to include the tie via DEPEND_1.)
To see the LABL_PTR_1 values referenced below, see the label_B_GSE variable definition below.
! Variable Data Number Record Dimension ! Name Type Elements Dims Sizes Variance Variances ! -------- ---- -------- ---- ----- -------- --------- "BGSE" CDF_REAL4 1 1 3 T T
! Attribute Data ! Name Type Value ! -------- ---- ----- "FIELDNAM" CDF_CHAR { "B field in GSE coordinates (1 min)" } "VALIDMIN" CDF_REAL4 { -65534.0 } "VALIDMAX" CDF_REAL4 { 65534.0 } "UNITS" CDF_CHAR { "nT" } "FORMAT" CDF_CHAR { "E13.6" } "SCALETYP" CDF_CHAR { "linear" } "CATDESC" CDF_CHAR { "Magnetic field vector in GSE " - "cartesian coordinates (1 min)" } "FILLVAL" CDF_REAL4 { -1.0e+31 } "LABL_PTR_1" CDF_CHAR { "label_B_GSE" } "DEPEND_0" CDF_CHAR { "Epoch" } "VAR_TYPE" CDF_CHAR { "data" }. "DISPLAY_TYPE" CDF_CHAR { "time_series" }.
Example of a 1D flux data variable. We show here the variable, Ion Differential Intensity, as it would appear in a CDF Skeleton table. We include all required variable attributes. Some recommended variable attributes are also shown. See the Display of this variable.
! Variable Data Number Record Dimension ! Name Type Elements Dims Sizes Variance Variances ! -------- ---- -------- ---- ----- -------- --------- "IDiffI_I" CDF_REAL4 1 1 12 T T
! Attribute Data ! Name Type Value ! -------- ---- ----- "FIELDNAM" CDF_CHAR { "Spin-avg Ion Diff Inten (EPIC/ICS)" } "CATDESC" CDF_CHAR { "Ion Diff. Intensity, at 12 energies " - "67-1361 keV (EPIC/ICS)" } "VALIDMIN" CDF_REAL4 { 1.000000e-04 } "VALIDMAX" CDF_REAL4 { 1.000000e+10 } "SCALETYP" CDF_CHAR { "log" } "UNITS" CDF_CHAR { "1/(cm^2 s sr keV)" } "LABLAXIS" CDF_CHAR { "dJ/dE" } "FORMAT" CDF_CHAR { "E9.3" } "DEPEND_0" CDF_CHAR { "Epoch" } "DEPEND_1" CDF_CHAR { "IDiffI_I_Energy" } "DELTA_PLUS_VAR" CDF_CHAR { "IDiffI_I_Uncert" } "DELTA_MINUS_VAR" CDF_CHAR { "IDiffI_I_Uncert" } "DICT_KEY" CDF_CHAR { "particle_flux>ion_differential" } "VAR_TYPE" CDF_CHAR { "data" } "FILLVAL" CDF_REAL4 { -1.000000e+31 } "DISPLAY_TYPE" CDF_CHAR { "spectrogram" } .
Example of a 2D sizes (28,12) data variable. We show here the variable, H+ number flux, as it would appear in a CDF Skeleton table. We include all required variable attributes. Some recommended attributes are also shown. See the Display of this variable.
! Variable Data Number Record Dimension ! Name Type Elements Dims Sizes Variance Variances ! -------- ---- -------- ---- ----- -------- --------- "Flux_H" CDF_REAL4 1 2 28 12 T T T
! Attribute Data ! Name Type Value ! -------- ---- ----- "FIELDNAM" CDF_CHAR { "H+ number flux" } "VALIDMIN" CDF_REAL4 { 0.0 } "VALIDMAX" CDF_REAL4 { 1.0e+08 } "UNITS" CDF_CHAR { "1/(cm^2 s keV/e sr)" } "FORMAT" CDF_CHAR { "e12.4" } "LABL_PTR_1" CDF_CHAR { "H_energy_of_flux" } "LABL_PTR_2" CDF_CHAR { "H_angle_of_flux" } "MONOTON" CDF_CHAR { "FALSE " } "FILLVAL" CDF_REAL4 { -1.0e+31 } "CATDESC" CDF_CHAR { "H+ number flux for for 28 energy and " - "3 selected angle bins." } "VAR_TYPE" CDF_CHAR { "data" } "DICT_KEY" CDF_CHAR { "particle_flux>number_species_proton" } "DEPEND_0" CDF_CHAR { "Epoch_H" } "DEPEND_1" CDF_CHAR { "energy" } "DEPEND_2" CDF_CHAR { "angle" } "AVG_TYPE" CDF_CHAR { "standard" } "DISPLAY_TYPE" CDF_CHAR { "spectrogram>y=energy,z=Flux_H(*,1),z=F" - "lux_H(*,7),z=Flux_H(*,12)" } "SCALETYP" CDF_CHAR { "log" } "VAR_NOTES" CDF_CHAR { "Negative values reflect low counting " - "rates and background subtraction. " } .
These are variables of secondary importance (e.g., time, energy_bands associated with particle_flux).
Support_data is always either Real or Integer type. Support_data is usually time invariant, but can be time varying.
If a support_data variable is attached to a data variable via DEPEND_i, then it must be of the same size as the dimension i. See example below. Real or Integer data are always defined as having one element.
The following variable attributes are required:
We show here the variable, Epoch as it would appear in a CDF Skeleton table. We include all required variable attributes. Some recommended attributes are also shown. Epoch is time varying and is attached to all time varying data variables via DEPEND_0. It is used for the x-axis in all displays below.
! Variable Data Number Record Dimension ! Name Type Elements Dims Sizes Variance Variances ! -------- ---- -------- ---- ----- -------- --------- "Epoch" CDF_EPOCH 1 0 T
! Name Type ! Attribute Data Value ! -------- ---- ----- "FIELDNAM" CDF_CHAR { "Time since 0 AD" } "VALIDMIN" CDF_EPOCH { 01-Jan-1994 00:00:00.000 } "VALIDMAX" CDF_EPOCH { 01-Jan-2020 00:00:00.000 } "LABLAXIS" CDF_CHAR { "Epoch" } "UNITS" CDF_CHAR { "ms" } "FILLVAL" CDF_REAL8 { -1.0e+31 } "VAR_TYPE" CDF_CHAR { "support_data" } "DICT_KEY" CDF_CHAR { "time>Epoch" } "SCALETYP" CDF_CHAR { "linear" } "MONOTON" CDF_CHAR { "INCREASE" } "CATDESC" CDF_CHAR { "Interval centered time tag rounded to " - "nearest msecond " }. ! RV values were not requested.
Example of a simple 1D size 12 time varying support_data variable - energy. We show here the variable, Ion Energy, as it would appear in a CDF Skeleton table. We include all required variable attributes. Some recommended attributes are also shown. This support_data variable is attached to a data variable (Ion Diff. Intensity, at 12 energies 67-1361 keV) of the same dimensionality and size. See this variable used in a Display.
! Variable Data Number Record Dimension ! Name Type Elements Dims Sizes Variance Variances ! -------- ---- -------- ---- ----- -------- --------- "IDiffI_I_Energy" CDF_REAL4 1 1 12 T T
! Attribute Data ! Name Type Value ! -------- ---- ----- "CATDESC" CDF_CHAR { "Ion Energy, at 12 channels energies " - "67-1361 keV (EPIC/ICS) " } "DELTA_PLUS_VAR" CDF_CHAR { "IDiffI_I_Eplus" } "DELTA_MINUS_VAR" CDF_CHAR { "IDiffI_I_Eminus" } "DEPEND_0" CDF_CHAR { "Epoch" } "DICT_KEY" CDF_CHAR { "energy>ion" } "FIELDNAM" CDF_CHAR { "Ion Energy (EPIC/ICS)" } "FILLVAL" CDF_REAL4 { -1.000000e+31 } "FORMAT" CDF_CHAR { "F7.1" } "LABLAXIS" CDF_CHAR { "Ion Energy" } "UNITS" CDF_CHAR { "keV" } "VALIDMIN" CDF_REAL4 { 67.3 } "VALIDMAX" CDF_REAL4 { 1361.0 } "VAR_TYPE" CDF_CHAR { "support_data" }.
These are variables of secondary importance e.g., a variable holding “Bx,By,Bz” to label magnetic field).
The following CDF variable specifications are required.
Metadata is always character type. Metadata is always time invariant if it is used to label a data variable. Metadata can be time varying if it is NOT used as a label.
If a metadata variable is attached to a data variable via LABL_PTR_i, then it must be of the same size as the dimension i. See example below.
Character metadata must define the number of elements to be the same as the number of characters used in its value - 6 in the example below.
The following variable attributes are required:
Example of a 1D size 3 metadata variable. This metadata variable labels the cartesian GSE magnetic field (1D size
! Variable Data Number Record Dimension ! Name Type Elements Dims Sizes Variance Variances ! -------- ---- -------- ---- ----- -------- --------- "label_B_GSE" CDF_CHAR 6 1 3 F T
! Attribute Data ! Name Type Value ! -------- ---- ----- "CATDESC" CDF_CHAR { "Label cartesian B" } "FIELDNAM" CDF_CHAR { "Label cartesian B" } "VAR_TYPE" CDF_CHAR { "metadata" }. "FORMAT" CDF_CHAR { "a6" } ! NRV values follow... [ 1 ] = { "Bx GSE" } [ 2 ] = { "By GSE" } [ 3 ] = { "Bz GSE" }
See also the metadata variable “label_B_GSE”.
See also the Support data variable “IDiffI_I_Energy”.
The 3 selected angle bins appear as separate panels. The energy attribute values (either the LABLAXIS value or the FIELDNAM value, along with the UNITS value) are used to label the y-axis on each panel. The energy support_data variable is attached to the data variable via the DEPEND_1 attribute.
The z-axis (color bar) is labeled with selected values from the H+ angle label variable that is attached to the data variable via the LABL_PTR_2 attribute. The z-axis units come from the data variable UNITS attributes.
The 3 selected energy bins appear as separate panels. The angle attribute values (either the LABLAXIS value or the FIELDNAM value, along with the UNITS value) are used to label the y-axis on each panel. The angle support_data variable is attached to the data variable via the DEPEND_2 attribute.
The z-axis (color bar) is labeled with selected values from the H+ energy label variable that is attached to the data variable via the LABL_PTR_1 attribute. The z-axis units come from the data variable UNITS attributes.
For IACG use (expanded international community with missions outside the core ISTP) and for ISTP higher resolution definitive data or for event data, only Epoch is now required. A quality flag is still recommended.
“Epoch” should be the first variable in each CDF data set. All time varying variables in the CDF data set will depend on the “Epoch” variable (or on a CDF_TIME_TT2000 (or CDF_EPOCH or CDF_EPOCH16) type variable) - more than one CDF_TIME_TT2000/EPOCH/EPOCH16 type variable is allowed in a data set to allow for more than one time resolution. For ISTP the time value of a record refers to the center of the accumulation period for the record if the measurement is not an instantaneous one.
Epoch allows for a scalar representation of time which provides for seamless crossings of day and year boundaries. Epoch time is simply the time in milliseconds A.D. CDF toolkit programs will display and expect CDF_EPOCH values in the format dd-mmm-yyyy hh:mm:ss.ccc where dd is the day of the month, mmm is the month, yyyy is the year, hh is the hour, mm is the minute, ss is the second and ccc is the millisecond (e.g., 01-Aug-1992 10:30:05.025). “Epoch” will be monotonically increasing so that the attribute MONOTON should be defined as “INCREASE”.
(Note: All CDF data sets using the Epoch variable should use the subroutines provided in the CDF toolkit for making the conversion between this value and year, month, day, etc. These routines are available as black boxes from SPDF. To determine Epoch time it is only necessary to call the subroutine compute_Epoch(year, month, day, hour, minute, second, msec, Epoch) with arguments as shown.for making the conversion between this value and year, month, day, etc. This ensures that all users use the same conversion when generating their CDF data sets and will therefore have the same view of the effects of the various calendar changes that have occurred over the last two thousand years.)
An Example of “Epoch” is shown below.
! Variable Data Number Record Dimension ! Name Type Elements Dims Sizes Variance Variances ! -------- ---- -------- ---- ----- -------- --------- "Epoch" CDF_TIME_TT2000 1 0 T
! VAR_COMPRESSION: None ! (Valid compression: None, GZIP.1-9, RLE.0, HUFF.0, AHUFF.0) ! VAR_SPARSERECORDS: None ! (Valid sparserecords: None, sRecords.PAD, sRecords.PREV) ! VAR_PADVALUE: 0000-01-01T00:00:00.000000000 ! Attribute Data ! Name Type Value ! -------- ---- ----- "CATDESC" CDF_CHAR { "Interval centered time tag (TBC)" } "FIELDNAM" CDF_CHAR { "Time since Jan 1, 1958" } "FILLVAL" CDF_TIME_TT2000 { 9999-12-31T23:59:59.999999999 } "LABLAXIS" CDF_CHAR { "Epoch" } "UNITS" CDF_CHAR { "ns" } "VALIDMIN" CDF_TIME_TT2000 { 2010-01-01T00:00:00.000000000 } "VALIDMAX" CDF_TIME_TT2000 { 2029-12-31T23:59:58.999000000 } "VAR_TYPE" CDF_CHAR { "support_data" } "SCALETYP" CDF_CHAR { "linear" } "MONOTON" CDF_CHAR { "INCREASE" } "DICT_KEY" CDF_CHAR { "time>Epoch" } "TIME_BASE" CDF_CHAR { "J2000" } "TIME_SCALE" CDF_CHAR { "Terrestrial Time" }. ! RV values were not requested.
Each ISTP CDF data set should contain at least one quality or status flag which is record varying. The CDF data set designer may choose to have more than one if the data warrants this.
Time_PB5 is the second variable in an ISTP KP CDF data set. It is not required for IACG or ISTP higher resolution or event data. Time_PB5 is another way of presenting time which allows for easy recognition of the time value when looking at the data, for instance in a data dump. For ISTP the time value of a record refers to the center of the accumulation period for the record if the measurement is not an instantaneous one.
Time_PB5 is given in YEAR (4 digit), DAY OF YEAR (note: January 1 is Day 1), and MSEC OF DAY (elapsed ms). These are all signed integer*4 numbers and are stored as the three elements of the one-dimensional variable named “Time_PB5”, e.g. 1997, 237, 0 would be August 25, 1997 at midnight.
“Time_PB5” has three “attached” variables which provide labels units and formats for the three components. These are called “label_time”, “unit_time”, and “format_time”, respectively, and are shown in the example below.
! Variable Data Number Record Dimension ! Name Type Elements Dims Sizes Variance Variances ! -------- ---- -------- ---- ----- -------- --------- "time_pb5" CDF_INT4 1 1 3 T T
! VAR_COMPRESSION: None ! (Valid compression: None, GZIP.1-9, RLE.0, HUFF.0, AHUFF.0) ! VAR_SPARSERECORDS: None ! (Valid sparserecords: None, sRecords.PAD, sRecords.PREV) ! VAR_PADVALUE: -2147483648 ! Attribute Data ! Name Type Value ! -------- ---- ----- "CATDESC" CDF_CHAR { "Time of observation in Year, Day, & " - "milliseconds" } "DEPEND_0" CDF_CHAR { "epoch_1hr" } "DICT_KEY" CDF_CHAR { "time>pb5" } "FIELDNAM" CDF_CHAR { "Time PB5" } "FILLVAL" CDF_INT4 { -2147483648 } "LABL_PTR_1" CDF_CHAR { "label_time" } "MONOTON" CDF_CHAR { "INCREASE" } "UNIT_PTR" CDF_CHAR { "unit_time" } "VALIDMIN" CDF_INT4 { 1997, 237, 0 } "VALIDMAX" CDF_INT4 { 2030, 365, 0 } "VAR_TYPE" CDF_CHAR { "support_data" } "SCALETYP" CDF_CHAR { "linear" } "FORM_PTR" CDF_CHAR { "format_time" } . ! RV values were not requested. ! Variable Data Number Record Dimension ! Name Type Elements Dims Sizes Variance Variances ! -------- ---- -------- ---- ----- -------- --------- "unit_time" CDF_CHAR 4 1 3 F T
! VAR_COMPRESSION: None ! (Valid compression: None, GZIP.1-9, RLE.0, HUFF.0, AHUFF.0) ! VAR_SPARSERECORDS: None ! (Valid sparserecords: None, sRecords.PAD, sRecords.PREV) ! VAR_PADVALUE: " \341X\207" ! Attribute Data ! Name Type Value ! -------- ---- ----- "CATDESC" CDF_CHAR { "Units for Time_PB5" } "DICT_KEY" CDF_CHAR { "label" } "FIELDNAM" CDF_CHAR { "Units for Time_PB5" } "FORMAT" CDF_CHAR { "a5" } "VAR_TYPE" CDF_CHAR { "metadata" } . ! NRV values follow... [1] = { "year" } [2] = { "day " } [3] = { "msec" } ! Variable Data Number Record Dimension ! Name Type Elements Dims Sizes Variance Variances ! -------- ---- -------- ---- ----- -------- --------- "label_time" CDF_CHAR 27 1 3 F T
! VAR_COMPRESSION: None ! (Valid compression: None, GZIP.1-9, RLE.0, HUFF.0, AHUFF.0) ! VAR_SPARSERECORDS: None ! (Valid sparserecords: None, sRecords.PAD, sRecords.PREV) ! VAR_PADVALUE: " \341X\2070" ! Attribute Data ! Name Type Value ! -------- ---- ----- "CATDESC" CDF_CHAR { "Label for Time_PB5" } "DICT_KEY" CDF_CHAR { "label" } "FIELDNAM" CDF_CHAR { "Label for Time_PB5" } "FORMAT" CDF_CHAR { "a28" } "VAR_TYPE" CDF_CHAR { "metadata" } . ! NRV values follow... [1] = { "Year " } [2] = { "Day of Year (Jan 1 = Day 1)" } [3] = { "Elapsed milliseconds of day" } ! Variable Data Number Record Dimension ! Name Type Elements Dims Sizes Variance Variances ! -------- ---- -------- ---- ----- -------- --------- "format_time" CDF_CHAR 2 1 3 F T ! VAR_COMPRESSION: None ! (Valid compression: None, GZIP.1-9, RLE.0, HUFF.0, AHUFF.0) ! VAR_SPARSERECORDS: None ! (Valid sparserecords: None, sRecords.PAD, sRecords.PREV) ! VAR_PADVALUE: " \341" ! Attribute Data ! Name Type Value ! -------- ---- ----- "CATDESC" CDF_CHAR { "Format for Time_PB5" } "DICT_KEY" CDF_CHAR { "label" } "FIELDNAM" CDF_CHAR { "Format for Time_PB5" } "FORMAT" CDF_CHAR { "a3" } "VAR_TYPE" CDF_CHAR { "metadata" } . ! NRV values follow... [1] = { "I4" } [2] = { "I3" } [3] = { "I8" }
At the May 1992 ISTP SWG it was decided that a `record varying “Post Gap” Quality Flag would also be included in each KP record.
(Note: This I*4 Flag is included in every record and is defined in the following way: 0 - no gap occurred immediately prior to this record [thus most of the time this Flag would be set to 0]; 1 - the gap occurred because the instrument was not in a mode that allowed for the production of KPs; 2 - the gap occurred because Level Zero or SIRIUS data were missing; 3 - the gap occurred because Level Zero or SIRIUS data were too noisy to compute KPs. Integer numbers above 9 can be used by the PI team to define other gap conditions, as required.)
A variable similar to this is recommended for inclusion in CDFs to indicate real and substantial data gaps. The detailed definition of this flag should appear in the CDF metadata in the VAR_NOTES attribute.