Cluster II IDL Pick Library for EDI

Mark Chutter
August 11, 1999

Oct 19, 1999 The Pick structures are kept in common blocks rather than getting
             defined in batch routines.  This make things much faster by 
             greatly reducing IDL overhead.  This change should be transparent
             to users.

Mar 15, 2000 Sun Reference Pulse and Spin Period information have been added.
             A number of changes/fixes have been implemented thanks to getting
             an IDL EQS version of the Pick Library working with EDIQ.

Apr 21, 2000 MSF and RDM file types are now treated differently.

Apr 28, 2000 The location of the SRP in tm is not indicative of the order
             in which they occur.  Now, srp will have the first SRP in
             time and srp1 will be the second.

May  5, 2000 ToF has been fixed in WW extraction; Tspin is a long value
             which contains the number of microseconds.

May 10, 2000 The time tag at TMR in WW modes was wrong

May 20, 2000 The frame bits had been included in the index for NM1.  Now that
             I better understand the index and frame bits, that has been 
             corrected.

Jun  6, 2000 Added magnetic field to structures.  To use magnetic field, set
             an environment variable called CL_MAG to anything for now.

Jun 12, 2000 Added Memory Dump option for NM1 and BM1

Jul 12, 2000 Fixed the CTM file format reading

Jul 19, 2000 Made correlator changes for WW: replace max counts with word14
             (low 12 bits for SMAX +/-) and min counts with word15 (16 bits
             for ALL).  The Geos part of the Pick Library is not affected
             because names were never assigned to correlator data in NM1, PM2

Aug 11, 2000 moved some "help" procedures that are not really a part of the
             Pick Library into ../test_pro and added some new ones. Added the
             FTP data type which are for files retrieved via ftp from the
             CDDS (Cluster Data Disposition System)

Sep 13, 2000 added get_calib_fgm; added more error checking

Jan  5, 2001 so many changes; the data_stream is now included; the raw 
             secondary is included

Aug 20, 2001 added VERBOSE env variable, bad data checking for beam 
             current indicies in m5_fill, option of getting HK data
             with science data from MSFs via the get_hk_data keyword.

Sep 20, 2001 WW qualities (sq1 and sq2) have been changed from byte to int type

Feb  6, 2002 added ambient GEOS which require a change in the common blocks

Requirements:
  This EDI Pick Library should run on any platform where IDL runs.
  
Description:
  The EDI Pick Library is an IDL tool that is used to extract and unpack EDI
  Science and Housekeeping data from the different telemetry sources.
  For a more complete description, please refer to the Cluster II
  EDI Pick Library manual, cl_pick_man.html.

Time:
  ctime: 2 long integers
  ctime[0]: seconds since Jan. 1, 1970
  ctime[1]: milliseconds within current second

  atime: 2 long integers
  atime[0]: seconds since Jan. 1, 1970
  atime[1]: microseconds within current second

I am trying to change things so that only the atime format is used which
makes my life simpler.  Your life is your problem.

There are 3 different time strings that are allowed by the Pick Library:
DD-MMM-YYYY HH:MM:SS.FFFFFF
MMM-DD-YYYY HH:MM:SS.FFFFFF
YYYY-MMM-DD HH:MM:SS.FFFFFF
The F is for decimal portion of the seconds.  There can be from 1 to
6 F digits.  The other digits should be obvious.  Kosta's Pick Library
allowed the use of 2 digits for the year in some of his allowed time formats.
These time formats will no longer work.

Status of the IDL Pick library:

  The current version does not have all the features of the old Pick Library.
  Some routines were meant for use with the RTA/QLA.  Of course, new features
  have been added.  make_subfile.pro and make_subfile_str.pro will produce a
  subset of an input file.  These were added because with Brett's software,
  an entire file must be read.  If you are only interested in a small portion
  of the file, this saves a great deal of time.  There are several useful
  "help" procedures in the test_pro dirctory which are either examples of how
  to use the Pick Library or handy utilities that use the Pick Library.
  This version has been tested with CDM (simulator dump files) and a CTM
  (Cluster Telemetry File from the EDI EGSE).  There
  has been very limited testing for FTP, RDM or MSF files because the prelaunch
  files have no EDI data, only empty packets.
  You can open data files, read science data, and close data files.  When
  reading science data, you can use PICKA_SC to read the next valid data packet
  or PICKST_SC to read the next valid packet after a specific time.  The 
  following bit of IDL code shows all that is required to get a valid packet
  of data:

; sname = 'cdm'
; fname = 'dumps/ntm11a.dmp'
;
; ...
; pick_init(sname,fname)
; sr = picka_sc(sc_pick_id,PSTATUS)
; ...
;

The Pick Library assumes that for RDM data (the data that will be on the
Cluster CDROMs), the Spacecraft HK data is in a directory
../hkd_X, where is X is the spacecraft ID, in relation to the EDI science
data.

I try to keep the mode specific information up to date with the latest changes
made by Barb.  If I fall behind, let me know.

A note about "FTP" and "RDM" formats.  "FTP" data is very similar to "RDM"
data.  The headers are different; the data is the same.  When you 
specify "RDM" as the file type, the Pick Library expects to find a 
directory structure just like on a CDROM.  The Pick Library does not expect
such a structure or to have spacecraft housekeeping data available.
