next up previous contents
Next: Spacecraft moments: introduction Up: qtmc_manual Previous: Contents   Contents

Subsections

QTMC - User guide

Licenses

QTMC falls under the GNU General Public License (GPL), though utilises libraries developed by the South West Research Institute (SwRI), which requires a separate license if the software is built from source. These are contained in the file LICENSES, distributed with this software.

Installation

QTMC relies on several other pieces of software and libraries to run:

  1. South West Data Display and Analysis System (SDDAS) - Instrument Data File Set (IDFS). These libraries handle the promotion and ingestion of spacecraft data - note they do not fall under the GNU General Public License, and if you build the software, then you must sign the associated license.
    http://www.idfs.org
  2. Common Data Format (CDF) is a format which stores data and metadata, and is extremely useful for the storage and manipulation of the corrected moment data. QTMC requires some of the CDF header files.
    http://nssdc.gsfc.nasa.gov/cdf/cdf_home.html
  3. CEF/QFT/CDF translator Qtran. This is not essential, since both the CEF and QFT formats are supported by QTMC, however without Qtran, no CDF files can be generated. Note that only the executable Qtran is needed, and QTMC can be built without the Qtran source code.
    http://www.space-plasma.qmul.ac.uk/QSAS/qtran_welcome.html

  4. If you wish to run the software in interactive mode, you will require the Qt libraries. These are distributed with flavours of Linux which support KDE.
    http://www.trolltech.com/products/qt/

QTMC can run in batch-mode (i.e. invoked with full options from the command line), or in interactive mode (which relies on the Qt libraries). The software can be built either with or without Qt by using the appropriate Makefile. If QTMC is built without Qt, then only the batch-mode can be used.

Additionally, there are some underlying libraries distributed with QTMC which are borrowed from another software suite, QSAS. You can uses those binaries, or rebuid them here: libqdos.so, libqblas.so and libqlapack.so, which reside in the ./lib directory. These are needed by QTMC for the way it handles the data objects and certain underlying mathematical functions. To clean out all binaries and begin, build these libraries by executing the lines:

build Makefile_no(w)Qt clean
build Makefile_no(w)Qt qdos
build Makefile_no(w)Qt qblas
build Makefile_no(w)Qt qlapack
which should build the .so or .a files and put them in ./lib.

Then, for the non-Qt version, execute the line:

build Makefile_noQt

This will put the binary executable QTMC_noQt in the ./bin directory.
For the Qt version, execute the line:

build Makefile_wQt

This will put the binary executable QTMC in the ./bin directory. So far QTMC has been successfully built on the Mandrake 9.x flavour of Linux.

The QTMC binary can be run directly if you add the necessary paths and environments to your shell, e.g., edit your (e.g.) .tcshrc file to include:

setenv QTMC_HOME /local/path/qtmc

and add the QTMC .lib directory to the LD_LIBRARY_PATH:

setenv LD_LIBRARY_PATH ${LD_LIBRARY_PATH}:${QTMC_HOME}/lib.

You'll also need to add the CDF libraries in the same way. Next, edit the build script to point to local locations of various dependencies (described above).

Alternatively, a script to do all this, qtmc_ui, is supplied in the QTMC bin directory. This script will need to be edited to point to your local installation. Other scripts for command-line operation are also provided, and can be modified to set the necessary environment variables.

Error codes

The main solver routine returns a code referring to the success or failure of the algorithm. A correcting failure is defined as a time when the algorithm failed to converge to a result within the maximum number of iterations allowed (50), or an error related to bad input data: the nature of this failure is described by an integer code. QTMC filters out all failures before outputting the final corrected data sequence, however the successful points are categorized by a `success' code - 0, 6 or 9. They can be described as follows:

0 successful solve
6 successful solve, but needed to invoke guessing loop
9 successful solve, but did not include BOTTOM moments

Code 9 refers to moment sums where the user has specified that the BOTTOM moments (PEACE nomenclature will be discussed later) should be used, but the solver has not used them for fear of contamination by photo-electrons. This feature is described fully below in section 1.7. The failed data points are invisible to the user, as the output series will never contain any failed points (though a record will exist if the user has output a log file). The most common form of failure is iteration exhaustion (code 7), though occasionally other errors will cause the solver to fail (e.g. incorrect inputs, corrupt data, etc.) - each code has an associated message which describes what went wrong.

The output file contains the successful codes as a time-series, which can be compared directly with the corrected moments to show regions where, for example, the B moments were not included, or points where the guess loop was needed.

Batch mode

QTMC can be executed in batch mode from the command line by providing the following arguments. Bold face type indicate mandatory inputs. Values in `[ ]' represent defaults.

-S{start time YYYY-MM-DDZHH:MM:SS.SSS or YYYY-DDDZHH:MM:SS.SSS}
-E{end time YYYY-MM-DDZHH:MM:SS.SSS or YYYY-DDDZHH:MM:SS.SSS}
-M{SC ([1]-4)}{SUM [BL1L2T]}
-C{location of calibration files}
-D{location of output data}
-O{output format [cef]/qft/cdf}
-U{delta Phi [0.0 V]}
-V{manual s/c potential [7.0 V]}
-o{over-ride EFW U_probe_sc yes/[no]}
-l{lower limit of guess loop [-200%]}
-u{upper limit of guess loop [200%]}
-r{resolution of guess loop [1%]}
-reticent {turn off logging}
-clobber {overwrite existing files}
-history {by-pass calculation and print basic command history}


The value of `manual spacecraft potential' is used if the EFW (Electric Fields and Waves) probe value is unavailable, or if the user requests that the EFW probe value should not be used (over-ride EFW option).

A minimal example of a literal QTMC execution could be (arguments have been put on separate lines for clarity):

QTMC -S2002-075Z14:00:00.000
-E2002-075Z15:00:00.000
-M2BH1H2T
-C$(QTMC_HOME)/cal
-D/data/cluster/sc2

This command corrects the BH1H2T moment sum from spacecraft 2 over the interval 1400 hrs to 1500 hrs on March 16, 2002. The data is put in some local repository, and the calibration files are retrieved from the QTMC distribution ./cal directory (QTMC_HOME should be set as an environment variable). These calibration files were accurate at the time of the last distribution (though could be updated in future). All other options have their default values. A more complicated example could be:

QTMC -S2002-075Z14:00:00.000
-E2002-03-16Z15:00:00.000
-M2BH1H2T
-C$(QTMC_HOME)/cal
-D/data/cluster/sc2
-U1.5
-l-1000
-u1000
-r100
-Ocdf
-clobber

This command accesses the same interval as above (but note the time format for -S and -E is different - this mixing is allowed). Here some more options are used: $\Delta\Phi$ is set to 1.5 Volts and the guessing loop has been extended. Finally, files are output in the CDF format and any existing files of the same name are overwritten (with -clobber). Filenames follow a strict convention:
TMC_SCN_YYYYDDDHHMM_YYYYDDDHHMM_SUM.cef(cdf,qft)
Where `TMC' indicates that these are corrected truncated moments; `SCN' refers to the spacecraft number (N ranges from 1 to 4); the first `YYYY...' block indicates the start of the data interval; and the second block the end of the interval. Finally `SUM' could be any one of L1L2, H1H2, BL1L2, BH1H2, L1L2T, H1H2T, BL1L2T or BH1H2T - depending on the requested sum. The extension follows from the user's output format request. If clobber is off, then files of the same name will have an `A' appended to the filename:
TMC_SCN_YYYYDDDHHMM_YYYYDDDHHMM_SUMA.cef(cdf,qft)
Subsequent files will have another `A' and so on.

If the -history is requested, then promotion of data will still occur, but the correction will be by-passed and basic PEACE operational data will be printed to the console. The energy-scan of both LEEA and HEEA are output (in telemetered units) and the user is informed if there are B, T or overlap moments present. This information is generated for the start and end of the interval requested, and for any scan changes within that interval. For example, the following command:
QTMC -S2002-01-03Z04:00:00.000 -E2002-01-03Z06:00:00.000 -M1 -history
gives the following output:

2002-01-03 03:59:58.18001067 - 1667 - 16
2002-01-03 05:59:56.73501067 - 1667 - 16

The columns are: time, B moments present? (Boolean), overlap (L1L2 or H1H2) moments present?, T moments present?, HEEA scan (upper - lower), LEEA scan (upper - lower) (both in telemetered units, or bin numbers). Note that the arguments contain a minimal subset of options described previously, without some of the mandatory options. This is somewhat of a hack, since for the command history only a data interval and spacecraft number is required: no real calculation is performed, and no output file is generated. Even the sum request (e.g. -M1BL1L2T) can be omitted, since it is inferred from the energy scan. It doesn't hurt to include the other options though. If PEACE data cannot be found, then a message will be printed to console.

Perhaps the most powerful application of QTMC is via a custom written script which can loop over hours, days, weeks or months worth of data. Such jobs can be left to run unaided by the user to provide an almost automated correcting facility. QTMC comes with both a simple Perl and Csh script to whet the user's appetite. They can be found in the ./bin directory.

Interactive mode

If only the switches -C and -D are provided, then QTMC will execute in GUI mode, providing a simple graphical interface which allows the user to control all the settings and options described above. Figure 1.1 is an image of the interface.
Figure 1.1: QTMC user interface.

\includegraphics[scale=0.5]{qtmc_screen_shot}

The log file

If the user has not specified that QTMC is executed in reticent mode, then a log will be written containing all the essential data describing each point which was attempted to correct. The log file has the same name structure as the data file and has the extension .log. It contains the following information:

The log file is very useful if correcting moments for an `unknown' period, as it contains data which is vital to the calculation, but not output in the final file. Even if logging is off, a small summary containing useful information is printed to the console at the end of the calculation.

Moment sums & $\Delta\Phi$

We can construct any of the following sums: L1L2, H1H2, BL1L2, BH1H2, L1L2T, H1H2T, BL1L2T and BH1H2T. Note we cannot mix L1L2 and H1H2 to form for example L1H2. If PEACE is operating in single sensor mode (i.e. all the moments are in TOP or BOTTOM), then the software will detect it and proceed as required (and report to the log file). It is possible to request for example the sum BL1L2T, when in fact there are no B or T moments (i.e. sensors are in complete overlap). Again, the software will check for this and behave accordingly - information about the sum is recorded in the log file (note - the filename structure will retain the requested sum).

We have devised a method to filter out the effects of contamination from photo-electrons where we make a choice not to include the B moments (if present) if the lower energy cut-off is within a certain threshold limit of the spacecraft potential (see Figure 1.2). This has the effect of not including the low energy (e.g. $<10$ eV) electrons at the expense of raising $E_L$ and eliminating a wider portion of phase-space. If the solver makes this decision, then an error code will be returned indicating that the B moments were not included for that record. By plotting or inspecting the sequence of error codes, the user can investigate regions of data where filtering has taken place. The threshold limit we call $\Delta\Phi$, and is a user-controllable parameter. Raising $\Delta\Phi$ will result in more instances where B moments are discarded, and lowering it will result in fewer instances where B moments are discarded. Its default setting is zero Volts; that is, excluding B moments for times when the spacecraft potential exceeds the lower energy cut-off.

Figure 1.2: Diagram demonstrating B moment filtering. The spacecraft potential is varying with time (thick line). The lower energy level can be considered constant (in reality it doesn't have to be). A threshold is set ($\Delta\Phi$) such that if the potential of the spacecraft breaches it, then the B moments are not used. It can be seen that increasing ($\Delta\Phi$) will result in more stringent filtering of the B moments.

Image delta_phi

Troubleshooting

Symptom Cause/Solution
Installation difficulties Check you have all the required libraries, etc.
  and that you have edited the build script
  to point to your local versions. If you don't have
  Qt, then build with the no-Qt option.
  If you have not done so already, try to rebuild the
  libqlapack, libqblas and libqdos
  sharable libraries.
   
Cannot get data sequence No data within time requested, check command log
  to see what PEACE's operating mode was. Note, if there
  is no EFW data, a manual value of potential can be used;
  also if there is no FGM data, the anisotropy will not
  be estimated.
   
Error separating Errant time-tag within series. Although QTMC
overlap moments filters out misplaced time-tags, it is possible that the
  occasional one will slip through. Try adjusting the data
  interval slightly.
   
Main calculation failure Exception encountered while in the main loop.
   
Input error Invalid input field. Common problems are: mixing of L1L2
  and H1H2 moments, and Invalid time entries. Note that the
  end time must be at least 10 seconds ahead of the start
  time. Times must be entered in the correct ISO format (see
  above).
   
Promotion error Data is accessed via an external server. If the server is
  down/inaccessable, then data will not be retrieved, and
  the user will be warned. Alternatively if there is insufficient
  space left on the disk, promotion will fail.
   
Join error See `Error separating overlap moments'
   
Nil sequence For whatever reason, a data sequence with no entries has
  been created. This will inevitably cause a problem somewhere
  in the calculation, so an error is returned when this
  situation arises.
   
Failed to create Insufficient memory to accomodate output series.
output sequence  
   
File read/write error Check you have the necessary permissions to read the
  calibration files and write data to the disk.
   
No anisotropy? If the FGM data is not available, then no anisotropy
  estimation will be made. The other outputs are unaffected.
  A message indicating that no anisotropy will be output is
  printed to the console.
   
Potential not what Again, if the EFW data is unavailable, then QTMC will use
I was expecting! a default value (which you can control, but initially set to 7 V).
  This information is again conveyed by the console.

Practical details

Running QTMC blind

Often, the user will know that there is on-board coverage of a certain interval before attempting to correct the moments. Conversely, it is possible to run the software `blind' - i.e. to attempt to correct an interval about which you have no prior knowledge of coverage. If there are any moments within the interval requested (and there is associated data such as AUX (attitude) data), then those moments will be corrected. The start and end times of these moments may represent only a short portion of the input start and end times (and this will be conveyed in the summary at the end of the calculation), however the filename will contain the input times as part of its construction. This could be slightly misleading to the unwary, so it is advisable to consult the command history of PEACE (or use the -history option). Note that if the EFW and/or FGM data is not found, then the correction will still proceed, but for the potential a constant user-defined value (default 7 V) will be used and/or no anisotropy will be estimated. Again, this is conveyed to the user on the console.

Scripting & optimization

Instead of running the corrector blind, you may wish to try to access and correct all the moments over a long duration (you would construct a script to accomplish this). Now the optimal data interval length becomes important, since the user must consider the managability of file sizes, and the possibility of reducing losses of data if the calculation fails during a particularly long run. Probably the maximum/optimal data interval (from existing user experience) is about 4 hours. This way, output files are easily managable, and each correction pass is relatively fast. Another way to speed up calculation on long batch operations is to make the guess loop step resolution quite coarse, and, erring on the side of caution, make the loop quite large (say -l-1000, -u1000, -r100).

Batch mode scripts can be written in your preferred language, for example Csh or Perl. The simple looping and arithmetic capabilties of scripting languages allow the user to construct powerful automated correcting facilities. Note that QTMC prints information to the console for (a) promotion of data, (b) progress of the calculation and (c) summary of calculation. In batch mode you should redirect this output to a file (say called console.log) and watch its progress with tail. For example, to execute the script `qtmcbatch':

@> chmod +x qtmcbatchMake the script executable (could also use chmod 755)
@> qtmcbatch > console.log &Run script in background and redirect output to log file
@> tail -f console.logWatch progress of console.log. Kill with Ctrl+C


next up previous contents
Next: Spacecraft moments: introduction Up: qtmc_manual Previous: Contents   Contents
Steve Schwartz 2005-03-26