IDS data structure and formats

First version: 2004-02-19
Previous versions: 2004-07-06, 2007-09-19
Current version: 2009-02-11

Contents

0. What kind of information on what center?
1. File naming conventions
1.1 Data
1.2 Products
1.3 Documentation

2. Data centers structure
2.1 Information (mirrored CB data)
2.2 Data
2.3 Products

3. Formats
3.1 Data
3.2 Products and their documentation

4. How to deliver products to the Data centers

                         

0.    What kind of information may be found on which center?

   IDS has three data/information centers:

 - CB: the Central Bureau web and ftp sites at CLS
 - DC: the Data Center(s): 
                 * CDDIS: web and ftp sites 
                 * IGN:   ftp site
 - AC: the Analysis Coordinator webpage

   The baseline storage rules are as follows.

 . DC store observational data, products, and ancillary information required for the use of these data and products 
      + formats and analysis descriptions.
      The DCs will issue monthly bulletins (with respective dates 
      shifted by 1/2 month) giving the current status of the data and 
      product storage.
 . CB produces/stores/maintains basic information on the DORIS system,
      including various standard models (satellites, receivers, signal,
      reference frames, etc).
 . AC refers to CB and DC information on the data and modelling,
      and generates/stores analyses of the products.

Two criteria are considered for deciding where files are stored/maintained: 
   1. the responsibility for their content and update,
   2. the ease of user access.

Data-directed software is stored and maintained at the CB, analysis-directed 
sofware is stored/maintained, or made accessible through the AC site.

To avoid information inconsistencies, duplication is minimized. Logical 
links and cross referencing between the three types of information 
centers are systematically used.

1.    File naming conventions

Filenames include only numbers and lower cap characters, including station 
names, with the exception of the ".Z" extension indicating compressed files.

Data and product files are compressed using UNIX compression format.
[=> top]

1.1   Data
    
1.1.1 Preprocessed data

      sssdataMMM.LLL.Z
Where   
      sss  = the satellite three-character abbreviation (1)
      data = fixed part
      MMM  = the three-digit cycle/arc number (2)
      LLL  = version number (starting with 001 for the initial version) 
             when the file is replaced
      Z (uppercase) indicates compressed file (Unix)

Note 1: Three-character satellite name abbreviations:

          SPOT-2: sp2           TOPEX/Poseidon: top         Jason-2:         ja2
          SPOT-3: sp3           Jason-1:        ja1         Cryosat-2:       cs2
          SPOT-4: sp4           Envisat:        en1         Saral:           srl
          SPOT-5: sp5           All satellites: zzz         HY2A:            h2a

Note 2: "cycles" refer to Topex/Poseidon and Jason 1 (about 10 days)
        "arcs" refer to the Spot satellites (x to y days, [TBD])
         Envisat: 7-day arcs (the cycle length is exactly 35 days)
  
Note 3: Spot2: the first 10 data files (31.03.90-02.11.92) are organized in
        monthly files. The file names range from sp2data9003.dat.Z through 
        sp2data9211.dat.Z
  
Note 4: Envisat: the first data file (en1data001.001.Z) corresponds to the 
        first arc of repeat cycle 7. The Envisat repeat cycle number for a 
        given DORIS Data Center arc number can be found using the formula 
        cycle=int(0.2*(arc+34)).


1.1.2 Raw data (RINEX files)

      sssrawMMM.LLL.Z
Where
      sss = the satellite three-character abbreviation (ja2 for Jason2)
      rx  = fixed part (rx for rinex)
      YY  = last two digits of year of first data
      DDD = three-digit day of year of first data
      LLL  = version number (starting with 001 for the initial version)
             when the file is replaced
      Z (upper cap) points to compacted file (Unix)

Ex: ja2rx09001.002.Z

1.1.3 quaternions data files

     sssqbodyYYYYMMDDHHMMSS_yyyymmddhhmnss.LLL 
and  (if anay)    
     sssqsolpYYYYMMDDHHMMSS_yyyymmddhhmnss.LLL

where
      sss = the satellite three-character abbreviation (ja1 for jason1, ja2 for Jason2)
      qbody or qsolp  = fixed part (qbody  for body quaternions, qsolp (if any) for solar panel quaternions)
      YYYY  = date: year of first data
      MM = date: month of the first data
      DD = date: day in the month of first data
      HH = time: hour of the first data
      MN = time: mn of the first data
      SS = time: second of the fisrt data
      yyyy  = date: year of last data
      mm =  date: month of the last data
      dd = date: day in the month of last data
      hh = time: hour of the first data
      mn = time: mn of the first data
      ss = time: second of the fisrt data

      LLL  = version number (starting with 001 for the initial version)
             when the file is replaced

 Ex:  ja2qbody20090201220000_20090203020000.001
      and 
      ja2qsolp20090201220000_20090203020000.001


1.1.4 Information on data

- Current DORIS data Holding by satellite and Month for year YYYY:

   doris.yyyy

- DORIS files available by satellite: 

   doris.files


- List of observation files available for satellite sss:

    sss.files
Where   
      sss  = the satellite three-character abbreviation (1)
      files = fixed part

- Observation counts per station for data file sssdataMMM.LLL:

      sssdataMMM.LLL.sum
Where   
      sss  = the satellite three-character abbreviation (1)
      data = fixed part
      MMM  = the three-digit cycle/arc number (2)
      LLL  = version number (starting with 001 for the initial version) 
             when the file is replaced
      sum  = fixed part

[=> top]

1.2   Products
     
1.2.1 Orbits 

      cccsssVV.bXXDDD.eYYEEE.sp1.LLL.Z     + format  gsc (TBD FLemoine)
Where
      ccc = center three-character acronym (1)
      sss = satellite three-character abbreviation 
      VV  = version number of the solution
      b   = fixed part ("begin")
      XX  = last two digits of year of first position 
      DDD = three-digit day of year of first position (2)
      e   = fixed part ("end")
      YY  = last two digits of year of last position 
      EEE = three-digit day of year of last position (2)
      sp1 = fixed part, referring to the SP1 format
      LLL = version number (starting with 001) when the file is replaced
      Z (upper cap) points to compacted file (Unix)

Note 1: Three-character analysis center name 

      cls : Collecte Localisation par Satellite
      csr : Center for Space Research
      grg : CNES/GRGS
      gsc : NASA/GSFC
      gop : Geodetic Observatory Pecny
      gau : Geoscience Australia
      eso : ESA/ESOC
      nlc : University of Newcastle
      iaa : Institute of Applied Astronomy (St Petersburg)
      ign : Institut Géographique National/Jet Propulsion Laboratory
      ina : INASAN (Moscow)
      lca : LEGOS-CLS
      sod : CNES/Service d'Orbitographie DORIS
      ssa : SSALTO

Note 2: [TBD] - convention about overlap/no overlap


1.2.2 Global Sinex solutions (cf IERS naming convention)

      cccWWuVV.snx.Z
Where
      ccc = three-character center acronym 
      WW  = last two digits of year of the solution submission
      u   = D for «Doris only», C for «multitechnique» 
      VV  = version number in year YY
      snx = fixed part (refers to the format)
      Z (upper cap) points to compacted file (Unix)

Note: The global solutions are the first type of IDS geodetic products, defined 
      within the scientific community working with IERS, IVS, IGS, ILRS and/or 
      IDS. The purpose of this product is to serve as input for global combinations.
      They include two types of estimated parameters and the corresponding 
      variance-covariance information, as follows.  
      - global astro-geodetic parameters that describe station positions and
        motions, Earth orientation, motion of the station network with respect
        to the Earth's center of mass, satellite motion (case of satellite
        geodesy) and radio source directions (case of VLBI);
      - parameters for local models, such as those describing the effect of
        the troposphere on the signal transmission.
      These solutions are provided in the SINEX (Software INdependent EXchange)
      format. 


1.2.3 Time series of sinex solutions (monthly, weekly, daily, ...)

      cccYYDDDtuVV.snx.Z           
Where
      ccc = center three-character acronym
      YY  = last two digits of year of the first observation 
      DDD = three-digit day of year of the first observation.
      t   = type of the solution : m/w/d for monthly/weekly/daily solutions
      u   = d for «Doris only», c for «multitechnique»
      VV  = version number of the solution
      snx = fixed part (refers to the format)
      Z (upper cap) points to compacted file (Unix)

Note 1: These solutions are defined similarly to the solutions above, except 
        for the time span of the data used: they may be based on monthly, weekly, 
        daily, etc observation sets. The purpose of this product is to serve as input for 
        multi-technique combinations of time series. They are provided in the 
        SINEX format. 

Note 2: There is no mention of the satellite name in the filename. This 
        information may be found in the headers of each file.

Note 3: Each sinex file in the series may have a summary file cccYYDDDtuVV.sum 
        giving statistics of this data set, e.g. global, per satellite, per station, 
        and any information useful for the user of the product file. 



1.2.4 Time series of specific geodetic products

      cccWWtuV.stcd.aaaa.Z  for station coordinates
      cccWWtuV.aaaa.gif     for plots of station coordinates
      cccWWtuVV.geoc.Z      for coordinates of the TRF origin (geocenter)
      cccWWtuVV.eop.Z       for Earth orientation parameters (EOP) 
Where 
      ccc  = the center three-character acronym
      WW   = Year of submission
      t    = type of the solution : m/w/d for monthly/weekly/daily solutions
      u    = d for «Doris only», c for «multitechnique»
      VV   = version number in year YY
      Z (upper cap) points to compacted file (Unix)

      stcd = fixed part   (refers to the content)
      aaaa = station name (four characters)
      geoc = fixed part   (refers to the content)
      eop  = fixed part   (refers to the content)

Note: In principle, these times series are extracted by the Central Bureau from the
      time series of SINEX solutions (see 1.2.3). In this case, the documentation 
      file is ccctuVV.snx.dsc (see hereafter). If the Analysis center provides 
      stcd/geoc/eop time series, they must provide the corresponding .dsc 
      documentation file(s).
[=> top]
1.2.5 Ionosphere products

cccsssVV.YYDDD.iono.Z

Where
      ccc  = center three-character acronym 
      sss  = satellite three-character abbreviation
      VV   = version number of the solution
      YY   = last two digits of year
      DDD  = three-digit day of year
      iono = fixed part (refers to the content)
      Z (upper cap) points to compacted file (Unix)

[=> top]

1.3   Documentation

Each data/product file, or series of data/product files, mentioned hereafter shall 
be accompanied by a file describing the analysis and modelling used to derive it. 
The .dsc files for products would be similar to the explanation files for the 
submission of contributions to ITRF. The filenames are constructed as follows.

---------------------------------------------------------------------------------
Data/Product item              Data/Product file              Description file
---------------------------------------------------------------------------------
Data                           sssdataMMM.LLL.Z               sssdataMMM.dsc
Raw data                       sssrxYYDDD.LLL.Z               sssrxYYDDD.desc
Data summary                   sssdataMMM.LLL.sum             sssdata.LLL.sum.dsc
Orbits                         cccsssVV.bXXDDD.eYYEEE.sp1.Z   cccsssVV.sp1.dsc
Global sinex solutions         cccWWuVV.snx.Z                 cccWWuVV.snx.dsc
Time series of sinex solns     cccYYDDDtuVV.snx.Z             ccctuVV.snx.dsc
                                                              cccYYDDDtuVV.sum 
Station coord. time series     cccWWtuVV.stcd.aaaa .Z         cccWWtuVV.stcd.dsc
TRF origin time series         cccWWtuVV.geoc.Z               cccWWtuVV.geoc.dsc
EOP time series                cccWWtuVV.eop.Z                cccWWtuVV.eop.dsc
Iono files                     cccsssVV.YYDDD.iono.Z          cccsssVV.iono.dsc
---------------------------------------------------------------------------------

[=> top]
2.    Data centers structure

2.1   Information

The /cb_mirror directory results from the mirroring by the DCs of a special zone of the 
CB data center, where the DORISMail and basic information on the DORIS system is 
maintained. 


/doris
    /cb_mirror             Direct access: ftp.cls.fr/pub/ids

        /dorismail         (DORISMail archive)
        /dorisreport       (DORISReport archive)

            ...
        /satellites
            sssatt.txt     (attitude for satellite sss) 
            ssscalend.rtf  (calendar of arcs/cycles and maneuvers, satellite sss)
            ssscmass.txt   (center of mass for satellite sss)
            sssman.txt     (maneuvers for satellite sss, most recent version)
            /maneuver-archive (for earlier versions of sssman.txt)
                sssman.yymmdd
            /mass-archive (for earlier versions of sssmass.txt)
            /macromodels 
                 sssmod.pdf : macromodels for satellite sss
            /statpoe
                  statpoe_ssssss_yyyy.pdf: POE residuals using ssssss sat measurements of year YYYY
            /ephemeris
              ...    ephemeris files in sp1 format
            /archive
              ...    archive DORIS instrument documents
        ... and other satellite DORIS instrument documents ...
                ...
        /stations
            antennas.pdf       (ground antennas geometrical characteristics)
            DORIS_int_ties_yyymmdd  (DORIS internal ties of date yyymmdd)
            3rdGenerationFrequencyShifts (frequency shifts of the 3rd beacons)
            beacon_RH_models_12961_10  (beacons RF models)
            itrf2000_doris_snx  (ITRF2000 station positions at epoch 1997.0 
                                 and velocities)

           /IDS_recommandations_for_ITRF2004: (files recoms for ITRF2004,
                                          period to delete, station breaks..)
           /earthquakes   
                  ...   (earthquakes files)
           /event
                  ...   (files relative to stations events, breaks, shifts..)
          /statpoe 
             statpoe_aaaa_yyyy.gif    diplays of POE residuals using aaaa
                                       station measurements on year yyyy)
                           

            ...
     /data
          readme.txt    list of documentation
          doris10.fmt   (1.0 format. See current doris.format)
          doris21.fmt   (2.1 format)
          doris22.fmt   (2.2 format)
          RINEX_DORIS.pdf (rinex format)
         ... Other DORIS desciption files
            ...
    /products           
         dsc.readme    (describing the purpose/content of .dsc files) 
         sp1.fmt       (describing the sp1 format) 
         sp3.fmt       (describing the sp3 format)
         snx.fmt       (describing the snx format)
         stcd.fmt      (describing the stcd format)
         geoc.fmt      (describing the geoc format)
         eop.fmt       (describing the eop format)
         iono.fmt      (describing the iono format 
         /SP1description
                           contains reference files describing sp1 format
            ...
   /centers           
         ccc.acn       (describing the permanent features of center ccc)
         ...

Note: The .acn information files concern permanent features of center, while 
      the .dsc information files concern a given set of analyses.  

[=> top]

2.2    Data

/doris
    /data                    (Same format valid for all satellites, 
                                            in specified time intervals)
       /sss                  (satellite sss)
            /YYYY             (YEAR YYY, from 2008 for satellites with the newer generation DORIS receiver)
                 sssrxYYDDD.LLL.Z (raw data file per 24h, rinex format)
                 /V001   for ja2 only, archive of first version of rinex files 
                   sssrxYYDDD.001.Z (raw data file per 24h, rinex format)
                   rinex_V001_readme: readme file for ja2 first version of rinex
                   file
             sss.files        (already existing in the current structure)
            sssdataMMM.LLL.Z (data files per arc/cycle, preprocessed format)
            ...
            /sum
                sssdataMMM.LLL.sum (summary of preprocessed observations per station)
                ...
    /ancillary/quaternions
             /sss          (satellite sss)
                sssqbodyYYYYMMDDHHMMSS_yyyymmddhhmnss.LLL (body quaternions data
                                                          file per 24h) 
                sssqsolpYYYYMMDDHHMMSS_yyyymmddhhmnss.LLL (if any, solar panel
                                                   quaternions file per 24h) 

            ...

       /campaign       (from stations not in permanent IDS network. Same 
                        formats)
          /camp_name              (e.g. sorsdal_2001)
              /sss                 (satellite sss)
                sss.files             
                sssdataMMM.LLL.Z (data files per arc/cycle, preprocessed format)
               ...
           /sss                 (satellite sss)
               ...
        /camp_name            
            ...

Note: as the move from format 1.0 to 2.1 is coordinated, it was decided that 
the data filenames would not include a reference to the format.

[=> top]

2.3    Products

/doris
    /products  (Products are deposited in ad hoc DCs areas. 
                The analysis centers need to have an ftp account at both DCs)

        /2002campaign                          (products stored as received)
            ...

        /orbits
            /ccc                               (analysis center)
                ccc.sp1.readme                 (ccc free text explanation)
                ...                
                /sss                           (one subdirectory per satellite)
                    cccsssVV.sp1.dsc           (normalized analysis description)
                    cccsssVV.bXXDDD.eYYEEE.sp1.LLL.Z
                    ...
                /sss
                ...                

        /sinex_global                (one file per center & two files per solution)
             ccc.snx_global.readme   (ccc center descr. of their list of solns)     (modified on 2007-09-19) 
             ...  
             cccWWuVV.snx.dsc        (analysis description)
             cccWWuVV.snx.Z          (results)
             ...

        /sinex_series           
            ccc.snx_series.readme    (ccc center descr. of their list of sinex time series)     
             ccc.snx.readme          (ccc center descr. of their list of soln) 

            ... 
            /ccctu                   (one directory per center)
               ccctuVV.snx.dsc       (analysis description)
               cccYYDDDtuVV.snx.Z    (weekly soln starting with YYDDD)
               cccYYDDDtuVV.sum      (summary file for soln cccYYDDDtuVV.snx.Z) 
               ...
            /ccctu                   (one directory per center)             
               ... 

        /2003campaign                (organisation similar to /sinex_series)
            ccc.snx.readme           (ccc center descr. of their list of solns) 
            ... 
            /ccctu                   (one directory per center)
               ccctuVV.snx.dsc       (analysis description)
               cccYYDDDtuVV.snx.Z    (weekly soln starting with YYDDD)
               ...
            /ccctu                   (one directory per center)             
               ... 
            
        /stcd                        (station coordinates time series)       
            ccc.stcd.readme          (ccc description of their list of solns) 
            ...
            /cccWWtu                   (one directory per solution)
                cccWWtuVV.stcd.dsc     (analysis description)
                cccWWtuVV.stcd.aaaa.Z  (time series for station 1)
                cccWWtuVV.aaaa.gif     (plot of station 1 coordinates) 
                cccWWtuVV.stcd.aaaa.Z  (time series for station 2) 
                cccWWtuVV.aaaa.gif     (plot of station 2 coordinates) 
                ... 
            /cccWWtu                 (one directory per solution) 
                ...  

        /geoc                        (trf-origin time series: two files per soln)
            ccc.geoc.readme          (ccc description of their list of solns) 
            ...
            cccWWtuVV.geoc.dsc       (analysis description)
            cccWWtuVV.geoc.Z         (time series)
            ...

        /eop                         (eop time series: two files per soln)
            ccc.eop.readme           (ccc free text explanations) 
            ...
            cccWWtuVV.eop.dsc        (analysis description)
            cccWWtuVV.eop.Z          (time series)
            ... 

        /iono
            /ccc                     (analysis center)
                ccc.iono.readme      (ccc free text explanation)
                ...                
                /sss                 (one subdirectory per satellite)
                    cccsssVV.iono.dsc      (analysis description)
                    cccsssVV.YYDDD.iono.Z  (results for day DDD of year YY)
                    ...                      
                /sss                 
                ...                
        /...


Note: The "time series" zones being subject to accumulation of new solutions
      which supersede earlier ones, but with another label (if different
      analysis strategy or models), some cleaning automatism may be 
      implemented, e.g. keeping the last one, two or three generations for
      each center. This could be included in the conventional agreements 
      between the DCs and the Analysis centers for the submission of products.
[=> top]

3.    Formats

3.1   Data

      DORIS data format description

3.2   Products

3.2.1 Orbits

      The standard format is SP1

3.2.2 Global Sinex solutions, Time series of Sinex solutions

      The Sinex format is known and accessible.

3.2.3 Time series of specific products

      Station coordinates: description, data
      Coordinates of the TRF origin (geocenter): description, data
      Earth orientation parameters (EOP): description, data (IERS format)

[=> top]

3.2.4 Products documentation files 

      As mentioned in section 1.3, the description of the analysis strategy 
      must be provided with every set of products under a normalized form. 
      Analysts are invited to use the proposed template.


4.    How to deliver products to the Data centers

      Analysis centers willing to submit products to the Data Centers should  
      first contact the Data centers, who will provide them with a username  
      and inform them about the files uploading rules:

      - CDDIS: Carey Noll 
      - IGN:   Bruno Garayt 

 
 
 

Copyright © 1999-2009 CNES-CLS. All rights reserved.