Data Logging Data structure

File content as of revision 11:bf816d33be80:

To make the data log format somewhat flexible and 
some what easy to debug while remaining somewhat
consise while remaining human readable.  The format
will be a custom format but is designed to be relatively
easy to parse while accomodating the following caveates:
   * Fields logged over time will change as firmware is reved.
   * Old log entries for the same record type will have different fileds.
   * storage in fram is realtive expensive so self describing data 
     formats like JSON would not be cost effective. 
   * When things mess up we want to be able to parse it and 
     analyze it and have some chance of recovery 
   * We need fast access to pont in time for at least 1 day 
     of data. 
     
will use a custom format.  

Each record type will have record type label which will be
composed of 4 characters separated from the data by a The known
record types are:

Records will be appended to the data log in sequential order. Each
record will be terminated with a \n.  Some records will contain fixed
length data but will still be terminated with \n and will never contain
a \n in the stored data. unless the first number after the record type 
is 16 bit number storing the number 

The general assumption is that all data will be stored for the short
term in high speed FRAM or SRAM chips and copied to EPROM or Micro 
SD chips in the background once sufficnet data has accumulated to 
minimize over-write fatigue in the long term storage.  I used FRAM 
specifically to allow after power loss data retention without having 
to worry about overrite fatigue for specific segments that would need
rapid updates.    The other assumption is that data is stored and logged
then will be accessed generally starting at a given date reading forward 

Short term Simplification Notes:
  Each time the CPU reboots it will record current record header field 
  names for all record types active at that time.  This will create some
  duplicates but save some overhead in firmware.
  
  The firmware will only record next log position internally.  It will
  simply write next log item at end of prior one.
  
  Firmware will not record day start indexes.
  
  Firmware will provide simple dump feature to send all data to serial
  IO line. 
  
  TODO:  How to read serial input in non blocking
     fashion. 


Beginning Address.  Since the log is presumably stored in a FRAM chip
without support for a file system driver we must know the offset of where
the log begins.  Space before the log is generally used for system configuration
data. 

Pre-Log Values: 
  DATOFNDX:
    Date Offset Index.  Array of 100 string formated number pairs
    containing dateOffset records. Each pair contains
    one number which is string containing the date of 
    the first record in that date offset in the format
    ccyymmd,offset the assumption is we can quickly scan
    or binary search this array to find the first offset
    record that describes the date range we are seeking then
    jump and read that offset record.  Each time we add a new 
    date offset log item we update this value. 
    

Record Types:

date: 
  Date types will be composed of the format ccyy-mm-dd hh:mm:ss
  a new date record will be recorded whenever the system detects 
  a new date when preparing to generate a new log item.  At items
  that occur after this date are presumed to occur on that date. 
  
Record Type active defenition:
  A space at first of system that stores the iformation about the
  currently active record types. 

Record Type defenition: 
  Prefix RTDE,RecordType,List of Field Names This is written into
  the log whenever the system reads a record type defenition from the 
  current record types that is different than the once currently used
  in the firmware.  When ths occurs it will write a new record type 
  defenition  
  

next_write_offset:  NWOF 
  Next write offset record stores the byte postion of the next
  postion in the log byte offset where the next 

date_offset: 
  Fixed Length Binary Data - 
  Preix: DATOF 
         NumRec
         Offset Of Next DATOF 32 bit number
         Offsets (array of 32 bit numbers)
         RecordOffset records are store in a region of  storage
  after the before the actual data logs are written.  These are
  fixed length records where the starting byte offset for each
  days of logging are stored as 32 bit unsigned integer numbers
  as an array. The first number contains the number of spaces 
  pre-allocated. The last number written will be the offset to 
  the next date_offset record.  In general we will allocate 365
  365 days worht of space so the total row length will be 
  32 + (365 * 32) + 32 bits = 11,744 bits
  X number of days with a a
  
  
reading: 

System state: