Joseph Ellsworth / data_log

Proivdes data log data structure for FRAM, EPROM chip with functions to read chip and send back on serial data string.

Dependencies:   W25Q80BV multi-serial-command-listener

Dependents:   xj-data-log-test-and-example

Data Logging Data structure

Both Read and write seem to be working fine but testing has been limited.

Motivation

I needed a flexible data log structure that could tolerate evolving data structures as I discovered more things that needed to be measured. I also wanted something that is mostly human readable while remaining sufficiently concise to make efficient use of expensive storage resources.

I found it challenging to track everything needed to perform after the fact analysis we need to improve our state machine. In addition what I wanted to measure changed with time and I needed a robust way to log this data so we could analyze it latter. without breaking or converting all the old data. A self describing data format like JSON or XML would work but FRAM is expensive so I wanted something flexible but still concise.

I am working on A2WH which is a electronic controller for a sophisticated product that balances many sensors, battery charging from photo voltaic panels, controlling speed of many different fans, humidity and environmental data. Our main challenge is we never have enough battery power to run everything so we have to make decisions about what to run in an effort to produce the maximum amount of water from the available solar power resource. Our 2nd challenge is that balancing system actions such as increasing or decreasing fan speeds is driven by a complex internal prediction model that attempts balance many competing thermodynamic requirements. To get all this right requires substantial after the fact analysis and that requires logging a large amount of evolving data.

Design Notes

See: data-log-read.me.txt in the same project

Sample Use and Basic Test

• https://developer.mbed.org/users/joeata2wh/code/xj-data-log-test-and-example/wiki/Homepage

Serial Command Interface

COMMANDS
  readall= send entire contents of log
  readlast 999
     999 = number of bytes from tail of log to retrieve
  tread 333 444
     333 = starting offset to start reading log
     444 = number of bytes to retrieve from log
  erase = erase log and start a new one
  help  = display this help

Other Chips

For legacy reasons I am using the library for "W25Q80BV.h" simply because I started with it. The actual FRAM chip I am using is 2 MBit FRAM MB85RS2MTPH-G-JNE I also tested it with SRAM 23LCV1024-I/P

Simplifying Design Decision

I made a simplifying assumption that every-time we generate a log entry I record the offset of the next write at a specific location in the chip. This works and is fast but it causes lots of updates against a single location. I prefer FRAM because this would rapidly fatigue FLASH chips like the W25Q80BV. Storing this pointer data in the CPU has the same fatigue problem.

Another other option would be to store this offset and our other critical configuration data in the clock chip but it is susceptible to loosing power and loosing this critical data.

One reason I don't log directly to the micro-sd is for the same fatigue problem but it is mostly for power management.

The FRAM chip provides adequate durability and data retention through power outage. The power outage retention is critical because the A2WH systems can be buried under feet of snow in the winter and solar panels do not provide much recharge under that condition.

One design option I have considered but not yet implemented is using a much smaller FRAM chip critical configuration data and rapid update data and then log directly to a larger and less expensive FLASH chip .

Journaling to micro-SD

I latter decided to add features to allow after the fact copying of the data to micro-sd cards to obtain larger log storage without soldering in more chips. I found the micro-sd consume quite a lot of power so I still want to log direct to the FRAM then copy to the micro-sd when I have surplus power available. Still thinking about consolidation tactics to allow re-use of FRAM after the data has been copied ot micro-sd.

Future

• Support fast indexing by date to only pull back log entries between two dates.
• Record most recent record headers for each record types where they are fast to access so we can send them with the data when only sending back portions of the data.
• Support wrap around use of data log to re-use storage on chip.
• Copy Data to micro SD card and consolidate FRAM chip for re-use.

License

By Joseph Ellsworth CTO of A2WH Take a look at A2WH.com Producing Water from Air using Solar Energy March-2016 License: https://developer.mbed.org/handbook/MIT-Licence Please contact us http://a2wh.com for help with custom design projects.

Added Python parser to convert DLOG format captured from readall command in serial port into CSV format for use in excel an R.