6 years, 9 months ago.

An issue on high data rate writing on micro SD using SDFileSystem

I’m developing an IMU + data logger and I am having an issue in storing the data to microSD card at high data rate. I am experiencing massive drop of writing speed every each 20 ms. I’m wondering if someone could give me any input for solving the issue.

Through some test ,I found out this issue is caused by SDFileSystem Library . There are massive delays every 4 to 6 data (about 220 bits).

I'm using Nucelo F411RE. /media/uploads/takumaoura/--.png

2 Answers

6 years, 9 months ago.

Can you share some of your code? How is it structured? from what you are seeing I suspect either you are reading data in the main loop or writing to the SD card in an interrupt.

The SDFileSystem will be buffering data into blocks and then writing them, those writes can take up to 100 ms at times.

The solution is to have two buffers, each able to hold at least 200 ms of data, ideally more if you have the RAM space.

You use an interrupt to log incoming data into the currently active buffer and keep track of how much data is there.

The main loop then looks to see if there is any data in the buffer, if there is it does the following:

  • disable interrupts
  • copy the value for how much data is in the buffer
  • change the active buffer pointer to point to the second data buffer
  • enable interrupts
  • write the contents of the buffer to the SD card.

This way you don't have to copy large amounts of data around, interrupts are only disabled for a very brief period so there is no risk of the incoming data overflowing the rx buffer and the SD write doesn't prevent new data from being logged.

Using this system I can log 7 values from a 2 kHz IMU (56kB/s) without any dropouts. If you track the size of the logged data it is mostly writing 1 value at a time but every now and then I'll get a block of >50 values that arrived while the sd card was busy indicating that the write function took over 25 ms to complete.

Accepted Answer

Hello Andy,

thank you for the comment! Here is a part my code. What it does is reading data from IMU and writing them in SD using fprintf in main loop.

MPU9250 SD

 while(1) {
  
  // If intPin goes high, all data registers have new data//
  if(mpu9250.readByte(MPU9250_ADDRESS, INT_STATUS) & 0x01) {  // On interrupt, check if data ready interrupt

    mpu9250.readAccelData(accelCount);  // Read the x/y/z adc values   
    // Now we'll calculate the accleration value into actual g's
    ax = (float)accelCount[0]*aRes - accelBias[0];  // get actual g value, this depends on scale being set
    ay = (float)accelCount[1]*aRes - accelBias[1];   
    az = (float)accelCount[2]*aRes - accelBias[2];  
   
    mpu9250.readGyroData(gyroCount);  // Read the x/y/z adc values
    // Calculate the gyro value into actual degrees per second
    gx = (float)gyroCount[0]*gRes - gyroBias[0];  // get actual gyro value, this depends on scale being set
    gy = (float)gyroCount[1]*gRes - gyroBias[1];  
    gz = (float)gyroCount[2]*gRes - gyroBias[2];   
  
    mpu9250.readMagData(magCount);  // Read the x/y/z adc values   
    // Calculate the magnetometer values in milliGauss
    // Include factory calibration per data sheet and user environmental corrections
    mx = (float)magCount[0]*mRes*magCalibration[0] - magbias[0];  // get actual magnetometer value, this depends on scale being set
    my = (float)magCount[1]*mRes*magCalibration[1] - magbias[1];  
    mz = (float)magCount[2]*mRes*magCalibration[2] - magbias[2];   
  }
   
    Now = t.read_us();
    lastUpdate = Now;
    
    sumCount++;
    
    fprintf(fp, "%d %d %f %f %f %f %f %f %f %f %f \r\n",Now, sumCount, ax, ay, az, gx, gy, gz, mx, my, mz);
    
    if (sumCount > 100){
         pc.printf("Done");
        break;
        }
}

UPDATE: So is it true that within the certain data output rate of IMU, average data writing speed to SD is higher than the data output speed ? I hope you understand what I'm trying to ask. I'm so sorry for my poor English.

posted by Takuma Oura 19 Jul 2017

As suspected you need to move the data reading into an interrupt so that the main loop is purely saving the data.

The issue is that the SD card is on average fast enough but SD cards work on blocks of data and so unless you close a file the data isn't always written to the card straight away. It is buffered up until a whole block needs to be written. This means that some writes are very fast and some are very slow. Your code gives writing to the SD card the same priority as reading the IMU and so when it gets a slow write it stops reading the IMU.

The solution is to use interrupts to make reading the IMU higher priority than the SD card but this means you must then be able to cope with having multiple readings between writes.

Does the IMU have a data ready output pin that you can use as an interrupt input? If not set a ticker to run at some rate faster than the IMU output, between 2 and 3 times the expected IMU data rate and then use that to check if there is data waiting.

As a first pass you probably want something like this (posted in two parts due to the size limit on comments):

// the size of the buffers to use, you may need to make this larger if you have the RAM.
const int bufferSize = 100;

// it makes life a lot easier to define a structure to hold the data from each sample
typedef struct adcData_s {
    float ax,ay,az;
    float gx,gy,gz;
    float mx,my,mz;
    uint32_t time;
} adcData_t;

// two buffers to store the data in, we rotate between them.
adcData_t buffer1[bufferSize];
adcData_t buffer2[bufferSize];

// set which buffer is being used.
adcData_t *activeBuffer = buffer1;

volatile int dataInBuffer = 0;

// call this when you know that there is date ready either from a ticker that checks the status register or a data ready interrupt input.
void onDataReady(void)
{

// basically your code but writing to the next empty entry in the active buffer.
    adcData_t *dataPointer = activeBuffer + dataInBuffer;

    mpu9250.readAccelData(accelCount);  // Read the x/y/z adc values
    // Now we'll calculate the accleration value into actual g's
    dataPointer->ax = (float)accelCount[0]*aRes - accelBias[0];  // get actual g value, this depends on scale being set
    dataPointer->ay = (float)accelCount[1]*aRes - accelBias[1];
    dataPointer->az = (float)accelCount[2]*aRes - accelBias[2];

    mpu9250.readGyroData(gyroCount);  // Read the x/y/z adc values
    // Calculate the gyro value into actual degrees per second
    dataPointer->gx = (float)gyroCount[0]*gRes - gyroBias[0];  // get actual gyro value, this depends on scale being set
    dataPointer->gy = (float)gyroCount[1]*gRes - gyroBias[1];
    dataPointer->gz = (float)gyroCount[2]*gRes - gyroBias[2];

    mpu9250.readMagData(magCount);  // Read the x/y/z adc values
    // Calculate the magnetometer values in milliGauss
    // Include factory calibration per data sheet and user environmental corrections
    dataPointer->mx = (float)magCount[0]*mRes*magCalibration[0] - magbias[0];  // get actual magnetometer value, this depends on scale being set
    dataPointer->my = (float)magCount[1]*mRes*magCalibration[1] - magbias[1];
    dataPointer->mz = (float)magCount[2]*mRes*magCalibration[2] - magbias[2];

    dataPointer->time = t.read_us();
    dataInBuffer++;
    if (dataInBuffer == bufferSize) {
        // ERROR - buffer overflow. You may want to turn an LED on or output an error message.
        // Increase the buffer size until this never happens and then add some extra just to be safe
    }
}
posted by Andy A 19 Jul 2017

part 2:

// writes a set of data to the sd card.
// assumes that the file pointer is global, if not pass it as a parameter.
void saveData(adcData_t *dataBuffer, int recordsToWrite, int recordNumber)
{
    adcData_t *dataPointer;
    // write each entry in turn...
    for (int count = 0; count<recordsToWrite; count++) {
        dataPointer = dataBuffer + count;
        fprintf(fp,  "%d %d %f %f %f %f %f %f %f %f %f \r\n",
                dataPointer->time, recordNumber+count,
                dataPointer->ax,dataPointer->ay,dataPointer->az,
                dataPointer->gx,dataPointer->gy,dataPointer->gz,
                dataPointer->mx,dataPointer->my,dataPointer->mz);
    }
}

void main (void)
{
    ...
// setup code
    ...
    int sumCount = 0;
    while (1) {
        if (dataInBuffer > 0) { // some data is ready to write
            __disable_irq();     // disable interrupts while we swap buffers
            int recordsToWrite = dataInBuffer; // note how many entries there are.
            adcData_t *saveBuffer = activeBuffer;             // get a pointer to the buffer we need to write
            if (activeBuffer == buffer1)             // set the active buffer to the other buffer.
                activeBuffer = buffer2;
            else
                activeBuffer = buffer1;
            dataInBuffer = 0;               // mark the active buffer as empty
            __enable_irq();                 // re-enable interrupts
            saveData(saveBuffer,recordsToWrite, sumCount);  //save the data to the card.
            sumCount += recordsToWrite;
            if (sumCount > 100) {
                pc.printf("Done");
                break;
            }
        }
    }
}

(note, I've not even hit compile on this code so there are probably a few bugs in there but the basic structure should be good)

posted by Andy A 19 Jul 2017

Andy

I really appreciate your help. I am using MPU-9250 as IMU and it seems to have a interrupt output function.But I'm going to try a ticker interrupt first.

By the way, may I ask what IMU and microprocessor you used? I would like to know if my Nucelo F411RE is good enough.

posted by Takuma Oura 20 Jul 2017

I was using an ADIS 16490 IMU running at 2 kHz over SPI and an LPC1768 processor (100MHz M3, no FPU). Rather than using printf I wrote the data to the card in binary and then wrote a PC application to convert the binary logs to csv. This more than halved the amount of data to be written and avoids the slow printf function call. I did try running at 4 kHz but it started to drop samples. I think I could have improved that by improving my IMU read functions, they had a number of waits in there due to IMU timing requirements, but since 2 kHz was good enough it didn't spend the time trying.

posted by Andy A 20 Jul 2017

Thank you for the information. I think my processor will work.

It looks like the average writing speed of my system is around 9kB/s thanks to the each 25 ms delay. Is it because I'm using fprintf?

posted by Takuma Oura 20 Jul 2017

I don't think that's a very fair test of your average write speed since you're not trying to write all the time with your current code. printf is slow but it's not that slow.

I have no idea if it would make any difference but with my code above you could change it to use sprintf to put all of the waiting data into a buffer and then write that full buffer in one go. That way when you have multiple entries to store the file system would see a single large write rather than lots of smaller ones.

posted by Andy A 20 Jul 2017

Hello Andy,

I tried your code and it worked pretty good! I set a ticker to call a simpler onDataReady function with 5 ms intervals (this is probably the maximum speed in my system. ) and I don't see any dropout.

Thank you for sharing your code with me. I really appreciate your help and generosity.

simpler onDataReady

void onDataReady(void)
{
// basically your code but writing to the next empty entry in the active buffer.
    adcData_t *dataPointer = activeBuffer + dataInBuffer;
 

    // Now we'll calculate the accleration value into actual g's
    dataPointer->ax = 1;  // get actual g value, this depends on scale being set
    dataPointer->ay = 1;

         ...
 
    dataPointer->time = t.read_us();
    dataInBuffer = dataInBuffer + 1;
    if (dataInBuffer == bufferSize) {
            printf("Error  ");

        // ERROR - buffer overflow. You may want to turn an LED on or output an error message.
        // Increase the buffer size until this never happens and then add some extra just to be safe
    }
}

UPDATE: Would you mind my asking you a personal question about your history with microprocessors?

posted by Takuma Oura 25 Jul 2017

My history? Only 3 years of real microprocessor experience. I'm a hardware engineer who ended up in a job which involved writing desktop pc software for 6 years. After that picking up the level in between was fairly easy. Now I'm a generalist, design the electronics, program the micro and then write the pc application to process the results with the odd web interface thrown in.

posted by Andy A 25 Jul 2017

I was wondering how you became so familiar with microprocessors and mbed libraries but now I can see that you did through your professional career.

Actually I'm a first-year masters student in a Graduate School of Mechanical Engineering and I started making this biologger in this April with no experience in a microprocessor and little experience in coding. I have no idea where to start, but thanks to your help I think I can manage to make a satisfactory logger.

posted by Takuma Oura 25 Jul 2017
6 years, 9 months ago.

Could you try this example https://github.com/ARMmbed/sd-driver/blob/master/README.md and try to observe the same behavior? It uses the latest mbed OS, which includes an SDFileSystem as part of the RTOS, rather than the separate library.

Ms. Marsh

Thank you for the response. I will try the example.

posted by Takuma Oura 19 Jul 2017