mbed Blog RSS Feed

Post-mortem debugging with ARM mbed

When it comes to programming microcontrollers, the one scenario that you never want to face is a device that suddenly hangs. It's already very frustrating while you're developing software, and tracing down these bugs can be very time-consuming; it's even worse when the device is already deployed in the field. Replicating the exact conditions in which the device failed can be almost impossible in the lab, especially when the failure manifested itself months after deployment.

Fortunately, mbed-enabled hardware ships with CMSIS-DAP or its successor DAPLink, which allow you to directly hook into devices using the built-in USB ports (on developer boards), or a debugging probe like SWDAP. CMSIS-DAP is responsible for mounting your mbed board as a mass-storage device for easy flashing, but it can also be used to dump the RAM and ROM of a running device, enabling you to do post-mortem debugging on a hanging device.

In this blog post we'll show you how to install all dependencies, crash a device, and subsequently do a post-mortem debug session. To follow along you'll need:

  1. A development board capable of running mbed.
  2. An account at developer.mbed.org - to write faulty firmware.
  3. ARM KEIL uVision 5 - to load the debug session. On OS X, run uVision in VMWare Fusion; it won't install in VirtualBox.

Note: Unfortunately, you may not be able to run the debug session in an unlicensed version of uVision 5. This depends on the size of your RAM, because of the debugger's 32 Kbyte program limit.

  1. Download dump_firmware_classic.py - a Python script that dumps the RAM and ROM off a device, and sets up a uVision 5 project.

This article assumes knowledge of building applications with the mbed online compiler. If you're unfamiliar with mbed, read Getting started with mbed first.

Creating a program that crashes

Here's an application that reads from a pointer at address 0xfffffff when the button at SW2 is pressed. 0xfffffff is an invalid address, and reading from here is not allowed, so the core hard-faults:

#include "mbed.h"

// change this to reflect a button on your own board
static InterruptIn btn(SW2);
static DigitalOut led(LED1);

static void blink() {
    led = !led;
}

static void btn_interrupt() {
    // this will crash your device!
    char* c = ((char*)0xfffffff);
    printf("I read... %c\n", c[0]);
}


int main() {
    btn.fall(&btn_interrupt);
    
    Ticker t;
    t.attach(blink, 0.5f);
    
    while(1) {}
}

Save the program and verify that it compiles.

Exporting to an offline toolchain

When you build an application with the online compiler you only get access to the firmware. If you want to do a post-mortem debug session you also need the debug symbols for this build. To obtain those you can export the program to an offline toolchain, and build locally.

Right click on your program in the online compiler, and select Export.

/media/uploads/janjongboom/postmortem1.png

Choose GCC (ARM Embedded) as the toolchain and click Export.

/media/uploads/janjongboom/postmortem2.png

A ZIP file with the source code of the application, all libraries and a Makefile is downloaded. Extract the ZIP file to a new directory on your computer.

Building locally

Open a terminal and navigate to the directory to which you extracted the ZIP file. We can now build the program locally and obtain the debug symbols.

$ make
# on some targets (like nrf51-dk), also call $ make merge

When the build finishes, your build directory will have either a *.bin or *-comibned.hex file, depending on your board. Use drag-and-drop to copy the file onto your board to flash the program.

The program starts running and blinks the LED (press the Reset button if the LED does not start blinking). Now press the SW2 button to crash the board.

Starting a post-mortem debug session

Now we can put the dump_firmware_classic.py Python script to good use. Run the script with the crashed device plugged in:

$ pip install intelhex pyOCD
$ python dump_firmware_classic.py
[1/6] Using build directory '.', verifying build...
[2/6] Build verified. Connecting to board...
[3/6] Board recognized as k64f. Downloading ROM...
[4/6] Dumped ROM. Downloading RAM...
[5/6] Dumped RAM. Creating uVision project...
[6/6] Done. Open 'crashdump/crashing-k64f.elf.uvprojx' in uVision 5 to debug.

A crashdump folder is created. It contains debug symbols, the RAM and ROM of the device, and a uVision 5 project file.

Double click on the project file to open uVision.

Loading the session in uVision 5

In uVision, choose Debug > Start/Stop Debug Session to start.

/media/uploads/janjongboom/postmortem3.png

In the Disassembly panel we see that we're in the HardFault_Handler, which is expected. We can drill down into the stack by selecting Debug > Step.

/media/uploads/janjongboom/postmortem4.png

Now we get more information about the state of the application. On the right bottom corner we see the Call Stack that led up to the crash. We can see that we passed an interrupt (handle_interrupt_in) and that we went through one of our own functions (btn_interrupt) right before the device hard faulted.

/media/uploads/janjongboom/postmortem5.png

Now that we found the function we can also find the actual line that our application crashed on. In the Call Stack panel, right click on btn_interrupt and select Show Callee Code.

/media/uploads/janjongboom/postmortem6.png

We now jump into the disassembled source. On the left panel we see the registers, and in the Disassembly panel we see the assembly that led up to the crash:

/media/uploads/janjongboom/postmortem7.png

When we read the assembly we see:

MVN           r3,#0xF0000000              // Move not
LDR           r0,[pc,#4]  ; @0x000004EC   // Load word from memory
LDRB          r1,[r3,#0x00]               // Load word from memory (type: unsigned byte)

On the last line before the crash we tried to read from registry R3 (0x0FFFFFFF) into registry R1. We also see that right after that, we called the printf function. This maps perfectly to our C++ code, and we found the line which crashed the device.

char* c = ((char*)0xfffffff);
printf("I read... %c\n", c[0]);

Never-ending loops

This approach can be used for more than just debugging hard faults. We can also use it when a device hangs because it entered a never-terminating loop. Let's look at the following code:

#include "mbed.h"

static DigitalOut led(LED1);
static void my_loop_function(void) {
    uint8_t turns = 20;

    while(turns-- > -1) {
        led = !led;
        wait_ms(200);
    }
}

int main() {
    my_loop_function();
    return 0;
}

Here turns will always be greater than -1 (because turns is a uint8_t), so the device will never exit the loop. When we start a post-mortem debug session, we can immediately see the function that we are currently stuck in:

/media/uploads/janjongboom/postmortem8.png

And if we're lucky, we can even see the values on the Call Stack, showing that turns overflowed and is now 0xED.

Conclusion

CMSIS-DAP on ARM mbed can be a tremendous help with hard to find issues. The ability to recover a crashed device from the field, and use post-mortem debugging on it, is a great addition to the toolchain of every developer.

To replicate the tests we used in this article, take a look at post-mortem-debugging on GitHub. You'll find the Python script and all source code used in this article. Happy disassembling!

-

This article was written by Russ Butler (Embedded Software Engineer working on DAPLink and pyOCD) and Jan Jongboom (Developer Evangelist IoT).

Summer mbed Events in Taiwan and China

This summer is gearing up to be very busy for the mbed team and we couldn't be more excited about the many events coming up in China and Taiwan, such as Maker Faire Taipei, the Thread Group Open House in Shanghai, Computex in Taipei, and more. These are a great opportunity to see the latest demos from the mbed team and get a hands-on experience at one of the local workshops.

Maker Faire, Taipei, May 7-8

/media/uploads/katiedmo/makerfairetaipei_IDtCv20.png

Meet us at the ARM booth at Maker Faire Taipei, you can’t miss us by the front door of the National Taiwan Science Education Centre. The first thing you will see from a distance is an exoskeleton which was developed by Neil Tan and his team and shown on Maker’s TV. This exoskeleton was designed to improve mobility in older people. The device estimates the user's intention by measuring the bending of the knees, foot contact force and the orientation of the torso. Depends on the user intention, motors apply torque to argument the one’s strength. It was built using an STM32F401 board running mbed.

Another interesting demo to look out for is the one we call Rob the builder, if you can’t see it you will definitely hear it from a distance as he is regularly hit on the head. Rob was built by those makers at IBM and Multitech to address the problem of construction workers exposed to dangerous impacts, loud noises, vibration and chemical exposure. Something that stands out from this product is the use of LoRa as a wearable radio technology, typically we see Bluetooth used in wearables. The reason for choosing LoRa is its exceptionally low power but also it’s very long range in harsh environments. The hard hat is fitted with an impact sensor which is connected to a Multitech mDot radio board, data is then fed to IBM Bluemix for processing. If Rob receives a particularly dangerous impact to his head, other workers are informed to offer him help –we have a baseball bat on site if you wish to hit him on the head.

Have you ever wanted to create your own IoT gadget but not sure where to start? Spend 5 minutes at our mbed challenge and we will show you how to write software and run it live on an ARM processor. If you can complete the challenge then take away a development board and continue creating your own IoT gadgets.

  • WHEN: May 7th-8th , 2016
  • WHERE: National Tawian Science Education Center

Open House for The Thread Group, Shanghai, May 10

/media/uploads/katiedmo/thread_logo.jpg

Join us and other Thread member companies in Shanghai on 10 May 2016 to learn about Thread, the new home networking protocol based on industry standard Internet protocols. Thread is a key ARM mbed technology, supporting silicon partners and OEMs to leverage growth in smart home markets. We will explain how Thread works and why it will become the foundation for connecting “Things” in the home. There will be food, beverages, and networking opportunities.

  • REGISTER: Here
    The nominal ¥325 CNY registration fee includes access to the event, food and beverages, networking, and a shirt.
  • WHEN: May 10, 2016 from 4-7 p.m. CTZ
  • WHERE: JW Marriott at Tomorrow Square, 399 Nanjing West Road, Shanghai, Huangpu District 200003, China

More about Thread
Thread is designed for consumers and devices in and around the home, Thread is an IPv6 networking protocol built on open standards for low-power 802.15.4 mesh networks that can easily and securely connect hundreds of devices.

Computex, Taipei, May 31-June 4

/media/uploads/katiedmo/computex_logo3.png

At Computex in Room 2510 of the Grand Hyatt we will be exhibiting a wide range of demos including: a Bluetooth Low energy gateway developed by Red Bear Labs which enables edge devices to be managed through mbed Device Connector. The gateway is based on raspberry Pi Zero and Red Bear’s IoT HAT coming in at under $15. Also have a close look at Zebras Time Tracking Solution for Acute Myocardial Infraction Patients outside of its hospital environment. This system allows medical staff to track how long a patient has been in their care and precise timing between treatment. As a patient is admitted to the hospital a wrist band it provided which regularly transmits a time signal which is received by various terminals throughout the hospital and transferred back to the patients records via the Zebra Zatar cloud system running mbed Device Server.

  • WHEN: May 31 - June 4, 2016
  • WHERE: Taipei World Trade Center (TWTC), No. 1, Section 5, Xinyi Rd, Xinyi District, Taipei City, Taiwan 110

2016 Robotics Forum & Robotics Capital Initiation Forum, Shenzhen, May 21

An mbed workshop will be hosted at the 2016 Robotics Forum and Robotics Capital Initiation Forum in Shenzhen. This workshop will take you through the basics of getting started with mbed and how to perform one simple robotics sensor exercise.

  • REGISTER: Free for all attendees, more info here
  • WHEN: May 21st, 2016
  • WHERE: Greater China Financial Center, 1003 Shenan Rd. Futian District, Shenzhen

More mbed Events

Now available: Secure WebSockets and MQTT over TLS libraries

Two weeks ago Real Time Logic released some very interesting libraries for mbed. Real Time Logic maintains the SharkSSL library, which contains a lightweight TLS client suitable for running on Cortex-M series microcontrollers. Previously SharkSSL was only available under a commercial license, but they have now released a light version of SharkSSL for mbed users.

While that in itself is already worthy of a blog post, they also released a number of (according to y'all in the forums) highly sought after examples: secure WebSockets (WSS) and MQTT over TLS.

/media/uploads/wini/xwebsocket-example.png.pagespeed.ic.jJ7tzhS72j.png

SharkSSL-Lite running on an mbed board, talking to a WSS server.

The examples currently run on boards with an Ethernet port (verified on an FRDM-K64F), but should be easy to port to other IP-based protocols like WiFi, Cellular and 6LoWPAN.

Building your own LoRa network

There is a lot of buzz around LoRa, a wide-area network solution that promises kilometers of range with very low power consumption. It was already possible to build LoRa end-nodes using mbed with the Multitech mDot platform or the SX1276MB1xAS shield, but a LoRa network is a lot bigger than just an end-device. If you're not in the vicinity of an existing LoRa network you'll need to think about gateways and software for a network service. Where to get started?!

That's why today we are publishing a step-by-step guide on how to build your own LoRa network. From end-device to gateway to application. The guide can be found here: Building your own private LoRa network on mbed Docs.

/media/uploads/janjongboom/mdot-conduit.jpg

LoRa Hackathon, April 5th in Santa Clara, with The Things Network and Multitech

Do you want to get started building LoRa devices with mbed, or show that you're the best long-range hacker? On April 5th - during the LoRa alliance meeting in Santa Clara - ARM, The Things Network and Multitech will be hosting a LoRa hackathon. This is your chance to build IoT devices with kilometers of range that run for years on a battery, and win prizes along the way.

The LoRa alliance meeting on April 4-6 is the go-to event for anyone involved in the LoRa ecosystem. If you were planning to attend the Open House on April 6th you'll get free additional access to the hackathon the day before, where mentors from Multitech, ARM and The Things Network will answer all your questions on the practical side of LoRa. You'll also have a chance to win mDot modules, Conduit gateways, development boards, and much more. We'll also bring a ton of sensors with us, so you can go as crazy as you want. In the afternoon of the 5th we'll be hosting some introduction sessions, and make sure everyone is set up. After that you'll be free to hack on whatever you want. We'll announce the winners on the morning of the 6th during the Open House.

To participate in the hackathon you'll need a ticket for the LoRa Alliance meeting (register as a non-member for the Open House). See you there!