CEPD News
(303) 415-1112 Broomfield

(303) 415-1112 Longmont

(970) 541-4777 Loveland

June 2021 - Vol 14, Issue 6
Watch Dogs on Mars
Classic watchdog timers, implemented either in dedicated hardware peripherals or interrupt driven firmware, have been a staple of embedded systems for decades. And recently, they’ve even made a high-profile appearance in one of NASA’s Mars missions.
 
A watchdog timer is a simple fallback mechanism intended to automatically reset a micro-controller (MCU) once its timer interval expires; the intention is that the MCU firmware will restart the timer continuously, usually as part of its top-level service loop. Under normal operation this means the timer will never actually expire and issue a reset. However, if a bug causes the MCU firmware to freeze up and it ceases to reset the timer, the watchdog will fire. This fallback reset is the last line of defense to restart the system to a known working state, and prevents a frozen system from hanging forever.
 
Prior to the now celebrated first flight of JPL’s Mars Helicopter Ingenuity, the small drone actually failed its first take off attempt. During the pre-flight rotor spin up sequence, an undiscovered bug in the vehicles pre-flight to flight state transition caused the entire software system to lock up. The software had entered an unrecoverable state, and the vehicle was left standing on the surface of an alien planet - with its only connection back to earth being its now inoperable short range radio link to the rover. Eventually, the humble watchdog timer fired, issuing system wide hardware resets. After resetting, the radio link returned to normal operation, and event logs could be downloaded back to earth for analysis. Once the bug was identified and patched, updated firmware could be sent back to Mars, and the mission went on to its now well-known success. It was ultimately an easy bug to solve, but if the main processor hadn’t been configured to utilize its watchdog timer, that single undetected firmware bug would have caused a complete loss of mission.
 
The Ingenuity helicopter’s close call on Mars is a good reminder to designers and operators of IoT and remote monitoring systems here on Earth. As remote, over the air firmware upgrades are becoming the norm, engineers need to take every possible precaution to ensure devices don’t become stranded in the field. If a bug makes it into the IoT fleet’s field devices and leaves them all unreachable for a patched update, the field service costs can quickly become enormous. Imagine scheduling and deploying field technicians to manually update firmware images for a fleet of 10,000+ remote IoT devices. Thorough firmware testing protocols are the first step, but even if an IoT operator could test to the standards of NASA flight hardware, the risk clearly still exists - and that’s where embedded fallbacks can offer additional protection. Watchdog timers are the trusted last resort, but they can also be combined with higher level fallback mechanisms such as telemetry reset timers (as featured prominently on the Voyager missions), and “fail to downgrade” automatic firmware rollback systems.

-C. Pifer
IEEE Denver Upcoming Events
Machine Learning for Wireless Communications and Networking: Motivations, Case Studies and Open Problems

Date and Time
Space Foundation Upcoming Events
36th Space Symposium

Date and Time
  • Venue: Broadmoor Hotel, Colorado Springs, CO
  • Date: August 23 - 26, 2021
  • More Information

Services
If you would like help developing a new product or if your projects are understaffed, CEPD can help. Our staff draws on years of diverse product design experience to provide creative and timely solutions for your product needs. Some of our specialties include:

  • Technical Project Management
  • Embedded Systems Hardware and Software
  • Digital Signal Processing (DSP)
  • Data Acquisition
  • Wireless Sensor and Telemetry Systems (Zigbee, Cellular, VHF, Bluetooth, ANT+, etc.)
  • IoT
  • Control Systems
  • Programmable Logic: FPGA / PLD
  • Analog Circuit Design
  • Switching Power Supply Design
  • Battery Charging (all Chemistries)
  • PCB Design and Layout
  • Analysis, Test and Documentation

We provide cost effective and expedient design options for our clients, regardless of the project's complexity. Our detailed proposals, accurate estimates and time schedules will help you manage each phase of the project. 
Sincerely,
The Staff of CEPD, Inc.
Colorado Electronic Product Design, Inc.,
700 Burbank St, Broomfield, CO 80020 (303)415-1112
601 3rd Ave, Ste 201, Longmont, CO 80501 (303)415-1112
1339 S. Garfield Ave, Loveland, CO 80537 (970)541-4777
© 2021, CEPD, Inc.