Automatic unattended update of ESP8266 firmware using http server

One of the design goals for my home automation system was to be able to performs OTA (over-the-air) updates. The Arduino IDE makes this very easy, but when you have many devices deployed, it becomes a trifle tedious.

An easier way is to run and "update server" on your network and have each device check at boot time (or via an MQTT message) whether a newer version exists and if so, update itself automatically.

The central issue then is building the update server. The server software, language etc are unimportant:  it's the logic of how it responds to the request that matters. The excellent ESP8266httpUpdate class makes it very simple.

My own server is a "flow" in my NODE-RED controller on a raspberry Pi and - as you will see shortly - the logic is actually very simple.

What makes this all possible is the extra "headers" that the ESP8266HTTPUpdate class adds to the outgoing http request. They allow a great deal of control (should you need it) over how each device is updated.

x-ESP8266-STA-MAC
x-ESP8266-AP-MAC
x-ESP8266-free-space
x-ESP8266-sketch-size
x-ESP8266-sketch-md5
x-ESP8266-chip-size
x-ESP8266-sdk-version
x-ESP8266-mode set to either "spiffs" or "sketch"
x-ESP8266-version

This information allows the server to check that the upload will fit into the requesting device's memory, or provide a specific binary for a specific chip using the unique MAC address. Note also that you can also update a SPIFFS image (e.g. you inbuilt web pages / images scripts etc). The most important item is the version. This can be anything you decide.

The simple protocol and logic is this:

Device: "Hi, I'm firmware version a.b.c, do you have a newer version?"
Server: "Let me see...ah yes I have a.b.d - here it comes..."
If a newer version does exists the server just sends it as a load of binary data in the http reply. The ESP8266httpUpdate class does the tricky part of copying the binary into memory, changing the firmware boot address to the new code than (if requested) rebooting the device to run the new coded

If on the other hand there is no higher version, it replies with a http 304 error which effectively says: "I have nothing for you" and your code continues to run as normal.

Here's my NODE-RED update flow which shows the simplicity of it:

The salmon pink "nodes" are functions written in javascript - before delving into the actual code (which is minimal) we need to talk a little more about my setup:

On my server, I have a folder called /home/pi/trucFirmware which contains:

-rw-r--r-- 1 pi pi   65536 Nov  7 10:22 spiffs_0_4_3_1M.bin
-rw-r--r-- 1 pi pi 1028096 Nov  7 10:22 spiffs_0_4_3_4M.bin
-rw-r--r-- 1 pi pi  352192 Oct 17 13:37 truc_0_4_0.ino.d1_mini.bin
-rw-r--r-- 1 pi pi  344336 Oct 17 13:34 truc_0_4_0.ino.esp01s.bin
-rw-r--r-- 1 pi pi  352208 Oct 17 13:39 truc_0_4_0.ino.nodemcu.bin
-rw-r--r-- 1 pi pi  344528 Oct 17 13:35 truc_0_4_0.ino.sonoff_basic.bin
-rw-r--r-- 1 pi pi  347552 Oct 17 13:36 truc_0_4_0.ino.sonoff_sv.bin
-rw-r--r-- 1 pi pi  348144 Nov  9 17:46 truc_0_4_3.ino.sonoff_basic.bin

I maintain a separate binary for each hardware type (from a single source file with a few #defines) and when a new release is ready I use the Arduino IDE "sketch/Export compiled Binary" menu command for each target device.

Note that even though there are 5 different hardware types, there are only two SPIFFS binaries: a 1M and a 4M version - constructed with the mkspiffs tool - since all the devices have either 1M or 4M flash.

Once you have the mkspiffs tool, building the spiffs is simple. I have a one-line batch file for the 1M version which takes the version number as a parameter (%1)

mkspiffs -c data/ spiffs_%1_1M.bin

and another for the 4M version:

mkspiffs -p 256 -b 8192 -s 0x0FB000 -c data/ spiffs_%1_4M.bin

I then copy all the compile binaries and the SPIFFS .bin files over to /home/pi/trucFirmware

The whole process then, goes like this:

The sketch contains the following code:

#include<esp8266httpupdate.h>
...
#define TRUC_VERSION "0_4_99"
// THIS_DEVICE is set earlier depending on various compile-time defines 
// which eventually define the hw type, e.g. #define THIS_DEVICE "d1_mini"
const char * updateUrl="http://192.168.1.4:1880/update/"THIS_DEVICE;
// this is my raspberry Pi server, the 1880 is the default NODE-RED port
// /update is the url I chose for the server to "listen" for, followed by the device type
...
bool ICACHE_FLASH_ATTR  _actualUpdate(bool sketch=false){
    String msg;
    t_httpUpdate_return ret;
     
    ESPhttpUpdate.rebootOnUpdate(false);
    if(sketch){
      msg="sketch";
      ret=ESPhttpUpdate.update(updateUrl,TRUC_VERSION);     // **************** This is the line that "does the business"   
    }
    else {
      msg="SPIFFS";
      ret=ESPhttpUpdate.updateSpiffs(updateUrl,CSTR(E.getSPIFFSVersion()));
    }
    if(ret!=HTTP_UPDATE_NO_UPDATES){
      if(ret==HTTP_UPDATE_OK){
        msg.concat(" upgraded");
        Esparto.publish("update",CSTR(msg));
        return true;
        }
      else {
        if(ret==HTTP_UPDATE_FAILED){
          msg.concat(" upgrade FAILED code:");
          msg.concat(ESPhttpUpdate.getLastError());
          msg.concat(" reason:");
          msg.concat(CSTR(ESPhttpUpdate.getLastErrorString()));
          Esparto.publish("update",CSTR(msg));
          }
        }
      }
  return false;
}

When I'm ready, I call _actualUpdate(true); and the action then moves to the server:

The first node in the diagram above "listens" for an http request to url http://192.168.1.4:1880/update with the device type appended. It passes this to "Construct search path" function node which has the following javascript code:

msg.type=msg.req.params.type;
var h=msg.req.headers;
msg.version=h["x-esp8266-version"];

msg.mode=h["x-esp8266-mode"];
if(msg.mode=="sketch"){
    msg.payload="/home/pi/trucFirmware/*.ino."+msg.type+".bin";
}
else {
    var sz=h['x-esp8266-chip-size'];
    msg.payload="/home/pi/trucFirmware/spiffs_*_"+(sz/1048576)+"M.bin";
}
return msg;

This just sets up the appropriate path with wildcard for the sys function which follows, which simply runs ls - r <msg.payload>

The output is then fed to the "Compare versions" function node:

var f=msg.payload.split("\n")[0];
msg.filename=f;

if(msg.mode=="sketch"){
    f=f.replace("/home/pi/trucFirmware/truc_","");
    f=f.replace(".ino."+msg.type+".bin","");
}
else {
    f=f.replace("/home/pi/trucFirmware/spiffs_","");
    f=f.replace(/_\dM\.bin/,"");
}

if(msg.version<f){
    node.warn("upgrade required");
    node.warn("will return "+msg.filename);
    return msg;
}
node.warn("no upgrade");
msg.statusCode=304;
msg.payload=[];

return msg;

The switch node then ensures that either the 304 "no update needed" message is sent or the actual new binary is returned and sent back to the device.

There you have it: automatic unattended updates. All you need to do is copy the new binaries to your sever and - voila! - the rest "just happens".  It certainly makes my life easier with 8x ESP-01S, 6x Wemos D1, 4x Sonoff Basic 10x Sonoff S20, 2x Sonoff SV and a NodeMCU - 31 devices in all to update when I make a code change.

Doing is this way is almost effortless.