Esparto V2 observation on AsyncWebSocket performance and limits

Esparto V2 nears completion! During the final testing I had some problem with crashes, and traced it to a high rate of writes to the AsyncWebSocket. If the user requests socket writes faster than the browser can consume them a queue / backlog builds up and the free heap decreases rapidly until...bang!

That's no great surprise, but it may help other folk to know what the limit is. Bear in mind that the following examples were deliberately throwing a sustained rate of writes with no pause for recovery. In the normal course of events where socket writes are fairly random, there won't be so much of an issue. My app sends socket writes to flash the pins on and off in "real time"  and also updates the sliding graphs, the UpTime and the pusling heartbeat. So there is a continual "background" of 3/sec even with no GPIO pin activity.

Esparto V2 web UI, dynamically updating system panel running on Wemos D1 min

If you connect a noisy sensor to an input pin (e.g. a sound sensor that fires thousands of pin transitions per second) a crash is the obvious result and so each pin can be "throttled" to avoid problems. It was in trying to find a "safe" throttling value that I generated the graph and I hope others will find it useful.

As can be seen, once past about 21 writes/sec, things go down hill rapidly (but predictably).  If your app does not have heap protection and/or throttling then you are going to run into problems if you sustain more than that amount. If you know your free heap then the graph will allow you to calculate what rate is safe for any given period.

For example, say you start with 30k free heap. Some part of your code fires a burst of socket writes at 50 per second. Reading up from 50 and then back to the vertical axis you will see that about 2kb heap per second will be consumed, giving you 30 / 2 = 15 seconds before you crash.

If you can organise your app so that it backs off before that time and allows the heap to recover back to 30k then you can safely repeat the process.

Also you can see that once the rate drops, the heap recovers rapidly, so bear this in mind when designing you app and try to minmise socket writes.

I created a lot of problems for myself early on by trying to load the whole UI as a monlithic block, with 40,50, 60 or more socket writes to set it up as soon as the socket opened. Trying to get round this problem led me to the rate throttling method I now use, thanks to the data behind this post.

A technique I use to update all four sliding graphs (1x per sec) as well as the current, min and max figures for each graph is by combing all those values  into a single write and allowing the browser Javascript to parse, spilt and distribute them accordingly.

Another technique is to build in pauses between groups of writes. I use "lazy loading" of the web UI. It loads in 5 sections, each doing a chunk of work then scheduling the next chunk a short while later. The "short while" is of course calculated using the above graph! Yes, this makes the UI a little slow to load, but it doesn't crash!

Further, only the "active panel" is loaded on inital AsyncWebServer request. In my case it's the WiFi configuration panel (curently hidden behind the graph panel) . Each of the six icons at the top left of the lower panel will cease any socket activity to the current panel, hide it, show the new panel over the top and start any dynamic updating with socket writes for the new panel. Thus socket writes are only sent for the panel that is currently visible.

For further info feel free to contact me @ esparto8266 -at- gmail.com

Automatic unattended update of ESP8266 firmware using http server

One of the design goals for my home automation system was to be able to performs OTA (over-the-air) updates. The Arduino IDE makes this very easy, but when you have many devices deployed, it becomes a trifle tedious.

An easier way is to run and "update server" on your network and have each device check at boot time (or via an MQTT message) whether a newer version exists and if so, update itself automatically.

The central issue then is building the update server. The server software, language etc are unimportant:  it's the logic of how it responds to the request that matters. The excellent ESP8266httpUpdate class makes it very simple.

My own server is a "flow" in my NODE-RED controller on a raspberry Pi and - as you will see shortly - the logic is actually very simple.

What makes this all possible is the extra "headers" that the ESP8266HTTPUpdate class adds to the outgoing http request. They allow a great deal of control (should you need it) over how each device is updated.

x-ESP8266-STA-MAC
x-ESP8266-AP-MAC
x-ESP8266-free-space
x-ESP8266-sketch-size
x-ESP8266-sketch-md5
x-ESP8266-chip-size
x-ESP8266-sdk-version
x-ESP8266-mode set to either "spiffs" or "sketch"
x-ESP8266-version

This information allows the server to check that the upload will fit into the requesting device's memory, or provide a specific binary for a specific chip using the unique MAC address. Note also that you can also update a SPIFFS image (e.g. you inbuilt web pages / images scripts etc). The most important item is the version. This can be anything you decide.

The simple protocol and logic is this:

Device: "Hi, I'm firmware version a.b.c, do you have a newer version?"
Server: "Let me see...ah yes I have a.b.d - here it comes..."
If a newer version does exists the server just sends it as a load of binary data in the http reply. The ESP8266httpUpdate class does the tricky part of copying the binary into memory, changing the firmware boot address to the new code than (if requested) rebooting the device to run the new coded

If on the other hand there is no higher version, it replies with a http 304 error which effectively says: "I have nothing for you" and your code continues to run as normal.

Here's my NODE-RED update flow which shows the simplicity of it:

The salmon pink "nodes" are functions written in javascript - before delving into the actual code (which is minimal) we need to talk a little more about my setup:

On my server, I have a folder called /home/pi/trucFirmware which contains:

-rw-r--r-- 1 pi pi   65536 Nov  7 10:22 spiffs_0_4_3_1M.bin
-rw-r--r-- 1 pi pi 1028096 Nov  7 10:22 spiffs_0_4_3_4M.bin
-rw-r--r-- 1 pi pi  352192 Oct 17 13:37 truc_0_4_0.ino.d1_mini.bin
-rw-r--r-- 1 pi pi  344336 Oct 17 13:34 truc_0_4_0.ino.esp01s.bin
-rw-r--r-- 1 pi pi  352208 Oct 17 13:39 truc_0_4_0.ino.nodemcu.bin
-rw-r--r-- 1 pi pi  344528 Oct 17 13:35 truc_0_4_0.ino.sonoff_basic.bin
-rw-r--r-- 1 pi pi  347552 Oct 17 13:36 truc_0_4_0.ino.sonoff_sv.bin
-rw-r--r-- 1 pi pi  348144 Nov  9 17:46 truc_0_4_3.ino.sonoff_basic.bin

I maintain a separate binary for each hardware type (from a single source file with a few #defines) and when a new release is ready I use the Arduino IDE "sketch/Export compiled Binary" menu command for each target device.

Note that even though there are 5 different hardware types, there are only two SPIFFS binaries: a 1M and a 4M version - constructed with the mkspiffs tool - since all the devices have either 1M or 4M flash.

Once you have the mkspiffs tool, building the spiffs is simple. I have a one-line batch file for the 1M version which takes the version number as a parameter (%1)

mkspiffs -c data/ spiffs_%1_1M.bin

and another for the 4M version:

mkspiffs -p 256 -b 8192 -s 0x0FB000 -c data/ spiffs_%1_4M.bin

I then copy all the compile binaries and the SPIFFS .bin files over to /home/pi/trucFirmware

The whole process then, goes like this:

The sketch contains the following code:

#include<esp8266httpupdate.h>
...
#define TRUC_VERSION "0_4_99"
// THIS_DEVICE is set earlier depending on various compile-time defines 
// which eventually define the hw type, e.g. #define THIS_DEVICE "d1_mini"
const char * updateUrl="http://192.168.1.4:1880/update/"THIS_DEVICE;
// this is my raspberry Pi server, the 1880 is the default NODE-RED port
// /update is the url I chose for the server to "listen" for, followed by the device type
...
bool ICACHE_FLASH_ATTR  _actualUpdate(bool sketch=false){
    String msg;
    t_httpUpdate_return ret;
     
    ESPhttpUpdate.rebootOnUpdate(false);
    if(sketch){
      msg="sketch";
      ret=ESPhttpUpdate.update(updateUrl,TRUC_VERSION);     // **************** This is the line that "does the business"   
    }
    else {
      msg="SPIFFS";
      ret=ESPhttpUpdate.updateSpiffs(updateUrl,CSTR(E.getSPIFFSVersion()));
    }
    if(ret!=HTTP_UPDATE_NO_UPDATES){
      if(ret==HTTP_UPDATE_OK){
        msg.concat(" upgraded");
        Esparto.publish("update",CSTR(msg));
        return true;
        }
      else {
        if(ret==HTTP_UPDATE_FAILED){
          msg.concat(" upgrade FAILED code:");
          msg.concat(ESPhttpUpdate.getLastError());
          msg.concat(" reason:");
          msg.concat(CSTR(ESPhttpUpdate.getLastErrorString()));
          Esparto.publish("update",CSTR(msg));
          }
        }
      }
  return false;
}

When I'm ready, I call _actualUpdate(true); and the action then moves to the server:

The first node in the diagram above "listens" for an http request to url http://192.168.1.4:1880/update with the device type appended. It passes this to "Construct search path" function node which has the following javascript code:

msg.type=msg.req.params.type;
var h=msg.req.headers;
msg.version=h["x-esp8266-version"];

msg.mode=h["x-esp8266-mode"];
if(msg.mode=="sketch"){
    msg.payload="/home/pi/trucFirmware/*.ino."+msg.type+".bin";
}
else {
    var sz=h['x-esp8266-chip-size'];
    msg.payload="/home/pi/trucFirmware/spiffs_*_"+(sz/1048576)+"M.bin";
}
return msg;

This just sets up the appropriate path with wildcard for the sys function which follows, which simply runs ls - r <msg.payload>

The output is then fed to the "Compare versions" function node:

var f=msg.payload.split("\n")[0];
msg.filename=f;

if(msg.mode=="sketch"){
    f=f.replace("/home/pi/trucFirmware/truc_","");
    f=f.replace(".ino."+msg.type+".bin","");
}
else {
    f=f.replace("/home/pi/trucFirmware/spiffs_","");
    f=f.replace(/_\dM\.bin/,"");
}

if(msg.version<f){
    node.warn("upgrade required");
    node.warn("will return "+msg.filename);
    return msg;
}
node.warn("no upgrade");
msg.statusCode=304;
msg.payload=[];

return msg;

The switch node then ensures that either the 304 "no update needed" message is sent or the actual new binary is returned and sent back to the device.

There you have it: automatic unattended updates. All you need to do is copy the new binaries to your sever and - voila! - the rest "just happens".  It certainly makes my life easier with 8x ESP-01S, 6x Wemos D1, 4x Sonoff Basic 10x Sonoff S20, 2x Sonoff SV and a NodeMCU - 31 devices in all to update when I make a code change.

Doing is this way is almost effortless.