Esparto v3.3 released at last

AT last! A "quick fix " to patch up a problem in someone else's library has just ended up being a 2-month ground-up rewrite. I am so fed up with the whole thing, I hope you lot all like it enough to cheer me up: Watch the video then go and have a play with it. Esparto v3.3 released at last. Still some documentation being "backfilled" but the code should be ok and with 61 examples, you should be fine....

Esparto 3.3v feature preview

Esparto v3.2 soon to be released: sneak preview of new features

Esparto v3.2 soon to be released: sneak preview of new features

Click to Watch

Some "upgrades" weren't published, they escaped. A week of woe.

I have just come through a programming problem so frustrating, so annoying, so intractable that not only has is it taken me well over a week of shouting at the screen and hair-pulling, but also the only way to recover is the catharsis of sharing my nightmare…and a very stiff drink.

There are many lessons to be learned herein: it’s going to long, it will be very detailed and all-too-recently painful, but I think it will be worth it if you can hang on in there.

Executive summary: code and library versions can be important (and sometimes harmful!) …and “latest” is not always “best”

That nightmare started shortly after I released version 3.1 of Esparto… a couple of folk noticed problems. One in particular was using the much-more up-to-date 2.5.0 core of Arduino ESP8266 whereas all my testing, development and release had been with 2.4.2…

I felt churlish telling him it wasn’t supported, sorry -  since I am the one behind the times and really should be up-to-date myself, I always advise others to do so! I therefore set up my “upstairs” machine with Arduino IDE 1.8.8 and ESP8266 core 2.5.0. to try to a) verify the problem and b) work out a fix. 

That's when the “fun” began. Sure enough, the problem he described was happening exactly the same to me “upstairs” while all was fine downstairs…before I continue, a touch of background is necessary:

Esparto relies fundamentally on the ESPAsyncWebServer library which in turn relies upon the same author’s ESPAsyncTCP library. For good measure (well, Amazon Echo uPNP discovery, actually) is also uses ESPAsyncUDP. Note the common thread: “async”. Any / all / every piece of serious code on the ESP8266 needs to be async...or it’s just a toy, but that’s a whole other book. Back to my upstairs machine: unloved and unused for many a moon, it lacked those essential libraries, so I installed them, automatically getting the latest version – of course. The stage is now set for Act I of the tragedy.

More background: Esparto has the fancy “real-time” GPIO lights on its web UI, comme ca:

These were controlled using websockets (as was the whole of the web user interface [web UI]) and after much similar recent wrestling, had led me to insert a ton of code to limit or “throttle” the amount / rate of websocket messages going to the web UI. For example when a very “busy” GPIO might be changing state hundreds of times a second, neither the ESP8266, nor any browser, nor indeed your eyes can keep up with that so it makes sense to limit it, as failing to do so has fairly rapid and disastrous consequences, namely the ESP8266 runs out of free heap and crashes.

Lesson 1 is that if there is one key technique behind successful ESP8266 programming, it is heap management. And being async! Two. Two main techniques. But heap management it absolutely vital since the ESP8266 has so little and some libraries are very cavalier about how they use it. At core 2.3.0 I couldn’t compile the whole of Esparto since the core libraries didn’t leave enough free heap! In the early days, I also experienced “random” crashes which took me a long time to put down to sudden massive heap exhaustion (6k / 8k / 10k within seconds etc) within the ESPAsyncWebServer library, with a) no warning and b) no way to check ahead of time when the “danger zone” was imminent…

 A quick bit of “long story short” here which has ironic repercussions to follow later. About 9 months ago, I “had words” with the developer, even going as far as to provide a graph showing the exact problem once I’d spent literally weeks tracking it down…in essence there was no internal check before queuing a websocket write – neither that there was sufficient heap nor whether a large queue was building because they were coming in faster than they could be despatched…hence when either / both those things occurred* the internal queue grew out of control and used all the heap, crashing the system.

* = over 19x per second. Yep, a massive 19. Any more and the free heap drops like a rock, irrecoverably. So three pins doing 6 transitions each (e.g. flashing on/off 3x per sec) is fine. Add a fourth, and make it change more than 1x per second and you are in a terminal one-way nose dive.

The complete lack of any way to detect / prevent this led to my fairly strong words with the developer.  I ended up doing a whole ton of work to calculate this magic number, then another whole ton of work to “throttle” the GPIO and/or any other control signals to /from the webUI. Anyway, that all worked. So I carried on and released it.

Back upstairs, I am wondering how the hell only ever 8 items get added to the “run” tab dropdown box. Wiresharking the conversation, I can see I’m actually only sending 8 even though the code calls the sending function 22 times. Coincidentally I’m only sending 8 GPIO updates too even though dozens are physically occurring. Very suspicious. Another “long story short” because it actually only took me an hour or so to find the problem and my jaw dropped open in horror when I did.

The websocket lib (with which I was far too familiar due to the problems above) has changed quite a lot. At the point of socket write it effectively says

 if messageQueue.length() < MAX_QUEUE_LENGTH, add message to queue. 

That’s it. No error code if it doesn’t get queued. No fall-back and requeue. No warning message, not even at debug level! No way to discover, let alone change the hard-coded value of MAX_QUEUE_LENGTH which is…you guessed it: 8. Nope, anything more than 8 outstanding requests just get silently ignored and thrown away. I rushed to the author's webpage – no mention WHATSOEVER of any of this in the documentation. Now what goddamn use is that to man or beast?

A caveat before I continue. Those libraries are close to works of art, and I had (note past tense) nothing but admiration and respect for the author (who does a ton of other seriously clever stuff too) and Esparto couldn’t work without them. Thus it not only pains me, but I actually find it quite difficult to come out and say just quite how shitty this “fix” is. I put it down to either a) the author having had a seriously bad night on the beer before “fixing” it or b) handing it off to a far, far, inferior programmer to “do a quick fix”. It is honestly so out of character, I simply don’t understand it. It stands out like a sore thumb compared with the rest of his highly impressive code.

Irrespective, it renders that implementation of aync websockets almost totally useless for any practical purposes. 19 per sec was pretty shabby already: knocking it back to a carved-in-stone silently-discarding 8 is unforgivable. Honestly, if the author had asked me to come up with a fix myself which must be the worst possible solution on every front, I doubt if I could have managed anything so appallingly slack-arsed as what now in there. If I were paying someone as a programmer and they wiped this off their shoe, they’d now be serving coffee at Starbucks. But it doesn’t run out of heap anymore. It is the equivalent of fixing your car’s brake problems by removing the distributor cap.

So...

After the swearing had stopped (I’m lying, I still sometimes swear at the insult of it, and we are two weeks on…). I remembered that Machiavelli said we must live in the world the way it is, not the way it should be: I needed another solution, and fast.

The Solution, Part 1: Out with the old

Websockets are bi-directional and I used them to react to UI events such as clicking a button to reboot, or change tabs etc, so immediately that all had to change. To be honest the JavaScript code was a bit “lace-curtain” so I smiled through gritted teeth as I was forced to tidy it all up and make classic “ajax” calls instead. Of course, I then had to re-jig a lot of the webserver handling. While I’m doing so, I may as well do the JSON part properly…

Everything JSON I looked at was massively top-heavy, so I wrote my own JSON encoder to handle a few specific cases of sending JSON data back to the UI from std::vectors / std::maps which is how a lot of it is internally held.

After the first few days then, with an almost totally new 600-lines of ajax-compatible JavaScript , completely reworked webserver functionality, websockets ripped out root and branch and a custom JSON encoder written…phew…I was ready to stitch in the “push” side of the bidirectionality, the little-known “Server-Sent Events” or SSE. Luckily, it is already included in the ESPAsyncWebServer library. I just prayed some ******* idiot hadn’t “throttled” it to a hardcoded 8-at-a-time like they’d done to the websockets! A brief look at the code showed it appeared thankfully unfettered in that respect. Oh how much too soon I smiled…

First, the elephant in the room. Why had I been working with such an old library? Simply because there is no easy way to find out it had changed. But 9months+??? Don’t shout yet, yes I know a lot of Arduino libraries come up on that bloody annoying “some of your boards…” popup when you start the Arduino IDE…the libs I’m talking about aren’t included. Why not, you will have to ask the author. Technically, I know the reason: their metadata doesn’t match exactly the format that Arduino uses to identify libraries when it trawls github. Exactly why not, you will have to ask the author. Yes, I know that’s the second time I said it. I’m not happy with him right now. And you’re probably only just over halfway through the “fun”. I had consulted the docs dozens of times in that 9 months. Nothing on the site to show a new release available. Docs unchanged, even to the trained eye. Why would I think it needed updating? At best it would have just shifted this problem back 7-8months.

Ironically (remember that far back) when you look at what changed…much of it was obviously as a result of my graph…As the Chinese say: “Be careful what you wish for”.

The Solution, Part 2: In with the new

Very new. I’d never heard of SSE before, so a bit of reading and experimentation was needed. In reasonably short order, I had a prototype working. The basic concept was there, now to the leg-work of replacing all the “push” stuff which was mostly the GPIO pretty flashing lights, but due to heap constraints, several of the “live” tabs have to be “fed” asynchronously and piecemeal too.

Again I’m smiling through gritted teeth because I am forced unwillingly to make my code a lot better and more robust. At last, I am able to implement the “persistence of vision” optimisation that allows me to “throttle” the GPIO light to a rate that is just slightly faster than the human eye can detect…no point throwing 100 events per sec (if the interface can handle it*) at the webUi of you can only spot the difference in 24 “frames” per sec, like a movie. 

Actually it’s 23.80 – doing some simple maths will explain why – all Esparto timing is in milliseconds…and all ESPArto “maths” uses integer arithmetic. Floating point bloats the bin incredibly and it overkill for a couple of small calculations. Giving a “delay” to a timer requires telling it how many milliseconds to wait. For example, 500mS makes it run 2x per second (1000/500). Q: What it is 1000/24 ? A: 41.6666 if using floating-point, which I’m not, so it’s actually 41 from a truncated integer divide. Now 1000/41 = 24.39 per second, slightly more than I need and - if I accept that 23.809523 is close enough – which I do, then the divisor becomes 42. I need say no more. On with the show.

The plan is that when someone is viewing the UI, I will “scan” all GPIOs @ 24ish times per second and send a list of any that have changed to the JavaScript that flips them red/green. Obviously, I don’t want that overhead if no-one is watching so I have to be able to turn on off the “cinema projector” when the theatre is empty, and restart it instantly as soon as another customer arrives.

Also, I wanted – at last – to overcome the cheap’n’cheesy limit that has long existed on only one web viewer. Slight trick with that is that different tabs open for different viewers need different “live view” push data as well as the GPIO/ MQTT status etc that all active viewers need. Similarly any "watching" the same tab e.g. the graphs all need to sse the graphs updte at the same time. Managing that required a fair deal of extra code and rewrite to the webUI. Since I was having to “gut” the whole code body, I thought I may as well take this chance to get it good once and for all. All I finally need then is an easy way to detect incoming and outgoing clients.

Ah.

With a websocket you get “onConnect” and “onDisconnect” events, which make it easy. SSE is one-way only "push" technology. It doesn’t really care or stop working if it is “shouting at an empty room” so while obviously you get a “connect” message; if the client dies or goes away, you never get a "disconnect". In fact, you don't get told at all.

This is a serious pain, but not fatal, on the JS side, the EventSource reconnects automatically after a user-configurable “timeout” and provides the messageID of the last message received, so you can tell the difference between a new client (who needs the whole page) and an existing reconnect who just needs continued live push data to whatever “tab” he’s on. With some hoop-jumping and a home-brewed clientID handshake / special HTTP header in the Ajax request / configurable periodic keep-alive “ping” – I can keep track of who’s connected …and close down the hot-running cine projector when all the clients timeout (strictly, after – on average – 1.5x the “ping” time…but anyway…)

Phew. A lot of rework, and a whole new technology, but a better “product” results.

Until that is, you realise two things:

1: Haunting echoes: “…throwing 100 events per sec (if the interface can handle it*)”. Asterisk = It can’t. To be fair, it can handle a helluva lot more than 8 at a time! Due to the way I now incorporate all changed GPIOs into a single “frame” it copes reasonably well with the chosen 24ish fps. So If I have five or six pins all changing quite rapidly that’s the equivalent of – say – 120 GPIO events / sec, shifting the bottleneck now to the JS in the browser which usually can’t cope that fast…but hey: the eye can’t tell. Also, anything throwing in hundreds per sec is still only going to get sampled @ 24fps so while it may be a way removed from the physical truth, again your eye can’t tell. At least we don’t get sudden huge heap losses. Until…

2: This:

Those downward spikes? The 4kB+ sudden "you-ain’t-go-it-no-more" bites out of the free heap that happen when the client side automatically reconnects. Since there is no “on disconnect” or equivalent, it takes some time of throwing 24fps at the library before it realises the client has gone and breaks the connection. In the meanwhile, it queues up the requests into a rapidly-growing heap-chomping MCU-crashing shitstorm just like... just like those old websockets. Talk about frying pan and fire! 

A week down the line and all I have is the same problem in a whole ton of new untested code – but it’s worse, again there is no way to predict when it will occur. Nor is there any way that the library can / will tell you.

It happens when the underlying network code decides to break the connection, i.e. it’s a) too low down in the “stack” of code for me to even want to go there b) probably beyond my knowledge c) it really is pretty random. Upstairs it dies it every couple of seconds, down here it can go two minutes or more without a blink. Only two things mitigate it: 1) it is usually* less than the chunk the websockets used to grab. 2) The EventSource technology auto-reconnects “for free” – so if you can stay alive through the big chunk – albeit with a dead UI – then it pings back pretty quickly once the connection is remade. It takes 3 -5 seconds for that to happen though so the user experience is really choppy / stuttery / horrible.

*usually. It is usually 4kish but can be up to 10k in a couple of chunks and you can’t prevent it from conflicting with another request e.g. an Alexa command or a 2^nd viewer using a REST call…either of which will take it further down the hill to never-land on a one-way journey to the power-recycle switch. Something has to be done.

A quick dive into the code shows that it has no checking, no low/ heap prevention – ironically it is just exactly as bad the old websocket code before the “fix”, but not yet quite as bad as the new. This time, I have to prevent the same (very low) quality of “fix” taking out my final chance if I notify the author and he “fixes” it up the same way.

I need another solution. The only thing that can be detected is a sudden loss of about 4kb heap in a very distinctive pattern. If  only I can spot that from MY side of the fence and then hit the “pause” button on the cine projector until at least most of it has come back up once the auto reconnect is complete…I might be back in business.

You are going to think my solution was a bad choice and now – more than a week later – so do I. At the time it seemed like an easy option(!): I built a” heap heuristics” module deep into the scheduler which did a variety of measurements at very high speed behind the scenes…instantaneous loss between calls, rolling average loss (cyclical buffer over last N <configurable> calls) percentage difference from a) last call b) rolling average…ultimately I ended up with 2^nd differential i.e. difference between last two differences to try to narrow down the “acceleration” of the sudden 45 degree downward slope. Hmmm, really?

I spent days trying to work out where best to put the code, as the actual values vary greatly depending on where in the call stack it sits…etc. Should it run on all jobs (with a view to preventing other future problems) or just the GPIO "NetFlix" job? Even if its just on NetFlix, do I need to keep a portion running to keep the overall rolling average accurate...or do I just use the NetFlix rolling average? Is the "granularity" of last 10 measurements enough to "see" the nose-dive in time? Will I need more or will only 5 do it?  And what of the trigger values? Even though they will (of course) be configurable - I know already they wont be the same at the next version! 

The number of concurrent viewers change the trigger values. Bad network connectivity changes them. More pins change them. Every ***ing thing changes them- to the point where I could just about catch the node-dive about ¼ way down the slope, with many false positives but – fatally – the occasional false negative, i.e. a missed - and potentially fatal -“trough”. So even ofr all the configurability and complexity, it still wasn't working.

Despite the odds, I persevered through four or five days getting ever more frustrated and dispirited trying every permutation of placement, parameters, method, technique – each variant requiring major rework…I wsa paraying a certain combination would hit a "sweet spot" where I could "predict" every imminent nosedive. All the time I’m swearing and shouting “why can’t I just see the "$!%!"£$%^ing !"£$%^  message Queue length?????????????” 

Because if I could, I'd be able to see when it starts getting longer than X and immediately "back off" sending! Simples!

Not quite, because there is no way to pause to let it “drain” since the only reason it starts to accelerate in the first place is because the underlying network has already broken the link, so nothing will ever “consume” the queue contents. It took me a few days before that crushing realisation hit me. All I can do is wait until the library spots the network problem and then discards the whole queue in one fell swoop, giving that sharp upward recovery seen on the graph, which takes 3-5 seconds. During whihc time- of course - the UI is "frozen". I'm sent stuttering back to square one.

But wait…when the library does notice the network problem and “chops” the queue…as mentioned earlier, the JS notices pretty quickly and reconnects automatically. If I set that reconnection time very low (1/2 sec say) and find a way to make the lib do that cleanup sooner, by -DING! Spark of genius alert! – simply arbitrarily force-closing the connection, even while it’s still “good”…

It might all be a bit “dirty” crashing and reopening the connection all the time, but it will work. After all, upstairs with the weaker signal it is already naturally doing it a lot anyway. All I’m doing is giving nature a gentle early nudge! All I need is to get the queue length from the lib. Sounds easy.

I had been avoiding relying on someone else’s code and goodwill to accept my “fix” – but it’s the only way. How do I know? Because I put in a quite easy fix requiring only about 2 small changes and a new 6 or 7 line routine called getAvgMQL: “get average Message Queue Length”. A slight wrinkle is that the AsyncEventSource has up to n clients and it is the client that has the message Queue, so I have to average all the clients to get an overall figure. By experimentation, a queue length of 15 seems a good balance between “choppiness” and a harbinger of impending doom. Look at the following screenshot: running constantly for over 7 hrs now with a lot of pin activity and a maximum “reconnect trough” of about 2.5k That is the longest it’s been up for about 10 days.

The vertical bars are added when I do the forced close. As you can see, a couple coincide with what might be the start of a "trough". Some don’t: they are either just natural “busyness” or plain and simple false positives. Since forcing a close has very little overall effect on the smoothness, and is actually now part of the "normal" mechanism, "Frankly, my dear, I don't give a damn!"

What I do care is that the behaviour is – at last – predictable and controllable as well as being a reasonably good user experience for quite a heavy load:

Esparto.flashPattern("   ... --- ...",250,LED4);
Esparto.flashPWM(1400,10,LED5);
Esparto.flashLED(500,LED6);

D4 is toggling at a rate of 125mS or 8x per second. D8(LED4) is flashing S-O-S in morse code, i.e. … --- … on a timebase of 250ms i.e. each dot / dash is about 1/2 - 1/4 sec long sec on a continuous loop. D7(LED5) is a 10% duty cycle blip in 1.4sec period and D6 (LED6) is a simple 2x per second “square wave” so when the “peaks” line up, you are getting 12 transitions per sec from 4 different pins and the UI appears pretty responsive. Being honest, it can’t quite “do” the three short Morse blips, but now that it is stable and predictable, I can try upping the frame rate as well as well as tweaking down the Q threshold to get the optimum result.

With hindsight, If I had chosen the ”investigate / reverse engineer / fix / update” someone else’s highly complex library” route first, I’d have saved myself well over a week and a lot of hair.

Lesson 2: Don’t shy away from tough decisions if - in the long run – they are the best.

Lesson 3: Check your library versions regularly even if there is no sign they may have changed!

Lesson 4: Not all “upgrades” are good ones. For example, core 2.5.0 adds over 30k to the bin file for Esparto. Thirty K!!! It wouldn’t be so bad, but Esparto is pretty big already – in core 2.4.2 its about 430k so 2.5.0 puts that up to 450k…kinda right on the limit of what can be OTA’d into a 1MB device with the minimum SPIFFS for the webUI… Worse there is less free heap to start with, and what brought us here? Correct: low heap management - 2.5.0 just makes it worse. Right now , for me 2.5.0 is a huge retrograde step – and if that weren’t bad enough, there’s a real killer:

Esparto will simply not serve up its web page in 2.5.0 – its like it is running in glue / treacle. The exact same line-for-line code that works (now!) perfectly under 2.4.2 cannot get to first base and – for now – I’m all done with fixing other peoples’ bad upgrades. I’ll have to find a way around it a some point.

On the plus side: The web code is much more robust, and more easily expandabel if / when new features come along. The same is true of the core code and a few quite useful new features have been added:

"repeatWhile" function. Repeat function f1 every n mSec while f2 returns true and call f3after f2 returns zero to stop the repeat. This is perfect for “worker” threads “chunking up” a job in the background to conserve heap. F1 repeatedly performs a “chunk”, adjusting a counter / pointer F2 return the counter / pointer while its non-zero (still work to do) f1 keeps repeating and once all the data has been done, f3 can clean up / reset counters for next time etc. All of this is “interleaved” with other tasks, minimising heap loss and streamlining overall throughput.

Its what my "netFlix" function uses, and yes, I did call it that:

	
if(!netFlix){		
    Serial.printf("Start strobing @ %d (=%d.%dfps)\n",CII(ESPARTO_FRAME_RATE),1000/CII(ESPARTO_FRAME_RATE),((1000%CII(ESPARTO_FRAME_RATE)*100)/CII(ESPARTO_FRAME_RATE)));
    netFlix=repeatWhile([](){ return tab::nViewers(); },CII(ESPARTO_FRAME_RATE),
		[](){
		    vector<string> cp;
		    ESPARTO_CFG_MAP cPin;
		    int n=0;
		    for(int i=0;i<ESPARTO_MAX_PIN;i++){
			if(_spPins[i].dirty) cp.push_back(_light(i));					
			_spPins[i].dirty=false;				
		    }
		    if(cp.size()) _spoolBcast(jNamedArray("gpio",cp));
		},
		[](){ netFlix=0;Serial.printf("LAST PICTURE SHOW\n"); }
    ,ESPARTO_SRC_WEB,"flix");
}

While there are any UI viewers, repeat the "send changed pin" @ ESPARTO_FRAME_RATE. Oncce all viewers have gone, reset initial conditions so next viewer restarts.

Very simple but incredibly useful is vBar. It simply draws a vertical bar of a given color (default = red) across all graphs asynchronously, i.e. when you want it to happen, to show when a particular event has occurred, in context. I could not have fixed this problem without it!

Thrown in are web basic auth and the ability to change MQTT parameters through the UI and Amazon Echo v3 compatibility among several others minor changes.

Esparto v3.2 will be (code-wise) very different from v3.1 (almost a re-write!) even though it will look almost 90% the same! It will be out as soon as I’ve had a day off and got some of the new documentation(yuk) fixed up and new example code written.

Videos of Esparto v3 in action

Video no 1: Overview and features

Video no 2: The code behind the overview

Esparto v3 finally released!

Blimey, it's finally done. Esparto v3.0.0 released! (subject to some documentation additions and tweaks) Some fun facts:
281 files in 72 folders totalling 11.8MB
Main code is 16 source files totalling 3742 lines of code
Includes 47 example sketches totalling 3704 lines of code..I'm happy to call it 7500 lines of code...

https://github.com/philbowles/esparto

Esparto V3.0 nears release

Esparto v3.0 will soon be released: here's a little taster of the pin types available - v3.0 allows you to add them at runtime with no programing!

How real is time?

More specifically, how fast is “real-time”? The very phrase “real-time” is a deception – nothing in the physical world happens in “real-time” except time itself. Even if it’s only a few nanoseconds, the voltage on an input pin takes time to rise. How quickly after you press the button should the LED come on before you call the cause and its subsequent action to be “real-time”? Is it sufficient for it to be undetectable to the human eye even though that might be an age in computer terms?


Real-time is therefore a subjective measure, by anyone’s standards. The difference between the cause and the action is known as “Latency”. Take the case of an interrupt. The ESP8266 takes a finite number of clock cycles to recognise an interrupt and that’s after the inherent latency of the voltage rise time from the actual initiating real world-event. Already we have some latency in the system. How much? At its standard 80MHz clock rate, typical its 10 us - 80us but can sometimes be as long as 300us.


That is only the start of the story, as an interrupt on its own is useless without some user code to then do something with it: The Interrupt Service Routine (ISR).Any half decent programmer used to dealing with interrupts will tell you that you have to keep the ISR as short as possible. It’s to a) reduce the latency and b) reduce the opportunity for cross-process interference in the absence of OS-enforced isolation. It’s common to just set a shared flag (often in global scope) to signify “something has happened” and then have the main loop check the flag as often as possible. The ISR has latency, the main loop check adds more.


Finally, doing what’s actually needed e.g. lighting an LED (which of course has its own output voltage rise-time latency to add…) will add plenty more. The point here is this: there will always be a measurable delay between the cause and the subsequent outcome.


The real question then becomes “what is the maximum acceptable delay for this application before we can consider it to be real time”. I don’t know if there’s an “official” answer to that. Personally, I happily run my house lights and alarms with my home-grown firmware “Esparto” and you should too. I would also happily run your life-support system on it, but I wouldn’t necessarily want mine connected to it.


With Esparto, there is an extra layer in the GPIO pin handling, as it has to react to each and every designated input pin so that it can maintain the pretty flashing lights when the web UI is open. Obviously I have tried to keep it as lean and mean as possible but it has all the overhead of a websocket and then the network transmission time to the viewing device.


On top of even all that, there can be more latency added by the internal serialiser / scheduler deep in the code which can occur when lots of other stuff is happening “at the same time”. There’ll be a lot more on this later but for now take it from me that the pretty flashing lights are for information and entertainment purposes rather than precise measurement. There can occasionally be a tiny (but visible) lag between the built-in LED going on and the web UI pin lighting up. The faster you flash, the worse it gets. The more other pins you have active, the worse they all get.


To the casual observer, its real-time. To me, its real-time. To the purists, nerds and boffins, it may be far removed from what they define as real-time. In my own odd ways I’m probably a bit of all three of the last group, so I know I’m slightly deluding myself when I say “To me, its real-time” – but it’s good enough.


In the world of IOT that we inhabit, there are very few things that need to be "real-time". Interrupt handling is complex and difficult to get right, even for experienced programmers. Novices should stick to polling - the rest of your code will be taking the Lion's share of the processor and any "loss" of latency will not be noticed - really, it won't!

IF you are an expert and you know - for example through some specific hardware timing issues - that you need interrupts, then by all means use them. That excludes 95% of the folk I encounter in groups and forums, so for the rest of us, forget interrupts, you don't need them. Either stick to simple polling - or use a library like Esparto that has already done all the hard work for you

Esparto v2.0 finally released!

Yes, it's "out of the door" at last. After some final "stress" testing and the creation / testing of 32 (yes, thirty-two) example programs, v2.0 finally goes public.

Intervening health problems added 8 months to the date(!) but at last I can now relax somewhat- with a well-earned beer.

Get Esparto v2.0  here at github 

Esparto V2 almost ready! The new web UI part 6

The lower panel(s)

Run

This is literally where the action happens. Esparto is designed so that commands can come from several sources:

• MQTT topics
• the web UI
• from within the app itself using the "invoke" functio

The first is one of the most common forms of communications between IOT "things" in home automation. You obviously need access to an MQTT "broker" (fancy name for "server", really) either your own or public one. At this very moment, there is no way to enter a username and password - except manually in your own code - so until I get that fixed... your own broker is the best bet. I use mosquitto (which can be found Mosquitto.org download page) on a raspberry Pi.

But don't despair if none of those options are available, you can still do a lot with Esparto through its own UI, or from your own code. Want to change GPIO0 (D3)'s debounce value for example? Either call Esparto.invoke("cmd/pin/cfg/0/15"); or come to this screen, select the cfg option form the dropdown menu, add the /0 to the Topic line and type 15 into the payload field. Then hit "Simulate MQTT" - Esparto's internals that actually "do the business" are called with exactly the same message as they would have received from a genuine MQTT server. Simples.

As commands are received and actioned (from any source) the stats are dynamically updated. Soon - and certainly before release - I shall add "Alexa" as a source because Esparto is fully Alexa-compatible by pretending to be a Belkin WeMo when asked in the right way. All it can do is "turn on "<your device> or "turn off" the same. That's as much as a lot of devices do, anyway.

The "all" source is special one built in to Esparto allowing you to send a single command to all Esparto device on your 'net, as well as addressing each one indvidually by its device name. For example - while not necessarily advised - "all/cmd/reboot" will do exactly what you think it would.

Users can to subscribe to any topic they choose - including # wildcards - when called back in onMQTTConnect (see previous post in the series for a simple example).

In the demo, the user has subscribed to a wildcard topic like so:

  Esparto.subscribe("wild/#",[](vector<string> vs){ 
        string suit=vs.front();
        Serial.printf("Wilcard handler suit is , card is %s\n",(CSTR(suit)),CSTR(vs.back()));
        if(suit=="hearts" || suit=="clubs" || suit=="diamonds" || suit=="spades"){
          Serial.printf("You chose the %s of %s\n",CSTR(vs.back()),CSTR(suit));
        }
        else Serial.printf("Invalid suit %s\n",CSTR(suit));
    },"cards"); 

(They also included  (not shown) a simpler topic "flash" which they wrote to call the same code as Alexa commands call. So when Alexa is told to "turn on testbed" it has the same effect as MQTT command "testbed/flash/1" for on  and ...0 for off. )

He/she has chosen to only allow this wildcard topic from another made-up source "cards"  so "testbed/wild/party" won't work, but" cards/wild/animal "will.

And therein lies a slight oddity - and a caution. Esparto cannot predict the billions of permutations that come after ...wild/ - only the user can decide that. Hence for the system to work Esparto not only has to accept anything of that form, it also has to add it to the above table and count it and that has consequences.

The user code in the demo rejects any subtopic except (rather suitably) hearts, clubs, diamonds and spades and would like the payload to be a card from 2 to 10 or J K Q A. Lazily, it doesn't actually validate the payload, but then all it does is parrot ack to you what you send it, so no harm done in this contrived case. In the real world such appalling coding (I should know, I wrote it - deliberately to bring out these points of course) will almost certainly lead to a crash if unexpected, unvalidated input and/or gibberish is fed to any wildcard topic - so be careful. Trust no-one, and validate everything to within an inch of its short life.

Another consequence is that the more wild rubbish you send that Esparto is duty-bound to accept, the longer that list will get, the slower the UI will become until finally Esparto's self-protection mechanisms will cut in and reject  everything from all sources until some memory is freed somehow. This will certainly cause erratic behaviour and possible meltdown at the nuclear power plant, so don't do it. If no memory can get freed, then your Esparto app will die a slow lingering death till you want to reboot it. When you do, remember it's your fault, not Esparto's. 

While this a shorter section than some others, this pane is probably one of the most useful of Esparto's many features. There is neither the space nor the time to go into detail about each of the commands and new ones are being thought of as I type...

You may find that all the ...dump... options are missing from the final release, as they are 90% used in debugging. It will be done in such a way that even average programmers will be able to hack into the code and just turn on a #define and recompile. They do steal heap though, whic is already in short supply so don't say you haven't been warned. On the other hand if I can trim some fat from elsewhere during my final code tidy, I might leave some/all of them in. Invoke("cmd/watch/this/space"), you might say.

Others are just fun to play with: .../pin/flash/... , .../pin/pwm/...  and  .../pin/pattern for example/...

Download the release when it's out "soon", edit in your own SSID/password/device name and give all the topics/commands a try - it's what Esparto is for.  Or wait 3 minutes for it to give up and go into AP mode then configure it with your phone.

And enjoy it!

Esparto V2 almost ready! The new web UI part 5

The lower panel(s)

Config

Now we start to see the real power of Esparto coming out. It has a configuration system where name/value pairs are automatically saved to SPIFFS (the ESP8266 Flash file system) as soon as they change and persist into the next reboot, i.e become permanent. Well, until the next factory reset, at least.

The demo code has a Latching push button on GPIO0 (Arduino digital pin D3) and a very "noisy" and sensitive  sound sensor (i.e. a high number of thousand IOs per second at the slightest cough) on GPIO12 (D6). It also configures the BUILTIN_LED for output. On a Wemos D1 mini that the demo was built on, this is on GPIO2 (D4). The hardware setup looks like this:

void setupHardware(){

  Esparto.Output(BUILTIN_LED,LOW,HIGH); // start with LED OFF                   

  Esparto.Latching(PUSHBUTTON,INPUT,10,buttonPress); // 10ms of debouncing

  Esparto.Raw(D6,INPUT,[](int s){ Serial.println("Do nothing"); });

  Esparto.throttlePin(D6,19);

}

I trust your first taste of the "esparto Way" wasn't too shocking or difficult? setupHardware() is equivalent to the standard Arduino-style setup() and you do the same kind of thing here as you would there - almost. You just do it Esparto-stylee - so for instance, no WiFi.begin and delay loops* Esparto is already connecting to your SSID "in the background" to speed things up.

Also you won't see pinMode calls: Esparto knows what mode to set automatically from the type of Esparto SmartPin you define. So really all we have is one line per I/O device, and often that's all you will need. The only "odd" or "tricky" thing is the throttlePin call. We'll get to the strange syntax in a minute, but first, what exactly is "throttling" and why do we need it here?

It is described in great detail in part 3 of this series, so if you want to know more read up on that first, but for now a simple one liner is that the sound sensors fires far more data than any tiny device can easily cope with -Esparto being no exception - so we have to slow it down, or "throttle" it. D6 is our noisy pin, so we tell Esparto only to allow through 19 of the thousands of 1s and 0s per second.

The reason this is such a low figure is explained in the earlier article. Your LED will still flash vaguely in time with your bangin' house or lounge jazz tracks...ish. Now to that weird syntax...

C++ Lambda functions:

See what? If you don't know about these already, ask Mr Google about them because you will quickly come to love them as much as I do. They are particularly good for callbacks and a lot of your code needs to be in callbacks so now is a good time to learn how to use them. If you are already frightened, fear not: you don't have to use them, the old-fashioned way still works. I will show you what that would look like in a moment and I'm sure you will soon be seeing the benefits of the new-fangled way.

What we want is for Esparto to tell us when pin D6 changes and what is has just changed to: a 0 or a 1. So we need to give Esparto a function that returns nothing (void) and takes a single int parameter, which holds the new state when the pin changed. Ordinarily we'd write:

void namedFunction(int s){

Serial.println("Do nothing");

}

and then our old-fashioned way would be:

Esparto.Raw(D6,INPUT,namedFunction);

But:

• It's more typing
• We have to invent a name for our free-standing "normal" function that doesn't do a whole lot
• namedFunction can live anywhere in your code base. If your code is large and you are anything like me, it can sometimes take a while to find, by which time you forgot where it was called from!

Let's break down the "new" way (it isn't new at all, it's been around since at least 2011)

,[](int s){ Serial.println("Do nothing"); }

[] = this is a lambda function - it has no name

(int s) = same as before, it takes an int parameter called s

{ Serial.println("Do nothing"); } = this is what the function does, its body. any valid C++ code can live inside the body including if/else blocks, other lambdas etc.

Not too painful, I trust? In summary it's a function with no name (an "anonymous function") that is "bolted in" to the place that needs to call it, instead of having to live outside on its own. It has many benefits:

• Less typing
• Less names to remember
• Lives alongside the thing that defined it and needs it: makes code more easy to understand and saves time hunting
• You can do things with it that you would never have dreamt of, like pass it, lock stock and barrel as an object to another function that can then call it on your behalf! That is beyond the scope of this post, though. Ask Mr Google.

I mention these in some detail because a lot of the example code uses them, for all the reasons above, and because I love them. Esparto could not have been written without them. I hope you come to love them too, and soon - they make working with Esparto a breeze and they're not really that tough are they? Welcome to the 21st century!

Why do nothing?

The demo is purely to show the raw LED beating closely-ish in time with either some music, clapping of hands, whistling, dogs howling etc. Since Esparto does all the checking for changes and SmartPins underneath does all the flashing automatically, there is nothing else for our demo code to do. This shows how powerful Esparto is. Ordinarily the lambda is where you would put your special code that makes your app different from the rest. I do exactly that with the Latching button, which starts and stops the LED flashing by calling buttonPress which you haven't seen yet, but is here in all its glory:

void buttonPress(bool hilo){

  if(!hilo) {

    uint32_t rate=Esparto.getConfigInt("blinkrate");

    Esparto.flashLED(rate); 

  }

  else  Esparto.stopLED();

}

User-defined config variables:

And in Esparto.getConfigInt("blinkrate"); you now see the Esparto magic starting to happen. I challenge you to look at the screen shot above and guess what happens when you change the value. Go on, have a go!

If you said "I bet the LED starts flashing at the new rate automatically", you're obviously catching on but you'd be wrong. Only because I'm teasing and you haven't yet pressed the pushbutton to start it flashing at the old rate in the first place. If you had already done that then yes, exactly correct: the LED instantly starts flashing at the new rate, well done! It's now no great leap of faith to correctly assume that changing the debounce value will, er, change the debounce value of the Latching button. You are getting a whole lot of functionality for free here.

But there's more: next time you reboot, the value will be brought back - the config system saves the value whenever it changes, you have nothing further to do. The BWF parameter just made up, to play with, does nothing, isn't used anywhere and you can type what you want in there just for the fun of seeing it survive a reboot. If you want real magic, read the next section on the run panel...

Yet more: send the command testbed/flash with a payload of 1 to start and 0 to stop from an MQTT client and guess what - correct the same thing happens as if you had pressed the button physically yourself. The code to make that happen? Here:

void onMqttConnect(void){

  Serial.printf("T=%d USER SAYS MQTT CONNECTED\n",millis());

  Esparto.subscribe("flash",[](vector<string> vs){ 

    Serial.printf("Doing my thing with %s\n",CSTR(vs.back()));

    buttonPress(!atoi(CSTR(vs.back())));

    });

}

Dont worry about the "vector" stuff, that's more C++ magic that is going to make your relationship with Esparto a much more fruitful one and will be covered in the future. For now be happy that you have just avoided 3 months of tearing your hair out and a learning curve like the side of a cliff, while getting an already pretty capable system "for free" from a mere handful of lines of code!

System config variables:

Anything starting with a "~" is a system variable which Esparto relies on to function properly. So:

1. never use "~" in your own config names
2. while you can put whatever you like in your own variables as long as your code knows what it means, the same cannot be said for system variables
3. never change a system variable unless you know what you are doing, and why!

Some system variables are easy to understand and make sense for the user to change. The ones I have chosen to expose for the demo are like that. By the time the full release comes round there will be a lot more, and they won't be as nice. I can safely predict that even when you read the "advanced guide" with a full explanation of what each does, you still won't want / dare / understand how to change them, so - just don't. Ever!

~fb2Ap: 

Is the millisecond count for the amount of time to wait for the WiFi to fail to connect before "falling back" to AP mode and offering yourself up to a phone, tablet etc to get in and configure a valid set of WiFi credentials. The demo has 3 minutes = 180,000 microseconds = 180 seconds. You may want less or more: feel free to change it to a sensible value that works for you.

~lh:

Is used to log the value of the heap every second to an MQTT broker, just in case the 3-minute graphs on the system page aren't enough. 1= start, 0= stop. It will publish /testbed/heap with a payload of the value once per second until you stop it, either by changing the value back to 0, publishing testbed/cmd/logheap/0 over MQTT or reading on to the next section on the run panel...

~mqXXX:

Unsurprisingly, the IP address, port and retry failure re-connection interval of your MQTT broker. Some day soon I will add ~mixer and ~mqPass to enable you to connect to an authenticating remote server. Some day...

Don't ask me about (or mess with!) the as-yet-unseen ~jitter variable - it's the plus / minus entropy timing spread adjustment factor to minimise asynchronous collision probability in the autoStats derived timer reset function. It is currently set to 10. Still fancy seeing what happens if you change it to 11? Or 243? No, I hoped not.

*Ever. No delay loops ever. They are bad, they break asynchronous libraries, stop other tasks from running and are generally BAD STYLE. Do not ever use one in an Esparto callback (or at all, in fact) you simply don't need to. If you think you do, trust me, you are wrong. There is always a better way. call Esparto.once(<x mSec delay>, functionToRunSoon); for example. Don't ever call delay(). Need I say it again?

Esparto V2 almost ready! The new web UI part 4

The lower panel(s)

Info

This is mostly self-explanatory but also raises a few points that are well worth knowing if you intend to become an "Esparto Expert".

These are all static values at the heart of the system, some of them permanently, some for example the IP address for this boot of the system only. They cannot be directly changed by the user. For some that can, see the next section.

Hardware Type:

This lets you know what's running "under the hood". It can be any one of these:

• ESP-01 (but why would you bother when there’s…)
• ESP-01S
• Wemos D1
• Wemos D1 mini
• Wemos D1 lite (and thus probably any other ESP8285 device)
• Wemos D1 pro
• NodeMCU 0.9
• SONOFF Basic
• SONOFF S20
• SONOFF SV

Esparto has been tested on all of the above. It will probably run on anything with an ESP-12 in it, but obviously I can't test every single device on the market/ If you want to send me one to try it out and modify if necessary...

I will be very interested to hear of anybody getting it running on any other platform e.g. NodeMCU 1.0 will probably work, as will (I expect) other SONOFFs

Unique Hardware ID:

This is the last 6 digits of the MAC address and is commonly used in new-out-of-the-box scenarios as a default name before choosing your own and setting your SSID / Password credentials the default name of the demo device would be ESPARTO-17D383. See part 2 earlier for more detail on this value and advice renaming your device to replace it.

IP Address:

Need I say more?

Flash Memory Size:

Same as above, except the answer is "Yes". 

Esparto weighs in at  over 410k - it has a lot of functionality and features. To support OTA updating (and who wouldn't want that?) a "sketch" (app) has to be able to fit into half the available flash size. On smaller devices e.g. the SONOFFs you get 1MB thus 512k is usable if you want OTA (and you do!).

As you can see things are already starting to squeak, so you need to keep your own additional code
small, efficient and light-weight. There is also a very limited amount of heap left. Esparto starts up with about 27k free, and that can go up and down rapidly - see the graphs in part 3 of this series for an example. Keep your heap use to a minimum and guard any heap-using routines with a check in what's free first if you want to avoid crashing (again, you do!).

The good news is that Esparto does so much for you that your own code will be small and consist mostly of short callback routines that Esparto will execute at the relevant time on your behalf. There is no loop() function and no setup() function. You will rapidly get used to doing things the "Esparto Way" once you see how easy it is.

H4 library version number:

If you want to get further than a simple "Blinky" it helps to understand the structure of Esparto. It is built from 3 main libraries, H4, SmartPins and Esparto itself.

H4 which handles all the timer functions, scheduling, task separation and "slip streaming" of asynchronous functions into the synchronous task queue which runs on the main loop. No more WDT resets, no more "volatile"s. When your task runs, it is (almost) the only player in town and the H4 library makes sure you have to try really really hard to break things or upset other tasks.

It comes with 7 of its own examples demonstrating how each and all of its functions work. Esparto "encapsulates" H4, so all of the H4 functions will appear to you as identical Esparto functions, so you do need to understand these first.

SmartPins library version number:

See above. Note the version shown is incorrect - by the time of  release it will also be 2.0.0 (actually it is, but I forgot to update the version number field before the demo - my bad!)

SmartPins as its name suggests manages all the input and output pins for you. It is what enables Esparto to give you the fancy real-time flashing LED display for all the pins. It also does everything you could ever want to do with a pin, including debouncing, interrupt handling (although there are good reasons why you would probably never need to use it), rotary decoding and much more.

The Encoder input type lets you manage a rotary with a single line of code - you tell it the name of a variable, and whenever you access the variable it will automagically have the current decoder value in it. One line of code! It's my favourite Esparto feature: most of my own mini-apps have some kind of "tweak" factor using a rotary, it's so easy. Some even have two...at the extra expense of one more line of code...I'll stop now, I think you have got the point.

SmartPins comes with nineteen sample program covering every in and out (literally!) of the many types of input modes it supports. It also has access to all H4's functions and relies on it 100% to function. Pretty much every example has at least one or two H4 features though of course they appear seamlessly as identical SmartPins features.

It is important then to work through the examples in order to fully understand the power and flexibility of Esparto, because in the same way, all Esparto functions are automatically the same as all SmartPins functions.

Even seasoned programmers will benefit, as Esparto works in a very different way from 99.235% of all the thousands of sample sketches you will find online. You need to learn the "Esparto Way", but for those with experience it won't take long at all.

Just as an example, here's the code (with comments removed for brevity) for the simple blinky. "Simple" includes having a fully debounced on/off switch unlike 99.476% of other blinkies.

NOTE: 

While H4 and SmartPins both have visible setup() and loop() functions, Esparto does not. There are two reasons for keeping them in:

1. To make the early examples more readily recognisable and ease you in to the "Esparto Way" and "chunk up" the amount of learning at each stage into bite-sized pieces.
2. To enable you to use them on their own without the full Esparto, although I can't think of any reason why you would want to unless you are the kind that likes to make things deliberately hard for themselves

#include <SmartPins.h>

SmartPins smartPins;

void buttonPress(bool hilo){

  if(!hilo) smartPins.flashLED(250);

  else smartPins.stopLED();

}

void setup(){

  Serial.begin(74880);

  Serial.println("LED will change state (flashing/off) on each separate button up/down press");

  smartPins.Output(BUILTIN_LED);

  smartPins.Latching(0,INPUT,15,buttonPress); // GPIO 0 + 15ms of debouncing

}

void loop(){

  smartPins.loop();

}

I hope you will agree both that it's pretty easy and also that you get "a lot for your money" for very little coding effort. That principle underlies the whole of the "Esparto Way": Esparto does 90% of the "heavy lifting", you plug in the remaining 10% which is specific to your IOT / home automation app. Esparto allows you to concentrate on just the code that's important to you - all the hard stuff "just works"

NBoot & Code:

These may be the first indication of a (hopefully very rare) problem. NBoot is the number of times this device has been rebooted and "Code" is the reason why. If it has just been freshly programmed then (as has the demo device) then it will read ESPARTO_BOOT_UNCONTROLLED.

What this means is that it was not shut down by user action, but forcibly rebooted, as the IDE does. You will also see this code if the device crashes for any reason.

If you click the Reboot button,. the code will become  ESPARTO_BOOT_UI. If you send an MQTT command e.g. testbed/cmd/reboot the code becomes ESPARTO_BOOT_MQTT and son on, although obviously you will replace "testbed" with your own device name first.

If you see an increased boot count and a reason you don't expect - something has gone wrong!

The "tXXX" values:

These measure the amount of milliseconds since boot up when:

tHW: 

The time after which your sensors, buttons, relays, remote controlled Gatling guns etc become ready to run. One of the fundamental design goals of Esparto is that your hardware should operate a) as early as possible b) whether you have a WiFi connection or not c) all the time, always.

Even if - as happens in the real world - bugs occur and the occasional crash occurs, your hardware will be back up ready to go in about 125 milliseconds. Impressive, non? It's one of the reason behind why Esparto won't let you play with setup() and loop(): it has quite a bit of complex setup of its own to do, and it has to happen fast, and in a very specific order.

tWiFi:

The time after which you can load up the web UI because your device now has a valid IP address.

tMQTT: 

Similarly, the time after which Esparto is actively listening for MQTT commands, both its own any any that you choose also to listen for. All Esparto command start with "cmd", so you must not use this in any of your own topics, or who knows when that Gatling gun may go off in error?

High values of either tWiFi or tMQTT may be early indications of problems with your router, network or MQTT broker. Or they may just be a sign of a slow network - only you will know. Personally, I'd worry about anything much more than the demo values. Again, I think 3.2 secs from power on to receiving MQTT commands is "in the zone".

Using C++ std library with ESP8266 Ardiuino IDE - some "gotchas" solved

The hunting of the Snark

In the latter stages of testing the forthcoming release of Esparto v2, I was "stress" testing it by throwing everything including the kitchen sink at it to see how it coped.

Short story is : it didn't. Low workloads were absolutely fine, but once the rate got above a certain figure - and I couldn't work out exactly how big - it fell in a screaming heap, which is the one thing it is designed specifically NOT to do.

Very frequently it would crash with a stack dump and it was clear that heap exhaustion was the culprit. What slowed me down was that I had spent a lot of time building in heap protection. But no matter how careful and extensive my protection was, it was failing. I searched - literally - for days, stitching in almost as much diagnostic code as there was "real" code. On a couple of occasions, I created a "heisenbug" with my diagnostics: after putting them in, the bug went away. Take them back out, it reappeared. "Aha!" I thought that's a sure-fire sign of a timing problem.

Given that the whole thing is wrapped very tightly round a microsecond-indexed custom priority_queue on the main loop which has to co-operate with asynchronous events via a mutex, it was no surprise. And I had had problems with that core module in the past. And I had made a lot of changes to it -so it was no surprise to find that that's where a lot of my "wasted" days were spent...

Ironically, it's not actually a lot of code - about 500 lines of which about 30-odd percent or more is comment and/or vertical spacing for clarity so maybe only 3-400 lines of code, but its hellishly complex - for me at least.

For a start although I have been Cing and C++ing for many a year, I only looked st the standard library about 9 months ago, as It had rapidly become clear that "functional programming" was  the only way I could get a timer to call back to a class method which was a fundamental pre-requisite for an Arduino library. Once I had got my head around lambdas and function objects and the how-did-I-ever-do-without-it bind mechanism with placeholder parameters, I was cooking! I then delved into containers and I'm now sold sold sold and wouldn't ever dream of doing things any other way. I did have to spend a lot of time with my head in Stroustrup and whining on Stack Overflow, but I got there. Mostly.

Secondly, there is the very nature of the beast: that rare area that mostly only OS designers play with more than once: task synchronisation, critical sections, interlocked atomic access, volatiles, mutexes and a slew of buzz phrases most programmers never meet in a full career...luckily I'd had some previous experience - on mainframes in the mid 1970s...Oddly enough, it was a huge help when it all came back to me. The words may be different in ESP8266land, but the tune is the same.

Thirdly I was new-ish to the ESP8266 having got into the embedded scene (as many others) with AVR-based Arduinos.

All in all, a "random" timing bug under heavy loads was all I didn't need. After days of searching and not finding, I came reluctantly to the Sherlock Holmes conclusion:

"When you have eliminated the impossible, whatever is left - no matter how improbable - must be the truth"

The truth in this case was that the reason I couldn't find the bug in my code is because - it wasn't in my code! Therefore it must be in someone else's! The prospect of digging through profoundly complex library code written by geniuses - obviously for what would be days and days - did not fill me with deep joy. Once I had finally tracked down and nailed the problem it turned out, it wasn't actually a bug but a combination of two things, both of them on the very limits of my knowledge comfort zone. But I'm jumping ahead.

The first big clue was my voluminous diagnostics revealing a sudden and huge drop in the available heap shortly before lights-out. Every time. At least that part had a pattern. I knew though bitter sweat and very very late nights to the sound of much swearing, I could account for every byte of memory I used, every pointer RAII'd until it bled, every heap fluctuation possible - I had the spreadsheets and graphs to prove it. Yet here was a sudden massive bite out of a very small cherry while I sat with my jaw clamped shut.

We all know the ESP8266 is heap-limited. There is a ton of other code on top of my core scheduler: a web server doing dynamic updates in real-time via web sockets, a real-time pin management library, auto WiFi / MQTT connection  management with AP fallback , yada yada yada all the way up to 410k plus. The app gets out of bed with less than 30k heap to play with. By the time it tells you MQTT is waiting to hear from you, there is only 27k left in the bank.

So who or what was stealing - out of nowhere - 16k or 60% of my available space? It would be OK when I was at 27, but if the heap-thief helped itself when I had 11k spare, guess what? Oblivion. Sadly, it did exactly that quite often.

Now the implementers of the std library on this platform are obviously well aware of the ESP8266's pint pot into which they had to shoehorn a barell-full of stdlib. Something had to give. Gone is any type info, gone is the ability to get the target of a function object...I could go on, suffice to say that while it is mostly functional for my humble needs it is a (necessarily) heavily-hacked offering.

The extensive stack back trace looked like an explosion in an alphabet soup factory in Moscow. I couldn't make head nor tail once it had got below the precious few of my own routines but I persevered and finally hit pay dirt as the second of the issues became apparent to me: my lack of experience in looking "under the hood" of stl.

It's a memory problem, I know that much. I also know I can't see the problem because its not in my code, so I'm going to have put diagnostics in - gasp - the stl code. After recovering from a near-faint at the thought, I vaguely recalled a little-if-ever used feature of the custom allocator. I had glossed over it as I knew I would never be needing it. Until now, of course! If I could write a custom allocator - yes, this is how desperate I was getting - I could sprinkle it with printfs and finally nail the problem.

By jingo, that's what I started to do. I found a skeleton malloc-based proof-of-concept template on't'web and blindly cut and pasted it it, as I sure as hell couldn't understand a word of it! So it was back to Bjarne and StackOverflow and I concentrated on how this foreign beast was meant to do it magic, I never even considered the when and why...

Blinkering my vision was the fact that because I "knew" I only had a few tasks in my queue, I was concentrating on numerous other potential culprits. What I hadn't seen was that a second (yes! how odd!) bug was causing me to double- and even triple-dip at stuffing tasks into the queue. The logic simply ignored the excess and/or duplicates, so it just wasn't apparent anywhere.

I think the exact Eureka! moment came as I delved deeper into the allocator's black box that I heard myself asking "Why is the last thing just after I have added one of my meagre handful of taks to the queue?" At this point, the experienced stl'ers amongst you will have probably already seen the problem, which could even be - by some standards - classed as a "schoolboy error". Those of you who use it on big machines with unlimited virtual memory may not yet have spotted it though, because what happens on an ESP8266 never happens in your big world.

Hiding complexity is one of stl's great strengths, but it has it's dark side too, as was the case here. I knew that these magic containers would stretch to fit. What I didn't know was a) when b) by how much. What I also couldn't see was that - albeit in error - I had filled up my small queue. How small? Who knew? Only the folk who implemented the stl on the '8266. Some arbitrary default initial size that would automatically stretch to fit as and when required.Without asking. Or telling. All that nasty implementation stuff is hidden from view, which - after all - is why we use the thing in the first place isn't it?

Drifting off into the new territory of dynamic resizing, I was trying to find the place in the allocator to put the printf with current heap size at the point where my queue was allocated, so I could see how much free heap there was and what else might be stealing it But I couldn't spot where an initial allocation was made and/or a resizing allocation was made. I didn't want to waste my efforts and put it anywhere near resizing code, because my queue was so small it would never need resizing! In an idle moment I mused on what might happen in that scenario: how much bigger would it grow and where in this mess would I even start to look for that?

"Because.." I thought, musing to myself to avoid having to solve this impossible bug, "...if  it did for some bizarre reason allocate a huge chunk...".

Then it hit me like a brick: I was looking for something under the hood that not only had the potential to, but was actually grabbing a huge chunk of memory without being asked, wasn't I? I'd got so many levels nested in the complexity that I'd forgotten my call return address. Once I unwound my own mental stack, I realised I was staring the problem right in the eyes: It wasn't something else stealing the memory, it was  the queue itself!

"But why? It's tiny, look, this extra diag will show the size....!. Oh. Oh, that's a shock - where did all those tasks come from? Is there a bug in clearing the queue? Nope, checked that, works fine. is it a timing thing? Queue so busy that some underlying memory-freeing thingamajig an't get a look in?"

No, it's me accidentally busting the queue and the queue very obligingly and silently trying to expand to nicely accommodate me ...by not-so-nicely grabbing 60% of the little I have left. Bingo!

I rapidly found out how to set the capacity of the underlying vector container, hunted down the duh! bug that was tripling my queue, daringly removed (well commented out, I'm not that stupid) almost all the diagnostics, crossed my fingers, legs, arms and eyes and hit "run".

You know the rest: bug splatted. All the other code worked fine too and suddenly I'm almost ready to release. See the more recent posts here for a quick look-in - ignore the earlier posts from a few months back, it's changed a lot (because of this) since then.

On a big machine in the real world, I'd have never even noticed this. I'd have gone for release with an inefficient, heap-hungry monster. Such beasts don't live long in the mbeeded wrold, This particular one lived a lot longer than it should have, but - hey, we learn by our mistakes. Or by reading about others'!

The morals of this story are manifold:

• Get to know a new technology in detail before playing with it in anger
• Preallocate stl containers wherever possible
• Where you can't preallocate, guard all accesses with heap checks
• When you have eliminated the impossible...
• Find the person who thinks 16k is a good size for reallocation on an '8266 and "have a word"

Anyone want to buy a brand-new completely unused custom stl memory allocator? It's an ideal self-teaching tool...

Esparto V2 almost ready! The new web UI part 3

The lower panel(s)

System

This panel will be used mostly for debugging your Esparto App. The ADC graph shows the raw value of the A0 pin from 0 to 1023 and can be useful when calibrating any sensor attached thereto.

The Heap graph:

The heap graph is probably the most important tool here. The ESP8266 has limited heap space and many of the underlying libraries chew up a fair amount. The main AsyncWebserver library is also very sensitive to low heap situations and despite it being a truly marvellous piece of software, it will crash unceremoniously if there is very little free heap space. I have not gone to the lengths of calibrating it exactly, but below 5 or 6k seems to be in the danger zone. Get the software on github.

Running out of heap is fatal in all circumstances and great care must be exercised in your code to make sure you use as little as possible. The graph is designed to help you do that. The window is 3 minutes wide, which is more than enough time to spot any problems. Trust me, when they happen, they happen fast!

Esparto has a built-in heap guard so that it will not let any process start if there is less than a defined amount of heap available. The value is configurable at compile=time and I usually run with it set to 20% of the initial free heap. You may need to "tweak" it while watching this graph to optimise the figure for your own app.

The downside of this is that once the limit is reached, any / every event requiring a task will be silently ignored, This can make your app behave strangely: LEDs may stop flashing, sensor values may not get sent to MQTT, your buttons may not work, or worse: work on the down stroke, but not on the up. The point is that whatever happens, the app will not crash. It reserves at least enough heap (I hope!) to be able to call up the UI panels so that you can spot the problem with this panel, fix it perhaps by changing a configuration value on the config panel or use the run panel to issue a command to stop the bad behaviour, while letting the good stuff continue. That's the idea, anyway.

The Q panel shows the size of Esparto's task queue. Most of the time this will read zero, i.e no outstanding tasks. Technically it should show at least one, but it clears the queue so quickly and the stats are only refreshed once per second, so they "miss" the occasional internal background management task. This raises an important point: all of these statistics (apart from the ADC) are indicative only: they cannot be 100% accurate (nor do they need to be) by simple virtue of the lag between the actual event, the network and the browser. They are certainly accurate enough to help you spot problems very early on and then tune them out.

Again, the queue can be sized at compile-time and some experimentation may be needed to maximise free heap with a small queue - but not so small that tasks start stacking up or being "throttled". The same caveat applies: once the queue limit is reached, tasks will be ignored until the level drops below the limit. All the same glitches as above may or may not occur, but still: no crash. In any event it is 90-odd percent certain the heap will be throttled well before the Q grows out-of-bounds.

Warning!

There is one sure-fire way of busting the queue: scheduling a repetitive task that runs for longer than the scheduling period. For example running a job every 5ms that takes 6ms to complete. It doesn't take much thought to realise that this situation is never going to end well. The queue will grow exponentially and given that 10 or 20 is a sensible practical limit, the rate of growth before throttling occurs will be of the order of milliseconds. You already know what starts (or more often "stops") when these limits are reached.

Another "don't try this at home, kids" method is to schedule a taks that schedules another, which schedules...etc in a chain which is longer than the queue size (minus 1 or 2 for system tasks). Granted, its rare and difficult to do, but here is almost certainly a better and safer way, so if you are thinking of long chains of tasks to get some wacky timing working, think again...

Not least for the fact that again timings are not 100% reliable: they depend on other tasks in the queue. If your new task to start in 10ms slips in behind one that is scheduled to run before yours and takes 15ms then yours wont start for 25ms: his 10 and your 15. If however the queue is empty - which it usually will be - then yours will start on time. The moral of the story is: don't run a nuclear power station or your mother's life support system with Esparto. Most other stuff will work just fine.

The Pins Graph:

Pin activity is the main culprit in heap depletion: Esparto has quite a lot to do when a pin change occurs. It has to check to see if the pin is throttled (more on that in a moment) and if so, discard inputs over the throttling limit, if the input survives that, Esparto has to light the appropriate raw LED then allow the specific pin-type handler to decide / calculate / guess (only kidding) the internal cooked state, get the value and light or extinguish the corresponding cooked LED. And update the pin statistics, and...

All of which uses up heap space. The snapshot above was deliberately chosen to show how a burst of high activity on the pins causes an immediate and severe drop in the free heap space. In case you are interested, it was caused by flashing 3 LEDs at a high rate while simultaneously "listening" to a sound sensor like this:

Buy on ebay for about £1

on pin D6 while Motorhead's "Ace of Spades" was playing at high volume. The track was chosen specifically because it is a "wall of noise" and causes the sensor to throw literally thousands of transitions per second into the pin, which brings us nicely on to Esparto's approach to pin throttling...

PinThrottling:

Very few ESP8266 apps can sustain thousands of transitions per second (peaking at 11000+ form Motorhead when turned up loud, which is - of course - the only way to listen to it) while also managing other hardware and a dynamically-updated web UI. Esparto is no different: it's not magic! It depends on underlying libraries that have limits. For example the AsyncWebServer library can only deliver about 20 requests to the browser per second. This of course depends on how fast the browser can consume them but when you look at what Esparto is doing on this panel alone, there's a lot to get through. By empirical observation, 21 inputs per second is the tipping point at which the heap starts to drop like a heavy rock. The chosen tune has no let-up, once the descent begins there's no way back before a very rapid heap exhaustion crash, except to  a) throttle the heap b) throttle the cause of the problem rather than the symptom: the rate of throughput on the input pin. Esparto does both.

So the LED is never going to flash in time with the beat. Cutting the rate from 11000+ to 21 is obviously some heavy clipping, but it is essential to prevent a crash. It's actually worse: because of some background tasks, even at 20/sec the heap depletes, albeit slowly. It doesn't actually stop entirely until 19. Unfortunately the ESP floating point code is so big and Esparto already weighs in at 410k that Esparto can only deal with integer arithmetic - for speed as much as size - but size is what prohibits "proper" math. So when calculating the throttle sample rate, 19 gets rounded down to 10.

The count on the pin is sampled 10x per second which is 100ms between checks. This is a fine balance between early notification, accuracy and impact on other process. Pin D6 is throwing 1s and 0s in at 11000 per second (that's 91 microseconds between) so the first time we get to check the pin count, it's already up to 1/10 of it maximum which is 1100 - a heap-busting amount of activity.

The maths behind this problem is much deeper than we need to get into here, but the result is that in its current guise, Esparto can only throttle pins in multiple of 10. Doing it at 20 would be accurate, as would 30 - or 10. Anything in between is rounded down by the integer divide. Rounding it up wouldn't make a lot of sense for a limit, now would it?

As a slight aside, the reason that pin D6 (GPIO12) is coloured yellow in the UI snapshot above is to indicate that it is throttled.