Esparto v3.3 released at last

AT last! A "quick fix " to patch up a problem in someone else's library has just ended up being a 2-month ground-up rewrite. I am so fed up with the whole thing, I hope you lot all like it enough to cheer me up: Watch the video then go and have a play with it. Esparto v3.3 released at last. Still some documentation being "backfilled" but the code should be ok and with 61 examples, you should be fine....

Esparto 3.3v feature preview

Esparto V2 almost ready! The new web UI part 5

The lower panel(s)

Config

Now we start to see the real power of Esparto coming out. It has a configuration system where name/value pairs are automatically saved to SPIFFS (the ESP8266 Flash file system) as soon as they change and persist into the next reboot, i.e become permanent. Well, until the next factory reset, at least.

The demo code has a Latching push button on GPIO0 (Arduino digital pin D3) and a very "noisy" and sensitive  sound sensor (i.e. a high number of thousand IOs per second at the slightest cough) on GPIO12 (D6). It also configures the BUILTIN_LED for output. On a Wemos D1 mini that the demo was built on, this is on GPIO2 (D4). The hardware setup looks like this:

void setupHardware(){

  Esparto.Output(BUILTIN_LED,LOW,HIGH); // start with LED OFF                   

  Esparto.Latching(PUSHBUTTON,INPUT,10,buttonPress); // 10ms of debouncing

  Esparto.Raw(D6,INPUT,[](int s){ Serial.println("Do nothing"); });

  Esparto.throttlePin(D6,19);

}

I trust your first taste of the "esparto Way" wasn't too shocking or difficult? setupHardware() is equivalent to the standard Arduino-style setup() and you do the same kind of thing here as you would there - almost. You just do it Esparto-stylee - so for instance, no WiFi.begin and delay loops* Esparto is already connecting to your SSID "in the background" to speed things up.

Also you won't see pinMode calls: Esparto knows what mode to set automatically from the type of Esparto SmartPin you define. So really all we have is one line per I/O device, and often that's all you will need. The only "odd" or "tricky" thing is the throttlePin call. We'll get to the strange syntax in a minute, but first, what exactly is "throttling" and why do we need it here?

It is described in great detail in part 3 of this series, so if you want to know more read up on that first, but for now a simple one liner is that the sound sensors fires far more data than any tiny device can easily cope with -Esparto being no exception - so we have to slow it down, or "throttle" it. D6 is our noisy pin, so we tell Esparto only to allow through 19 of the thousands of 1s and 0s per second.

The reason this is such a low figure is explained in the earlier article. Your LED will still flash vaguely in time with your bangin' house or lounge jazz tracks...ish. Now to that weird syntax...

C++ Lambda functions:

See what? If you don't know about these already, ask Mr Google about them because you will quickly come to love them as much as I do. They are particularly good for callbacks and a lot of your code needs to be in callbacks so now is a good time to learn how to use them. If you are already frightened, fear not: you don't have to use them, the old-fashioned way still works. I will show you what that would look like in a moment and I'm sure you will soon be seeing the benefits of the new-fangled way.

What we want is for Esparto to tell us when pin D6 changes and what is has just changed to: a 0 or a 1. So we need to give Esparto a function that returns nothing (void) and takes a single int parameter, which holds the new state when the pin changed. Ordinarily we'd write:

void namedFunction(int s){

Serial.println("Do nothing");

}

and then our old-fashioned way would be:

Esparto.Raw(D6,INPUT,namedFunction);

But:

• It's more typing
• We have to invent a name for our free-standing "normal" function that doesn't do a whole lot
• namedFunction can live anywhere in your code base. If your code is large and you are anything like me, it can sometimes take a while to find, by which time you forgot where it was called from!

Let's break down the "new" way (it isn't new at all, it's been around since at least 2011)

,[](int s){ Serial.println("Do nothing"); }

[] = this is a lambda function - it has no name

(int s) = same as before, it takes an int parameter called s

{ Serial.println("Do nothing"); } = this is what the function does, its body. any valid C++ code can live inside the body including if/else blocks, other lambdas etc.

Not too painful, I trust? In summary it's a function with no name (an "anonymous function") that is "bolted in" to the place that needs to call it, instead of having to live outside on its own. It has many benefits:

• Less typing
• Less names to remember
• Lives alongside the thing that defined it and needs it: makes code more easy to understand and saves time hunting
• You can do things with it that you would never have dreamt of, like pass it, lock stock and barrel as an object to another function that can then call it on your behalf! That is beyond the scope of this post, though. Ask Mr Google.

I mention these in some detail because a lot of the example code uses them, for all the reasons above, and because I love them. Esparto could not have been written without them. I hope you come to love them too, and soon - they make working with Esparto a breeze and they're not really that tough are they? Welcome to the 21st century!

Why do nothing?

The demo is purely to show the raw LED beating closely-ish in time with either some music, clapping of hands, whistling, dogs howling etc. Since Esparto does all the checking for changes and SmartPins underneath does all the flashing automatically, there is nothing else for our demo code to do. This shows how powerful Esparto is. Ordinarily the lambda is where you would put your special code that makes your app different from the rest. I do exactly that with the Latching button, which starts and stops the LED flashing by calling buttonPress which you haven't seen yet, but is here in all its glory:

void buttonPress(bool hilo){

  if(!hilo) {

    uint32_t rate=Esparto.getConfigInt("blinkrate");

    Esparto.flashLED(rate); 

  }

  else  Esparto.stopLED();

}

User-defined config variables:

And in Esparto.getConfigInt("blinkrate"); you now see the Esparto magic starting to happen. I challenge you to look at the screen shot above and guess what happens when you change the value. Go on, have a go!

If you said "I bet the LED starts flashing at the new rate automatically", you're obviously catching on but you'd be wrong. Only because I'm teasing and you haven't yet pressed the pushbutton to start it flashing at the old rate in the first place. If you had already done that then yes, exactly correct: the LED instantly starts flashing at the new rate, well done! It's now no great leap of faith to correctly assume that changing the debounce value will, er, change the debounce value of the Latching button. You are getting a whole lot of functionality for free here.

But there's more: next time you reboot, the value will be brought back - the config system saves the value whenever it changes, you have nothing further to do. The BWF parameter just made up, to play with, does nothing, isn't used anywhere and you can type what you want in there just for the fun of seeing it survive a reboot. If you want real magic, read the next section on the run panel...

Yet more: send the command testbed/flash with a payload of 1 to start and 0 to stop from an MQTT client and guess what - correct the same thing happens as if you had pressed the button physically yourself. The code to make that happen? Here:

void onMqttConnect(void){

  Serial.printf("T=%d USER SAYS MQTT CONNECTED\n",millis());

  Esparto.subscribe("flash",[](vector<string> vs){ 

    Serial.printf("Doing my thing with %s\n",CSTR(vs.back()));

    buttonPress(!atoi(CSTR(vs.back())));

    });

}

Dont worry about the "vector" stuff, that's more C++ magic that is going to make your relationship with Esparto a much more fruitful one and will be covered in the future. For now be happy that you have just avoided 3 months of tearing your hair out and a learning curve like the side of a cliff, while getting an already pretty capable system "for free" from a mere handful of lines of code!

System config variables:

Anything starting with a "~" is a system variable which Esparto relies on to function properly. So:

1. never use "~" in your own config names
2. while you can put whatever you like in your own variables as long as your code knows what it means, the same cannot be said for system variables
3. never change a system variable unless you know what you are doing, and why!

Some system variables are easy to understand and make sense for the user to change. The ones I have chosen to expose for the demo are like that. By the time the full release comes round there will be a lot more, and they won't be as nice. I can safely predict that even when you read the "advanced guide" with a full explanation of what each does, you still won't want / dare / understand how to change them, so - just don't. Ever!

~fb2Ap: 

Is the millisecond count for the amount of time to wait for the WiFi to fail to connect before "falling back" to AP mode and offering yourself up to a phone, tablet etc to get in and configure a valid set of WiFi credentials. The demo has 3 minutes = 180,000 microseconds = 180 seconds. You may want less or more: feel free to change it to a sensible value that works for you.

~lh:

Is used to log the value of the heap every second to an MQTT broker, just in case the 3-minute graphs on the system page aren't enough. 1= start, 0= stop. It will publish /testbed/heap with a payload of the value once per second until you stop it, either by changing the value back to 0, publishing testbed/cmd/logheap/0 over MQTT or reading on to the next section on the run panel...

~mqXXX:

Unsurprisingly, the IP address, port and retry failure re-connection interval of your MQTT broker. Some day soon I will add ~mixer and ~mqPass to enable you to connect to an authenticating remote server. Some day...

Don't ask me about (or mess with!) the as-yet-unseen ~jitter variable - it's the plus / minus entropy timing spread adjustment factor to minimise asynchronous collision probability in the autoStats derived timer reset function. It is currently set to 10. Still fancy seeing what happens if you change it to 11? Or 243? No, I hoped not.

*Ever. No delay loops ever. They are bad, they break asynchronous libraries, stop other tasks from running and are generally BAD STYLE. Do not ever use one in an Esparto callback (or at all, in fact) you simply don't need to. If you think you do, trust me, you are wrong. There is always a better way. call Esparto.once(<x mSec delay>, functionToRunSoon); for example. Don't ever call delay(). Need I say it again?

Using C++ std library with ESP8266 Ardiuino IDE - some "gotchas" solved

The hunting of the Snark

In the latter stages of testing the forthcoming release of Esparto v2, I was "stress" testing it by throwing everything including the kitchen sink at it to see how it coped.

Short story is : it didn't. Low workloads were absolutely fine, but once the rate got above a certain figure - and I couldn't work out exactly how big - it fell in a screaming heap, which is the one thing it is designed specifically NOT to do.

Very frequently it would crash with a stack dump and it was clear that heap exhaustion was the culprit. What slowed me down was that I had spent a lot of time building in heap protection. But no matter how careful and extensive my protection was, it was failing. I searched - literally - for days, stitching in almost as much diagnostic code as there was "real" code. On a couple of occasions, I created a "heisenbug" with my diagnostics: after putting them in, the bug went away. Take them back out, it reappeared. "Aha!" I thought that's a sure-fire sign of a timing problem.

Given that the whole thing is wrapped very tightly round a microsecond-indexed custom priority_queue on the main loop which has to co-operate with asynchronous events via a mutex, it was no surprise. And I had had problems with that core module in the past. And I had made a lot of changes to it -so it was no surprise to find that that's where a lot of my "wasted" days were spent...

Ironically, it's not actually a lot of code - about 500 lines of which about 30-odd percent or more is comment and/or vertical spacing for clarity so maybe only 3-400 lines of code, but its hellishly complex - for me at least.

For a start although I have been Cing and C++ing for many a year, I only looked st the standard library about 9 months ago, as It had rapidly become clear that "functional programming" was  the only way I could get a timer to call back to a class method which was a fundamental pre-requisite for an Arduino library. Once I had got my head around lambdas and function objects and the how-did-I-ever-do-without-it bind mechanism with placeholder parameters, I was cooking! I then delved into containers and I'm now sold sold sold and wouldn't ever dream of doing things any other way. I did have to spend a lot of time with my head in Stroustrup and whining on Stack Overflow, but I got there. Mostly.

Secondly, there is the very nature of the beast: that rare area that mostly only OS designers play with more than once: task synchronisation, critical sections, interlocked atomic access, volatiles, mutexes and a slew of buzz phrases most programmers never meet in a full career...luckily I'd had some previous experience - on mainframes in the mid 1970s...Oddly enough, it was a huge help when it all came back to me. The words may be different in ESP8266land, but the tune is the same.

Thirdly I was new-ish to the ESP8266 having got into the embedded scene (as many others) with AVR-based Arduinos.

All in all, a "random" timing bug under heavy loads was all I didn't need. After days of searching and not finding, I came reluctantly to the Sherlock Holmes conclusion:

"When you have eliminated the impossible, whatever is left - no matter how improbable - must be the truth"

The truth in this case was that the reason I couldn't find the bug in my code is because - it wasn't in my code! Therefore it must be in someone else's! The prospect of digging through profoundly complex library code written by geniuses - obviously for what would be days and days - did not fill me with deep joy. Once I had finally tracked down and nailed the problem it turned out, it wasn't actually a bug but a combination of two things, both of them on the very limits of my knowledge comfort zone. But I'm jumping ahead.

The first big clue was my voluminous diagnostics revealing a sudden and huge drop in the available heap shortly before lights-out. Every time. At least that part had a pattern. I knew though bitter sweat and very very late nights to the sound of much swearing, I could account for every byte of memory I used, every pointer RAII'd until it bled, every heap fluctuation possible - I had the spreadsheets and graphs to prove it. Yet here was a sudden massive bite out of a very small cherry while I sat with my jaw clamped shut.

We all know the ESP8266 is heap-limited. There is a ton of other code on top of my core scheduler: a web server doing dynamic updates in real-time via web sockets, a real-time pin management library, auto WiFi / MQTT connection  management with AP fallback , yada yada yada all the way up to 410k plus. The app gets out of bed with less than 30k heap to play with. By the time it tells you MQTT is waiting to hear from you, there is only 27k left in the bank.

So who or what was stealing - out of nowhere - 16k or 60% of my available space? It would be OK when I was at 27, but if the heap-thief helped itself when I had 11k spare, guess what? Oblivion. Sadly, it did exactly that quite often.

Now the implementers of the std library on this platform are obviously well aware of the ESP8266's pint pot into which they had to shoehorn a barell-full of stdlib. Something had to give. Gone is any type info, gone is the ability to get the target of a function object...I could go on, suffice to say that while it is mostly functional for my humble needs it is a (necessarily) heavily-hacked offering.

The extensive stack back trace looked like an explosion in an alphabet soup factory in Moscow. I couldn't make head nor tail once it had got below the precious few of my own routines but I persevered and finally hit pay dirt as the second of the issues became apparent to me: my lack of experience in looking "under the hood" of stl.

It's a memory problem, I know that much. I also know I can't see the problem because its not in my code, so I'm going to have put diagnostics in - gasp - the stl code. After recovering from a near-faint at the thought, I vaguely recalled a little-if-ever used feature of the custom allocator. I had glossed over it as I knew I would never be needing it. Until now, of course! If I could write a custom allocator - yes, this is how desperate I was getting - I could sprinkle it with printfs and finally nail the problem.

By jingo, that's what I started to do. I found a skeleton malloc-based proof-of-concept template on't'web and blindly cut and pasted it it, as I sure as hell couldn't understand a word of it! So it was back to Bjarne and StackOverflow and I concentrated on how this foreign beast was meant to do it magic, I never even considered the when and why...

Blinkering my vision was the fact that because I "knew" I only had a few tasks in my queue, I was concentrating on numerous other potential culprits. What I hadn't seen was that a second (yes! how odd!) bug was causing me to double- and even triple-dip at stuffing tasks into the queue. The logic simply ignored the excess and/or duplicates, so it just wasn't apparent anywhere.

I think the exact Eureka! moment came as I delved deeper into the allocator's black box that I heard myself asking "Why is the last thing just after I have added one of my meagre handful of taks to the queue?" At this point, the experienced stl'ers amongst you will have probably already seen the problem, which could even be - by some standards - classed as a "schoolboy error". Those of you who use it on big machines with unlimited virtual memory may not yet have spotted it though, because what happens on an ESP8266 never happens in your big world.

Hiding complexity is one of stl's great strengths, but it has it's dark side too, as was the case here. I knew that these magic containers would stretch to fit. What I didn't know was a) when b) by how much. What I also couldn't see was that - albeit in error - I had filled up my small queue. How small? Who knew? Only the folk who implemented the stl on the '8266. Some arbitrary default initial size that would automatically stretch to fit as and when required.Without asking. Or telling. All that nasty implementation stuff is hidden from view, which - after all - is why we use the thing in the first place isn't it?

Drifting off into the new territory of dynamic resizing, I was trying to find the place in the allocator to put the printf with current heap size at the point where my queue was allocated, so I could see how much free heap there was and what else might be stealing it But I couldn't spot where an initial allocation was made and/or a resizing allocation was made. I didn't want to waste my efforts and put it anywhere near resizing code, because my queue was so small it would never need resizing! In an idle moment I mused on what might happen in that scenario: how much bigger would it grow and where in this mess would I even start to look for that?

"Because.." I thought, musing to myself to avoid having to solve this impossible bug, "...if  it did for some bizarre reason allocate a huge chunk...".

Then it hit me like a brick: I was looking for something under the hood that not only had the potential to, but was actually grabbing a huge chunk of memory without being asked, wasn't I? I'd got so many levels nested in the complexity that I'd forgotten my call return address. Once I unwound my own mental stack, I realised I was staring the problem right in the eyes: It wasn't something else stealing the memory, it was  the queue itself!

"But why? It's tiny, look, this extra diag will show the size....!. Oh. Oh, that's a shock - where did all those tasks come from? Is there a bug in clearing the queue? Nope, checked that, works fine. is it a timing thing? Queue so busy that some underlying memory-freeing thingamajig an't get a look in?"

No, it's me accidentally busting the queue and the queue very obligingly and silently trying to expand to nicely accommodate me ...by not-so-nicely grabbing 60% of the little I have left. Bingo!

I rapidly found out how to set the capacity of the underlying vector container, hunted down the duh! bug that was tripling my queue, daringly removed (well commented out, I'm not that stupid) almost all the diagnostics, crossed my fingers, legs, arms and eyes and hit "run".

You know the rest: bug splatted. All the other code worked fine too and suddenly I'm almost ready to release. See the more recent posts here for a quick look-in - ignore the earlier posts from a few months back, it's changed a lot (because of this) since then.

On a big machine in the real world, I'd have never even noticed this. I'd have gone for release with an inefficient, heap-hungry monster. Such beasts don't live long in the mbeeded wrold, This particular one lived a lot longer than it should have, but - hey, we learn by our mistakes. Or by reading about others'!

The morals of this story are manifold:

• Get to know a new technology in detail before playing with it in anger
• Preallocate stl containers wherever possible
• Where you can't preallocate, guard all accesses with heap checks
• When you have eliminated the impossible...
• Find the person who thinks 16k is a good size for reallocation on an '8266 and "have a word"

Anyone want to buy a brand-new completely unused custom stl memory allocator? It's an ideal self-teaching tool...