Esparto v3.3 released at last

AT last! A "quick fix " to patch up a problem in someone else's library has just ended up being a 2-month ground-up rewrite. I am so fed up with the whole thing, I hope you lot all like it enough to cheer me up: Watch the video then go and have a play with it. Esparto v3.3 released at last. Still some documentation being "backfilled" but the code should be ok and with 61 examples, you should be fine....

Esparto 3.3v feature preview

Esparto v3.2 soon to be released: sneak preview of new features

Esparto v3.2 soon to be released: sneak preview of new features

Click to Watch

Videos of Esparto v3 in action

Video no 1: Overview and features

Video no 2: The code behind the overview

Esparto v3 finally released!

Blimey, it's finally done. Esparto v3.0.0 released! (subject to some documentation additions and tweaks) Some fun facts:
281 files in 72 folders totalling 11.8MB
Main code is 16 source files totalling 3742 lines of code
Includes 47 example sketches totalling 3704 lines of code..I'm happy to call it 7500 lines of code...

https://github.com/philbowles/esparto

Why "The Cloud" is the bottled water of the Internet

I have been "on the Internet" since before most people had heard of it, and certainly well before the "World Wide Web" was invented. As a nerd, a geek, an early adopter and a technical purist (some would  - rightfully,perhaps - say "pedant")  I groaned when "ignorant " folk said "Internet" when they really meant "world wide web"...

It didn't take long for the two to become synonymous as the backwaters of the Internet wherein resided archie and gopher and ftp and smtp and pop3 etc were subsumed into the great beast that was "the web". "The web" -  where you could read email in a browser! And download files - from a browser! And....etc ....from a browser! OK, OK, I give in, I'll stop calling it the "world wide web" and start calling it the "Internet" just to make you happy even though I know it's wrong, but I insist on keeping that capital "I", ok???

What I will not do, no nay never is call it the fucking "cloud". And yes, I'm angry about it.

For one small moment after the execrable a*hole from marketing / sales invented this meaningless term (can you tell I don't like it, yet?)  - for one tiny, small, fleeting, worrying  moment - I thought "the cloud" was something new that I needed to know about. I had a huge technological FOMO experience and rushed to find out what it was. What I found was - if you'll excuse the expression - a cloud of obfuscation, buzzword bullshit bingo, marketing jargon and management-speak for (cough) - the Internet. Or, as I saw in recently in someone else's facebook post - in a rare moment of clarity for that platform - "someone else's computer".

Yep, that sums it up nicely. "The Cloud" is a salesman's invented concept to sell you access to someone else's computer that you don't actually need. To get that access, you use the Internet, but thats all too technical and and complex, so we jsut say "In the Cloud". As in where your head must be when  you buy bottled water.

In most developed nations, or indeed any country where the water is safe to drink, paying for bottled water is the result of a marketing man's greatest wet (literally) dream: Get people to pay for something they already get for free. Package it in toxic, planet destroying plastic, invent a sexy deceptive name evocative of cleanliness, purity (yeah those same requirements for the stuff that comes out of your tap) and charge money for it to gullible suckers who think it's different from the stuff that comes out of their taps - which is perfectly good anyway - when sometimes it's even the same stuff out of the same tap.

Yes, there have been several documented cases where unscrupulous* operator have taken mains water , bottled it, given it a fancy name and sold it at something like 10gazillion percent markup. The same stuff. People actually pay money for the same stuff that comes out of their taps. *It makes it hard to call such operators "unscrupulous" when the general public is so dumb and lemmingly free with their money...what next? Getting people to pay to wear your advertising logo on their clothes and shoes...damn, that ship has sailed...

Seriously, any time I see anyone buy bottled water I want to punch them really, really hard. I get a similar twitch when I see branded clothing, but that's less than it used to be.

Some may say that the music business are the kings of "reinventing the format" to make you pay again for what you already own - and that is true, but at least something has changed each time: CD was more robust and less scratch-prone and hissy, mp3 is waaay more convenient and portable etc

Neither bottled water nor "The Cloud" have that redeeming feature, both are actually worse than the equivalent you already paid for: either by pollution in the former case and massive security isses in the latter. But the principle of "pay up, pay up, and pay up again" is the same driver behind them all.

In the (g)olden days, you paid for a cardboard box with a CD and a manual in it. For "paid" read " often a very large sum of money" - but then you could sit back and use it for many a year. The provider could no longer milk you. Solution? Put it "in the cloud" and charge a subscription, now you are hooked into them for life. Open your wallet and repeat after me "Help yourself".

You would, of course, need to invent some reason why it was better - especially if it wasn't - but that's what marketing and sales are best at: fancy new names for the same old shit, subterfuge and glitz to part you from your hard-earned cash. Now you have been duped - sorry - persuaded to put your private data onto somone else's computer - sorry "in the cloud" you need security, privacy protection, backup etc etc all of which - of course - cost more. Its brilliant if you are a provider, bullshit if you are a consumer. Unnecessary, expensive bullshit at that.

A lot of brilliant, resourceful and respected people invented the Internet. No technician ever had a hand in inventing "The Cloud".

Remember, it is just "someone else's computer". Personally I prefer to keep my data on my own computer - its cheaper, and I'm not going to go bust or disappear overnight or sell all my data to the highest bidder or get specifically targeted by hackers...

"In the Cloud" = "On someone else's computer".

Esparto V3.0 nears release

Esparto v3.0 will soon be released: here's a little taster of the pin types available - v3.0 allows you to add them at runtime with no programing!

How real is time?

More specifically, how fast is “real-time”? The very phrase “real-time” is a deception – nothing in the physical world happens in “real-time” except time itself. Even if it’s only a few nanoseconds, the voltage on an input pin takes time to rise. How quickly after you press the button should the LED come on before you call the cause and its subsequent action to be “real-time”? Is it sufficient for it to be undetectable to the human eye even though that might be an age in computer terms?


Real-time is therefore a subjective measure, by anyone’s standards. The difference between the cause and the action is known as “Latency”. Take the case of an interrupt. The ESP8266 takes a finite number of clock cycles to recognise an interrupt and that’s after the inherent latency of the voltage rise time from the actual initiating real world-event. Already we have some latency in the system. How much? At its standard 80MHz clock rate, typical its 10 us - 80us but can sometimes be as long as 300us.


That is only the start of the story, as an interrupt on its own is useless without some user code to then do something with it: The Interrupt Service Routine (ISR).Any half decent programmer used to dealing with interrupts will tell you that you have to keep the ISR as short as possible. It’s to a) reduce the latency and b) reduce the opportunity for cross-process interference in the absence of OS-enforced isolation. It’s common to just set a shared flag (often in global scope) to signify “something has happened” and then have the main loop check the flag as often as possible. The ISR has latency, the main loop check adds more.


Finally, doing what’s actually needed e.g. lighting an LED (which of course has its own output voltage rise-time latency to add…) will add plenty more. The point here is this: there will always be a measurable delay between the cause and the subsequent outcome.


The real question then becomes “what is the maximum acceptable delay for this application before we can consider it to be real time”. I don’t know if there’s an “official” answer to that. Personally, I happily run my house lights and alarms with my home-grown firmware “Esparto” and you should too. I would also happily run your life-support system on it, but I wouldn’t necessarily want mine connected to it.


With Esparto, there is an extra layer in the GPIO pin handling, as it has to react to each and every designated input pin so that it can maintain the pretty flashing lights when the web UI is open. Obviously I have tried to keep it as lean and mean as possible but it has all the overhead of a websocket and then the network transmission time to the viewing device.


On top of even all that, there can be more latency added by the internal serialiser / scheduler deep in the code which can occur when lots of other stuff is happening “at the same time”. There’ll be a lot more on this later but for now take it from me that the pretty flashing lights are for information and entertainment purposes rather than precise measurement. There can occasionally be a tiny (but visible) lag between the built-in LED going on and the web UI pin lighting up. The faster you flash, the worse it gets. The more other pins you have active, the worse they all get.


To the casual observer, its real-time. To me, its real-time. To the purists, nerds and boffins, it may be far removed from what they define as real-time. In my own odd ways I’m probably a bit of all three of the last group, so I know I’m slightly deluding myself when I say “To me, its real-time” – but it’s good enough.


In the world of IOT that we inhabit, there are very few things that need to be "real-time". Interrupt handling is complex and difficult to get right, even for experienced programmers. Novices should stick to polling - the rest of your code will be taking the Lion's share of the processor and any "loss" of latency will not be noticed - really, it won't!

IF you are an expert and you know - for example through some specific hardware timing issues - that you need interrupts, then by all means use them. That excludes 95% of the folk I encounter in groups and forums, so for the rest of us, forget interrupts, you don't need them. Either stick to simple polling - or use a library like Esparto that has already done all the hard work for you

Esparto v2.0 finally released!

Yes, it's "out of the door" at last. After some final "stress" testing and the creation / testing of 32 (yes, thirty-two) example programs, v2.0 finally goes public.

Intervening health problems added 8 months to the date(!) but at last I can now relax somewhat- with a well-earned beer.

Get Esparto v2.0  here at github 

Esparto V2 almost ready! The new web UI part 6

The lower panel(s)

Run

This is literally where the action happens. Esparto is designed so that commands can come from several sources:

• MQTT topics
• the web UI
• from within the app itself using the "invoke" functio

The first is one of the most common forms of communications between IOT "things" in home automation. You obviously need access to an MQTT "broker" (fancy name for "server", really) either your own or public one. At this very moment, there is no way to enter a username and password - except manually in your own code - so until I get that fixed... your own broker is the best bet. I use mosquitto (which can be found Mosquitto.org download page) on a raspberry Pi.

But don't despair if none of those options are available, you can still do a lot with Esparto through its own UI, or from your own code. Want to change GPIO0 (D3)'s debounce value for example? Either call Esparto.invoke("cmd/pin/cfg/0/15"); or come to this screen, select the cfg option form the dropdown menu, add the /0 to the Topic line and type 15 into the payload field. Then hit "Simulate MQTT" - Esparto's internals that actually "do the business" are called with exactly the same message as they would have received from a genuine MQTT server. Simples.

As commands are received and actioned (from any source) the stats are dynamically updated. Soon - and certainly before release - I shall add "Alexa" as a source because Esparto is fully Alexa-compatible by pretending to be a Belkin WeMo when asked in the right way. All it can do is "turn on "<your device> or "turn off" the same. That's as much as a lot of devices do, anyway.

The "all" source is special one built in to Esparto allowing you to send a single command to all Esparto device on your 'net, as well as addressing each one indvidually by its device name. For example - while not necessarily advised - "all/cmd/reboot" will do exactly what you think it would.

Users can to subscribe to any topic they choose - including # wildcards - when called back in onMQTTConnect (see previous post in the series for a simple example).

In the demo, the user has subscribed to a wildcard topic like so:

  Esparto.subscribe("wild/#",[](vector<string> vs){ 
        string suit=vs.front();
        Serial.printf("Wilcard handler suit is , card is %s\n",(CSTR(suit)),CSTR(vs.back()));
        if(suit=="hearts" || suit=="clubs" || suit=="diamonds" || suit=="spades"){
          Serial.printf("You chose the %s of %s\n",CSTR(vs.back()),CSTR(suit));
        }
        else Serial.printf("Invalid suit %s\n",CSTR(suit));
    },"cards"); 

(They also included  (not shown) a simpler topic "flash" which they wrote to call the same code as Alexa commands call. So when Alexa is told to "turn on testbed" it has the same effect as MQTT command "testbed/flash/1" for on  and ...0 for off. )

He/she has chosen to only allow this wildcard topic from another made-up source "cards"  so "testbed/wild/party" won't work, but" cards/wild/animal "will.

And therein lies a slight oddity - and a caution. Esparto cannot predict the billions of permutations that come after ...wild/ - only the user can decide that. Hence for the system to work Esparto not only has to accept anything of that form, it also has to add it to the above table and count it and that has consequences.

The user code in the demo rejects any subtopic except (rather suitably) hearts, clubs, diamonds and spades and would like the payload to be a card from 2 to 10 or J K Q A. Lazily, it doesn't actually validate the payload, but then all it does is parrot ack to you what you send it, so no harm done in this contrived case. In the real world such appalling coding (I should know, I wrote it - deliberately to bring out these points of course) will almost certainly lead to a crash if unexpected, unvalidated input and/or gibberish is fed to any wildcard topic - so be careful. Trust no-one, and validate everything to within an inch of its short life.

Another consequence is that the more wild rubbish you send that Esparto is duty-bound to accept, the longer that list will get, the slower the UI will become until finally Esparto's self-protection mechanisms will cut in and reject  everything from all sources until some memory is freed somehow. This will certainly cause erratic behaviour and possible meltdown at the nuclear power plant, so don't do it. If no memory can get freed, then your Esparto app will die a slow lingering death till you want to reboot it. When you do, remember it's your fault, not Esparto's. 

While this a shorter section than some others, this pane is probably one of the most useful of Esparto's many features. There is neither the space nor the time to go into detail about each of the commands and new ones are being thought of as I type...

You may find that all the ...dump... options are missing from the final release, as they are 90% used in debugging. It will be done in such a way that even average programmers will be able to hack into the code and just turn on a #define and recompile. They do steal heap though, whic is already in short supply so don't say you haven't been warned. On the other hand if I can trim some fat from elsewhere during my final code tidy, I might leave some/all of them in. Invoke("cmd/watch/this/space"), you might say.

Others are just fun to play with: .../pin/flash/... , .../pin/pwm/...  and  .../pin/pattern for example/...

Download the release when it's out "soon", edit in your own SSID/password/device name and give all the topics/commands a try - it's what Esparto is for.  Or wait 3 minutes for it to give up and go into AP mode then configure it with your phone.

And enjoy it!

Esparto V2 almost ready! The new web UI part 5

The lower panel(s)

Config

Now we start to see the real power of Esparto coming out. It has a configuration system where name/value pairs are automatically saved to SPIFFS (the ESP8266 Flash file system) as soon as they change and persist into the next reboot, i.e become permanent. Well, until the next factory reset, at least.

The demo code has a Latching push button on GPIO0 (Arduino digital pin D3) and a very "noisy" and sensitive  sound sensor (i.e. a high number of thousand IOs per second at the slightest cough) on GPIO12 (D6). It also configures the BUILTIN_LED for output. On a Wemos D1 mini that the demo was built on, this is on GPIO2 (D4). The hardware setup looks like this:

void setupHardware(){

  Esparto.Output(BUILTIN_LED,LOW,HIGH); // start with LED OFF                   

  Esparto.Latching(PUSHBUTTON,INPUT,10,buttonPress); // 10ms of debouncing

  Esparto.Raw(D6,INPUT,[](int s){ Serial.println("Do nothing"); });

  Esparto.throttlePin(D6,19);

}

I trust your first taste of the "esparto Way" wasn't too shocking or difficult? setupHardware() is equivalent to the standard Arduino-style setup() and you do the same kind of thing here as you would there - almost. You just do it Esparto-stylee - so for instance, no WiFi.begin and delay loops* Esparto is already connecting to your SSID "in the background" to speed things up.

Also you won't see pinMode calls: Esparto knows what mode to set automatically from the type of Esparto SmartPin you define. So really all we have is one line per I/O device, and often that's all you will need. The only "odd" or "tricky" thing is the throttlePin call. We'll get to the strange syntax in a minute, but first, what exactly is "throttling" and why do we need it here?

It is described in great detail in part 3 of this series, so if you want to know more read up on that first, but for now a simple one liner is that the sound sensors fires far more data than any tiny device can easily cope with -Esparto being no exception - so we have to slow it down, or "throttle" it. D6 is our noisy pin, so we tell Esparto only to allow through 19 of the thousands of 1s and 0s per second.

The reason this is such a low figure is explained in the earlier article. Your LED will still flash vaguely in time with your bangin' house or lounge jazz tracks...ish. Now to that weird syntax...

C++ Lambda functions:

See what? If you don't know about these already, ask Mr Google about them because you will quickly come to love them as much as I do. They are particularly good for callbacks and a lot of your code needs to be in callbacks so now is a good time to learn how to use them. If you are already frightened, fear not: you don't have to use them, the old-fashioned way still works. I will show you what that would look like in a moment and I'm sure you will soon be seeing the benefits of the new-fangled way.

What we want is for Esparto to tell us when pin D6 changes and what is has just changed to: a 0 or a 1. So we need to give Esparto a function that returns nothing (void) and takes a single int parameter, which holds the new state when the pin changed. Ordinarily we'd write:

void namedFunction(int s){

Serial.println("Do nothing");

}

and then our old-fashioned way would be:

Esparto.Raw(D6,INPUT,namedFunction);

But:

• It's more typing
• We have to invent a name for our free-standing "normal" function that doesn't do a whole lot
• namedFunction can live anywhere in your code base. If your code is large and you are anything like me, it can sometimes take a while to find, by which time you forgot where it was called from!

Let's break down the "new" way (it isn't new at all, it's been around since at least 2011)

,[](int s){ Serial.println("Do nothing"); }

[] = this is a lambda function - it has no name

(int s) = same as before, it takes an int parameter called s

{ Serial.println("Do nothing"); } = this is what the function does, its body. any valid C++ code can live inside the body including if/else blocks, other lambdas etc.

Not too painful, I trust? In summary it's a function with no name (an "anonymous function") that is "bolted in" to the place that needs to call it, instead of having to live outside on its own. It has many benefits:

• Less typing
• Less names to remember
• Lives alongside the thing that defined it and needs it: makes code more easy to understand and saves time hunting
• You can do things with it that you would never have dreamt of, like pass it, lock stock and barrel as an object to another function that can then call it on your behalf! That is beyond the scope of this post, though. Ask Mr Google.

I mention these in some detail because a lot of the example code uses them, for all the reasons above, and because I love them. Esparto could not have been written without them. I hope you come to love them too, and soon - they make working with Esparto a breeze and they're not really that tough are they? Welcome to the 21st century!

Why do nothing?

The demo is purely to show the raw LED beating closely-ish in time with either some music, clapping of hands, whistling, dogs howling etc. Since Esparto does all the checking for changes and SmartPins underneath does all the flashing automatically, there is nothing else for our demo code to do. This shows how powerful Esparto is. Ordinarily the lambda is where you would put your special code that makes your app different from the rest. I do exactly that with the Latching button, which starts and stops the LED flashing by calling buttonPress which you haven't seen yet, but is here in all its glory:

void buttonPress(bool hilo){

  if(!hilo) {

    uint32_t rate=Esparto.getConfigInt("blinkrate");

    Esparto.flashLED(rate); 

  }

  else  Esparto.stopLED();

}

User-defined config variables:

And in Esparto.getConfigInt("blinkrate"); you now see the Esparto magic starting to happen. I challenge you to look at the screen shot above and guess what happens when you change the value. Go on, have a go!

If you said "I bet the LED starts flashing at the new rate automatically", you're obviously catching on but you'd be wrong. Only because I'm teasing and you haven't yet pressed the pushbutton to start it flashing at the old rate in the first place. If you had already done that then yes, exactly correct: the LED instantly starts flashing at the new rate, well done! It's now no great leap of faith to correctly assume that changing the debounce value will, er, change the debounce value of the Latching button. You are getting a whole lot of functionality for free here.

But there's more: next time you reboot, the value will be brought back - the config system saves the value whenever it changes, you have nothing further to do. The BWF parameter just made up, to play with, does nothing, isn't used anywhere and you can type what you want in there just for the fun of seeing it survive a reboot. If you want real magic, read the next section on the run panel...

Yet more: send the command testbed/flash with a payload of 1 to start and 0 to stop from an MQTT client and guess what - correct the same thing happens as if you had pressed the button physically yourself. The code to make that happen? Here:

void onMqttConnect(void){

  Serial.printf("T=%d USER SAYS MQTT CONNECTED\n",millis());

  Esparto.subscribe("flash",[](vector<string> vs){ 

    Serial.printf("Doing my thing with %s\n",CSTR(vs.back()));

    buttonPress(!atoi(CSTR(vs.back())));

    });

}

Dont worry about the "vector" stuff, that's more C++ magic that is going to make your relationship with Esparto a much more fruitful one and will be covered in the future. For now be happy that you have just avoided 3 months of tearing your hair out and a learning curve like the side of a cliff, while getting an already pretty capable system "for free" from a mere handful of lines of code!

System config variables:

Anything starting with a "~" is a system variable which Esparto relies on to function properly. So:

1. never use "~" in your own config names
2. while you can put whatever you like in your own variables as long as your code knows what it means, the same cannot be said for system variables
3. never change a system variable unless you know what you are doing, and why!

Some system variables are easy to understand and make sense for the user to change. The ones I have chosen to expose for the demo are like that. By the time the full release comes round there will be a lot more, and they won't be as nice. I can safely predict that even when you read the "advanced guide" with a full explanation of what each does, you still won't want / dare / understand how to change them, so - just don't. Ever!

~fb2Ap: 

Is the millisecond count for the amount of time to wait for the WiFi to fail to connect before "falling back" to AP mode and offering yourself up to a phone, tablet etc to get in and configure a valid set of WiFi credentials. The demo has 3 minutes = 180,000 microseconds = 180 seconds. You may want less or more: feel free to change it to a sensible value that works for you.

~lh:

Is used to log the value of the heap every second to an MQTT broker, just in case the 3-minute graphs on the system page aren't enough. 1= start, 0= stop. It will publish /testbed/heap with a payload of the value once per second until you stop it, either by changing the value back to 0, publishing testbed/cmd/logheap/0 over MQTT or reading on to the next section on the run panel...

~mqXXX:

Unsurprisingly, the IP address, port and retry failure re-connection interval of your MQTT broker. Some day soon I will add ~mixer and ~mqPass to enable you to connect to an authenticating remote server. Some day...

Don't ask me about (or mess with!) the as-yet-unseen ~jitter variable - it's the plus / minus entropy timing spread adjustment factor to minimise asynchronous collision probability in the autoStats derived timer reset function. It is currently set to 10. Still fancy seeing what happens if you change it to 11? Or 243? No, I hoped not.

*Ever. No delay loops ever. They are bad, they break asynchronous libraries, stop other tasks from running and are generally BAD STYLE. Do not ever use one in an Esparto callback (or at all, in fact) you simply don't need to. If you think you do, trust me, you are wrong. There is always a better way. call Esparto.once(<x mSec delay>, functionToRunSoon); for example. Don't ever call delay(). Need I say it again?

Esparto V2 almost ready! The new web UI part 4

The lower panel(s)

Info

This is mostly self-explanatory but also raises a few points that are well worth knowing if you intend to become an "Esparto Expert".

These are all static values at the heart of the system, some of them permanently, some for example the IP address for this boot of the system only. They cannot be directly changed by the user. For some that can, see the next section.

Hardware Type:

This lets you know what's running "under the hood". It can be any one of these:

• ESP-01 (but why would you bother when there’s…)
• ESP-01S
• Wemos D1
• Wemos D1 mini
• Wemos D1 lite (and thus probably any other ESP8285 device)
• Wemos D1 pro
• NodeMCU 0.9
• SONOFF Basic
• SONOFF S20
• SONOFF SV

Esparto has been tested on all of the above. It will probably run on anything with an ESP-12 in it, but obviously I can't test every single device on the market/ If you want to send me one to try it out and modify if necessary...

I will be very interested to hear of anybody getting it running on any other platform e.g. NodeMCU 1.0 will probably work, as will (I expect) other SONOFFs

Unique Hardware ID:

This is the last 6 digits of the MAC address and is commonly used in new-out-of-the-box scenarios as a default name before choosing your own and setting your SSID / Password credentials the default name of the demo device would be ESPARTO-17D383. See part 2 earlier for more detail on this value and advice renaming your device to replace it.

IP Address:

Need I say more?

Flash Memory Size:

Same as above, except the answer is "Yes". 

Esparto weighs in at  over 410k - it has a lot of functionality and features. To support OTA updating (and who wouldn't want that?) a "sketch" (app) has to be able to fit into half the available flash size. On smaller devices e.g. the SONOFFs you get 1MB thus 512k is usable if you want OTA (and you do!).

As you can see things are already starting to squeak, so you need to keep your own additional code
small, efficient and light-weight. There is also a very limited amount of heap left. Esparto starts up with about 27k free, and that can go up and down rapidly - see the graphs in part 3 of this series for an example. Keep your heap use to a minimum and guard any heap-using routines with a check in what's free first if you want to avoid crashing (again, you do!).

The good news is that Esparto does so much for you that your own code will be small and consist mostly of short callback routines that Esparto will execute at the relevant time on your behalf. There is no loop() function and no setup() function. You will rapidly get used to doing things the "Esparto Way" once you see how easy it is.

H4 library version number:

If you want to get further than a simple "Blinky" it helps to understand the structure of Esparto. It is built from 3 main libraries, H4, SmartPins and Esparto itself.

H4 which handles all the timer functions, scheduling, task separation and "slip streaming" of asynchronous functions into the synchronous task queue which runs on the main loop. No more WDT resets, no more "volatile"s. When your task runs, it is (almost) the only player in town and the H4 library makes sure you have to try really really hard to break things or upset other tasks.

It comes with 7 of its own examples demonstrating how each and all of its functions work. Esparto "encapsulates" H4, so all of the H4 functions will appear to you as identical Esparto functions, so you do need to understand these first.

SmartPins library version number:

See above. Note the version shown is incorrect - by the time of  release it will also be 2.0.0 (actually it is, but I forgot to update the version number field before the demo - my bad!)

SmartPins as its name suggests manages all the input and output pins for you. It is what enables Esparto to give you the fancy real-time flashing LED display for all the pins. It also does everything you could ever want to do with a pin, including debouncing, interrupt handling (although there are good reasons why you would probably never need to use it), rotary decoding and much more.

The Encoder input type lets you manage a rotary with a single line of code - you tell it the name of a variable, and whenever you access the variable it will automagically have the current decoder value in it. One line of code! It's my favourite Esparto feature: most of my own mini-apps have some kind of "tweak" factor using a rotary, it's so easy. Some even have two...at the extra expense of one more line of code...I'll stop now, I think you have got the point.

SmartPins comes with nineteen sample program covering every in and out (literally!) of the many types of input modes it supports. It also has access to all H4's functions and relies on it 100% to function. Pretty much every example has at least one or two H4 features though of course they appear seamlessly as identical SmartPins features.

It is important then to work through the examples in order to fully understand the power and flexibility of Esparto, because in the same way, all Esparto functions are automatically the same as all SmartPins functions.

Even seasoned programmers will benefit, as Esparto works in a very different way from 99.235% of all the thousands of sample sketches you will find online. You need to learn the "Esparto Way", but for those with experience it won't take long at all.

Just as an example, here's the code (with comments removed for brevity) for the simple blinky. "Simple" includes having a fully debounced on/off switch unlike 99.476% of other blinkies.

NOTE: 

While H4 and SmartPins both have visible setup() and loop() functions, Esparto does not. There are two reasons for keeping them in:

1. To make the early examples more readily recognisable and ease you in to the "Esparto Way" and "chunk up" the amount of learning at each stage into bite-sized pieces.
2. To enable you to use them on their own without the full Esparto, although I can't think of any reason why you would want to unless you are the kind that likes to make things deliberately hard for themselves

#include <SmartPins.h>

SmartPins smartPins;

void buttonPress(bool hilo){

  if(!hilo) smartPins.flashLED(250);

  else smartPins.stopLED();

}

void setup(){

  Serial.begin(74880);

  Serial.println("LED will change state (flashing/off) on each separate button up/down press");

  smartPins.Output(BUILTIN_LED);

  smartPins.Latching(0,INPUT,15,buttonPress); // GPIO 0 + 15ms of debouncing

}

void loop(){

  smartPins.loop();

}

I hope you will agree both that it's pretty easy and also that you get "a lot for your money" for very little coding effort. That principle underlies the whole of the "Esparto Way": Esparto does 90% of the "heavy lifting", you plug in the remaining 10% which is specific to your IOT / home automation app. Esparto allows you to concentrate on just the code that's important to you - all the hard stuff "just works"

NBoot & Code:

These may be the first indication of a (hopefully very rare) problem. NBoot is the number of times this device has been rebooted and "Code" is the reason why. If it has just been freshly programmed then (as has the demo device) then it will read ESPARTO_BOOT_UNCONTROLLED.

What this means is that it was not shut down by user action, but forcibly rebooted, as the IDE does. You will also see this code if the device crashes for any reason.

If you click the Reboot button,. the code will become  ESPARTO_BOOT_UI. If you send an MQTT command e.g. testbed/cmd/reboot the code becomes ESPARTO_BOOT_MQTT and son on, although obviously you will replace "testbed" with your own device name first.

If you see an increased boot count and a reason you don't expect - something has gone wrong!

The "tXXX" values:

These measure the amount of milliseconds since boot up when:

tHW: 

The time after which your sensors, buttons, relays, remote controlled Gatling guns etc become ready to run. One of the fundamental design goals of Esparto is that your hardware should operate a) as early as possible b) whether you have a WiFi connection or not c) all the time, always.

Even if - as happens in the real world - bugs occur and the occasional crash occurs, your hardware will be back up ready to go in about 125 milliseconds. Impressive, non? It's one of the reason behind why Esparto won't let you play with setup() and loop(): it has quite a bit of complex setup of its own to do, and it has to happen fast, and in a very specific order.

tWiFi:

The time after which you can load up the web UI because your device now has a valid IP address.

tMQTT: 

Similarly, the time after which Esparto is actively listening for MQTT commands, both its own any any that you choose also to listen for. All Esparto command start with "cmd", so you must not use this in any of your own topics, or who knows when that Gatling gun may go off in error?

High values of either tWiFi or tMQTT may be early indications of problems with your router, network or MQTT broker. Or they may just be a sign of a slow network - only you will know. Personally, I'd worry about anything much more than the demo values. Again, I think 3.2 secs from power on to receiving MQTT commands is "in the zone".

Esparto V2 almost ready! The new web UI part 3

The lower panel(s)

System

This panel will be used mostly for debugging your Esparto App. The ADC graph shows the raw value of the A0 pin from 0 to 1023 and can be useful when calibrating any sensor attached thereto.

The Heap graph:

The heap graph is probably the most important tool here. The ESP8266 has limited heap space and many of the underlying libraries chew up a fair amount. The main AsyncWebserver library is also very sensitive to low heap situations and despite it being a truly marvellous piece of software, it will crash unceremoniously if there is very little free heap space. I have not gone to the lengths of calibrating it exactly, but below 5 or 6k seems to be in the danger zone. Get the software on github.

Running out of heap is fatal in all circumstances and great care must be exercised in your code to make sure you use as little as possible. The graph is designed to help you do that. The window is 3 minutes wide, which is more than enough time to spot any problems. Trust me, when they happen, they happen fast!

Esparto has a built-in heap guard so that it will not let any process start if there is less than a defined amount of heap available. The value is configurable at compile=time and I usually run with it set to 20% of the initial free heap. You may need to "tweak" it while watching this graph to optimise the figure for your own app.

The downside of this is that once the limit is reached, any / every event requiring a task will be silently ignored, This can make your app behave strangely: LEDs may stop flashing, sensor values may not get sent to MQTT, your buttons may not work, or worse: work on the down stroke, but not on the up. The point is that whatever happens, the app will not crash. It reserves at least enough heap (I hope!) to be able to call up the UI panels so that you can spot the problem with this panel, fix it perhaps by changing a configuration value on the config panel or use the run panel to issue a command to stop the bad behaviour, while letting the good stuff continue. That's the idea, anyway.

The Q panel shows the size of Esparto's task queue. Most of the time this will read zero, i.e no outstanding tasks. Technically it should show at least one, but it clears the queue so quickly and the stats are only refreshed once per second, so they "miss" the occasional internal background management task. This raises an important point: all of these statistics (apart from the ADC) are indicative only: they cannot be 100% accurate (nor do they need to be) by simple virtue of the lag between the actual event, the network and the browser. They are certainly accurate enough to help you spot problems very early on and then tune them out.

Again, the queue can be sized at compile-time and some experimentation may be needed to maximise free heap with a small queue - but not so small that tasks start stacking up or being "throttled". The same caveat applies: once the queue limit is reached, tasks will be ignored until the level drops below the limit. All the same glitches as above may or may not occur, but still: no crash. In any event it is 90-odd percent certain the heap will be throttled well before the Q grows out-of-bounds.

Warning!

There is one sure-fire way of busting the queue: scheduling a repetitive task that runs for longer than the scheduling period. For example running a job every 5ms that takes 6ms to complete. It doesn't take much thought to realise that this situation is never going to end well. The queue will grow exponentially and given that 10 or 20 is a sensible practical limit, the rate of growth before throttling occurs will be of the order of milliseconds. You already know what starts (or more often "stops") when these limits are reached.

Another "don't try this at home, kids" method is to schedule a taks that schedules another, which schedules...etc in a chain which is longer than the queue size (minus 1 or 2 for system tasks). Granted, its rare and difficult to do, but here is almost certainly a better and safer way, so if you are thinking of long chains of tasks to get some wacky timing working, think again...

Not least for the fact that again timings are not 100% reliable: they depend on other tasks in the queue. If your new task to start in 10ms slips in behind one that is scheduled to run before yours and takes 15ms then yours wont start for 25ms: his 10 and your 15. If however the queue is empty - which it usually will be - then yours will start on time. The moral of the story is: don't run a nuclear power station or your mother's life support system with Esparto. Most other stuff will work just fine.

The Pins Graph:

Pin activity is the main culprit in heap depletion: Esparto has quite a lot to do when a pin change occurs. It has to check to see if the pin is throttled (more on that in a moment) and if so, discard inputs over the throttling limit, if the input survives that, Esparto has to light the appropriate raw LED then allow the specific pin-type handler to decide / calculate / guess (only kidding) the internal cooked state, get the value and light or extinguish the corresponding cooked LED. And update the pin statistics, and...

All of which uses up heap space. The snapshot above was deliberately chosen to show how a burst of high activity on the pins causes an immediate and severe drop in the free heap space. In case you are interested, it was caused by flashing 3 LEDs at a high rate while simultaneously "listening" to a sound sensor like this:

Buy on ebay for about £1

on pin D6 while Motorhead's "Ace of Spades" was playing at high volume. The track was chosen specifically because it is a "wall of noise" and causes the sensor to throw literally thousands of transitions per second into the pin, which brings us nicely on to Esparto's approach to pin throttling...

PinThrottling:

Very few ESP8266 apps can sustain thousands of transitions per second (peaking at 11000+ form Motorhead when turned up loud, which is - of course - the only way to listen to it) while also managing other hardware and a dynamically-updated web UI. Esparto is no different: it's not magic! It depends on underlying libraries that have limits. For example the AsyncWebServer library can only deliver about 20 requests to the browser per second. This of course depends on how fast the browser can consume them but when you look at what Esparto is doing on this panel alone, there's a lot to get through. By empirical observation, 21 inputs per second is the tipping point at which the heap starts to drop like a heavy rock. The chosen tune has no let-up, once the descent begins there's no way back before a very rapid heap exhaustion crash, except to  a) throttle the heap b) throttle the cause of the problem rather than the symptom: the rate of throughput on the input pin. Esparto does both.

So the LED is never going to flash in time with the beat. Cutting the rate from 11000+ to 21 is obviously some heavy clipping, but it is essential to prevent a crash. It's actually worse: because of some background tasks, even at 20/sec the heap depletes, albeit slowly. It doesn't actually stop entirely until 19. Unfortunately the ESP floating point code is so big and Esparto already weighs in at 410k that Esparto can only deal with integer arithmetic - for speed as much as size - but size is what prohibits "proper" math. So when calculating the throttle sample rate, 19 gets rounded down to 10.

The count on the pin is sampled 10x per second which is 100ms between checks. This is a fine balance between early notification, accuracy and impact on other process. Pin D6 is throwing 1s and 0s in at 11000 per second (that's 91 microseconds between) so the first time we get to check the pin count, it's already up to 1/10 of it maximum which is 1100 - a heap-busting amount of activity.

The maths behind this problem is much deeper than we need to get into here, but the result is that in its current guise, Esparto can only throttle pins in multiple of 10. Doing it at 20 would be accurate, as would 30 - or 10. Anything in between is rounded down by the integer divide. Rounding it up wouldn't make a lot of sense for a limit, now would it?

As a slight aside, the reason that pin D6 (GPIO12) is coloured yellow in the UI snapshot above is to indicate that it is throttled.

Esparto V2 almost ready! The new web UI part 2

The lower panel(s)

WiFi configuration

Recall the original web server UI image:

Starting on the top row (but right-to-left) the Reboot and Factory Reset buttons should be fairly self-explanatory: reboot "does what it says on the tin". Factory Reset erases all WiFi configuration,  all user-defined configuration data (to be covered in more detail in the future) and reboot completely "clean".

Before we go on to the main panel selector button, a quick word about "system health". While ever the ESP8266 is running the heart icon will throb once per second (using some nifty - but quite complex - CSS). The LED to its left shows green while a connection to your MQTT server exists and red if it goes offline for any reason.

The main set of icons are the panel selectors. Clicking on each causes a new panel to open and the rest of this series will cover each panel individually, starting here with the default WiFi configuration panel. Again this is reasonably self-evident, with a couple of important points regarding how Esparto works.

If your code does not define the SSID / PSK then the device will fall back to AP mode with a captive portal allowing you to configure them dynamically, Once configured and connected, they can be changed at any time and the device will continue to run without rebooting, using the new credentials.

If on the other hand you compile the credentials in, then it will connect immediately to your router in STA mode. Once connected, if you subsequently change the credentials, the new values are stored, and will be used in preference to the compiled defaults until you perform a Factory Reset.

Similarly, if you do not compile in a device name, it will default to ESPARTO-<chip ID> where chip ID is the last 6 digits of the device MAC address. You can rename it to something more sensible using this panel once it has booted into STA mode. Again, once changed via this panel, the new name will take precedence over both the default chip ID option and / or any compiled-in value.

Choose the name carefully - NO SPACES! - as it is the name by which it will be known on the local network: "testbed.local" in the above example. More importantly, it is the name by which all MQTT commands are prefixed: sending testbed/cmd/reboot from your broker will...you guessed it.

Incidentally if you have many devices, you can also send all/cmd/reboot to...well, you can probably guess that one too.

Esparto V2 almost ready! The new web UI part 1

Soon, you will be able to get your hands on Esparto V2 and start putting together robust, flexible and functional IOT and home automation apps of your own with minimal effort. Let's dive straight in to the built in web server and show how it allows you to do pretty much anything you could want to with your app.

Before we begin

Esparto introduces a new concept in GPIO input pin management: "Raw" and "cooked" pin status. It is easiest to explain when consider two common types of of switch, a "tact" or momentary push button which springs back when you release it, like this:

Typical tact switch - click to buy 100 for less than $2

And a locking or "toggle" switch which stays down once pressed and has to be pressed again to release it, like this:

Typical toggle switch - click to buy

This second type is also sometimes called a "latching" switch - it "latches" in one position until pushed again.

Let's consider the state of  a GPIO pin to which either of these is connected. First the tact switch will go from 0 to 1 and then back to 0 again very quickly. Do you want to react to the 0 or the 1? The toggle switch on the other hand will go from 0 to 1 and may well stay that way till you power off your device. Or it may go back from 1 to 0 a few seconds later when you press it again. 

Now imagine you only have tact switches, but you want your app to behave as if they were toggle switches.You would need to monitor a 0 to 1 transition and then hold some internal state denoting "on". The next time you get a 0 to 1 transition you would have to remember is was already "on" (in your program's mind) and obviously set your internal state to "off".

Conceptually, we have created a difference between a 1 state and being "on" as well as "0" no longer meaning "off"...enter the Esparto method of "raw" and "cooked".

Raw is simply the current binary value of the pin. Cooked is the on/off state your program needs to know. In the above example, if you tell Esparto your tact switch is a "Latching" switch, it will do all of the above for you, i.e. the raw state will go from 0 to 1, but the cooked state will go to 1, or conceptually "on". When you press and release again, the raw state will (again) go from 0 to 1, but this time Esparto will set the cooked state to 0 or conceptually "off".

Once you have got your head around that, imagine a PIR sensor. Typically they will re-trigger themselves and stay "on" while ever anything big and warm (like a burglar) moves within their field of view. They will stay that way until a certain time has elapsed after the last detected movement. Digitally, we might have something like:

1...0....1........0.....1....................0

where the dots represent the passing of time. The first 1 might be our burglar entering the garden, so on goes the security light - he freezes and the sensor drops back to zero, but the light doesn't go off yet because the gap between the 0 and 1 is too short for the timeout. Then he moves again, for a longer time and then jumps back over the fence. Many dots later the PIR has detected no movement and so turns off. 

Esparto has a "Retriggering" type where the raw pattern will be exactly the same, but the cooked value will be:

1.................................................0

Which is to say that you will get told when to turn on the light and when to turn off the light, which - let's face it - is all you wanted to know in the first place. Esparto takes care of resetting the timer on any intervening raw inputs, but only notifies you via the cooked input when something "interesting" happens.

Debouncing

To complicate matters, no switch in reality actually behaves as simply as described above, they "bounce". This means that instead of the tact going 0...1 it might go 01010...1..0......1, bounce back and forth as the springy mechanical contacts settle.(just thing of a carton boi-oi-oi-ongggg! noise). It might look something like this:

If you hook that switch to a light, it will flash on and off horribly before staying on. The tricky process of making sense of all those ups and downs is called "debouncing". There is much more to it than this, read more here if you are interested, but the good news is: Esparto takes care of all that for you too. No matter how bouncy the raw switch gets, you will get a clean cooked 0..1 every time.

Now we can return to image of the web UI: the top row of LEDs are the raw pin values and the bottom row (with only two LEDs in it, because the demo code only defines two pins) is the cooked value. In between is the mapping between ESP8266 GPIO numbers and Arduino digital pin numbers - very handy to make sure your wiring is correct!

The LEDs flicker as close as possible to the real-time status of the pin so you can actually see what your code is doing.

In part 2 and onwards, we will look at the lower panels and the wide range of facilities that Esparto provides.

Esparto V2 observation on AsyncWebSocket performance and limits

Esparto V2 nears completion! During the final testing I had some problem with crashes, and traced it to a high rate of writes to the AsyncWebSocket. If the user requests socket writes faster than the browser can consume them a queue / backlog builds up and the free heap decreases rapidly until...bang!

That's no great surprise, but it may help other folk to know what the limit is. Bear in mind that the following examples were deliberately throwing a sustained rate of writes with no pause for recovery. In the normal course of events where socket writes are fairly random, there won't be so much of an issue. My app sends socket writes to flash the pins on and off in "real time"  and also updates the sliding graphs, the UpTime and the pusling heartbeat. So there is a continual "background" of 3/sec even with no GPIO pin activity.

Esparto V2 web UI, dynamically updating system panel running on Wemos D1 min

If you connect a noisy sensor to an input pin (e.g. a sound sensor that fires thousands of pin transitions per second) a crash is the obvious result and so each pin can be "throttled" to avoid problems. It was in trying to find a "safe" throttling value that I generated the graph and I hope others will find it useful.

As can be seen, once past about 21 writes/sec, things go down hill rapidly (but predictably).  If your app does not have heap protection and/or throttling then you are going to run into problems if you sustain more than that amount. If you know your free heap then the graph will allow you to calculate what rate is safe for any given period.

For example, say you start with 30k free heap. Some part of your code fires a burst of socket writes at 50 per second. Reading up from 50 and then back to the vertical axis you will see that about 2kb heap per second will be consumed, giving you 30 / 2 = 15 seconds before you crash.

If you can organise your app so that it backs off before that time and allows the heap to recover back to 30k then you can safely repeat the process.

Also you can see that once the rate drops, the heap recovers rapidly, so bear this in mind when designing you app and try to minmise socket writes.

I created a lot of problems for myself early on by trying to load the whole UI as a monlithic block, with 40,50, 60 or more socket writes to set it up as soon as the socket opened. Trying to get round this problem led me to the rate throttling method I now use, thanks to the data behind this post.

A technique I use to update all four sliding graphs (1x per sec) as well as the current, min and max figures for each graph is by combing all those values  into a single write and allowing the browser Javascript to parse, spilt and distribute them accordingly.

Another technique is to build in pauses between groups of writes. I use "lazy loading" of the web UI. It loads in 5 sections, each doing a chunk of work then scheduling the next chunk a short while later. The "short while" is of course calculated using the above graph! Yes, this makes the UI a little slow to load, but it doesn't crash!

Further, only the "active panel" is loaded on inital AsyncWebServer request. In my case it's the WiFi configuration panel (curently hidden behind the graph panel) . Each of the six icons at the top left of the lower panel will cease any socket activity to the current panel, hide it, show the new panel over the top and start any dynamic updating with socket writes for the new panel. Thus socket writes are only sent for the panel that is currently visible.

For further info feel free to contact me @ esparto8266 -at- gmail.com

Esparto v2.0 - sneak preview: Inside

"A picture is worth..." as they say:

The following 21 lines (one of which is a comment...) are all you need to turn a Sonoff Basic, S20 or SV into an MQTT device with a web interface...etc etc as described in the previous post "...outside". If you don't want diagnostics, you can lose the Serial,begin and cut another line.

The Sonoffs have a push button on GPIO0 and a mains relay on GPIO12. That's all they have, hardware-wise

#include <ESPArto.h>
// ToiioT-Etage is my SSID, pw="" (I live in the forest) my raspi mosquitto is on 192.168.1.4
ESPArto Esparto("ToiioT-Etage", "", "esparto666", "192.168.1.4", 1883); 
void buttonPressed(bool hilo){
  if(hilo) toggleRelay();
}
void mqttSwitch(String topic,String payload){
  toggleRelay();
  Esparto.publish("state",digitalRead(12) ? "ON":"OFF");
}

void setupHardware(){
   Serial.begin(74880);
   Esparto.Debounced(0,INPUT,15,buttonPressed); // 15 = ms debounce time
   pinMode(12,OUTPUT); // relay / switch
}
void onMqttConnect(){
  Esparto.subscribe("switch",mqttSwitch);
}
void toggleRelay(){
  digitalWrite(12,!digitalRead(12));
}

And the code above is all they need, and I ask you: "What could be simpler?"

True, you will have to physically FLASH upgrade it first time with a FTDI adapter, but after that, Esparto will update itself automatically as needed. It will appear on your WiFi network as esparto666.local and respond to an MQTT "switch" command, by toggling the power relay and will reply with an MQTT "state" message with a payload of "ON" or OFF". It will reconnect after any network failure and all the while, the manual button will still turn it on an off.

Plus it's inside your own network. No snazzy (but often rubbish) App to download. No security problems. No worrying if XYZ corp go out of business and close their cloud, that your lights will never work again...If you can use a web browser, you can control it. If you have an MQTT server, you can control it in much more detail. If you have a NODE-RED server, you can start to do really clever things with your whole house.

Let's look inti the code in more detail (shouldn't take long)

It doesn't look much like a typical Arduino sketch. There is no setup() function and no loop function. Esparto takes care of both, to make sure things are done in the "right" order and to prevent your code from accidentally breaking things or stopping it working.

Your code is all driven asynchronously by Esparto using callbacks. If you don't know what that means, you need to read the sidebar articles under "Essential Information". It starts with setupHardware. This is where you do what you'd normally do in setup. Having said that, much of what you'd "normally do" isn't needed any more.

Esparto.Debounced(0,INPUT,15,buttonPressed);

Tells Esparto that you want the button on GPIO0 debounced (for 15ms) and to call buttonPressed when someone pushes it or lets it go - i.e. when it changes. When it goes HIGH (the button on a Sonoff is "reversed" in sense: it goes LOW when you press it and HIGH when released) the relay is set to the opposite of what it is now. If its already on, it goes off etc - and that is the same as the standard firmware that it comes with when you buy it.

When Esparto has established a valid MQTT connection it calls onMqttConnect. Here, your code tells Esparto you want to receive "switch" topic message and when it gets one, it will call your code in mqttSwitch. As for a button press, you call toggleRelay which then publishes the current switch state to MQTT.

"And that's that"...as they also say.

Adding sensors to a "homebrew" board and adding lots of functionality on top of this is going to get more complex of course, but Esparto is designed to take a lot of the hard work out of that process too. It has a lot of "Hooks" where you can add callbacks in exactly the places you need to create a new "layer" of your own HA system on top of Esparto. That's how my own Chez Toi ioT system works: 90% of the code in each device is Esparto. Esparto has 9 different types of input pin it can manage for you, including rotary encoders and each of those only require one or two lines of code.

As an example my truc firmware when I fisrt wrote it was about 1200 lines of (pretty hairy) code and a lot of bugs. Now it handles GPIO on every pin of a WemosD1 and runs temperature PIR, sound, light, button and touch sensors. It also controls 433MHz RF switches, and it auto-updates itself. It's far more robust, easier to control and has two web pages (one of which is a live GPIO view) when the old one had only one very basic config page. Using Esparto, its only about 300 lines long and most of those 300 lines are a lot simpler and a lot more obvious to read.

But the biggest "gift" it brings is this: it runs all your code on the main loop thread, in a non-overlapping "job queue". All asynchronous events are "serialised" into the queue so no more problems of resource clashes, hangups, WDT resets and a hundred other headaches. It also provides tools for you to do the same from your own code. Each task runs separately in turn and can't interfere with/break/stop any other task, unless you deliberately make it do so. Again, if you don't know what all that means, read the "Essential Information" but what it translates to is: It prevents you from accidentally falling into about 90% of the common traps that newcomers fall into - and not all of them are obvious even to some experts. Some of them confuse experienced programmers for days and make grown men weep. Kiss 'em all goodbye.

It's fair to say that some of the complexity that Esparto hides (by deliberate design) would easily put off a lot of beginners, so having "MQTT in a box" is a huge help to getting started in the world of Home Automation and IOT. Now its absolutely true that if you can write a simple sketch to flash an LED you can also write one to produce your own Sonoff firmware. How's that sound for starters?

Of course Sonoff aren't the only player in town: that exact same sketch above will compile and run on Wemos D1, NodeMCU and (with a touch of "fettling" and shifting pin 12 to e.g. GPIO2) even an ESP-01 or ESP-01S.

Esparto is the result of 2years' worth of  thousands of mistakes, false starts, burned fingers, frustration and swearing - so that you don't have to go through it all again yourself.

The "truc" Home Automation "ecosystem": The NODE-RED controller

Did I already mention that I love NODE-RED ? I don't think I did, so: I love NODE-RED. If you haven't come across it before, google it now and then get it, install and run it and enjoy it.

I use it to centralise control of my many IOT devices and provide a "global" view of the system with a nice UI. It also provides my "update server" so that all devices automatically upgrade themselves when I change the firmware. In future it may tweet me when odd things happen and send facebook posts showing weekly summaries so I can see how my home is doing while on holiday...on second thoughts, I'm not sure I'm quite that sad yet!

One of its main benefits is that by putting all the "wiring" between the devices in one place on a raspberry Pi, the code in each device can be a lot smaller. It also allows me to "link" devices so that - for example - if one device in a zone goes on, they all do. I can also send MQTT control messages at the press of a button such as "tell all Sonoff s20's to upgrade now" or "tell all floor lamps to turn on".

I'm going to go into a lot more detail about how it does those things at a later date, once Esparto v2.0 has been released, but for now for the sake of completeness, here's a few snapshots:

1) The main "estate view"

2) The "Device" view

 3) The internal "routing" flow (one of five NODE-RED "flow"s)

Esparto v2.0 - sneak preview: Outside

"Esparto" is a type of coarse grass. That's neither here nor there really but it's a nice name for this Arduino ESP8266 library. Originally it stood for ESP All-purpose Run-Time Object, but it could just as well be "ESP All Ready To Operate" because that's what it is.

It is a C++ class object which does almost everything you need to do to turn your Wemos D1, NodeMCU, ESP-01S or Sonoff device from a pretty brick into an MQTT-controlled IOT device. Some folk would call it "firmware" and indeed you can upload a binary if you choose, but the real power comes when you extend it yourself with your own code.

As it stands, it will connect to your WiFi and MQTT controller and automatically reconnect after they fail - without rebooting. It will give you a visual (near*) real-time view of all the GPIO pins, with the correct labels on each depending on the device its deployed on. It will accept OTA updates and runs a web server that allows simple reconfiguration.

Using MQTT you can switch it on of off, reboot it (though you shouldn't need to) or factory-reset it. You can even dynamically change its deviceID and SSID+Password. You can read or write any IO pin and/or reconfigure them. What that last one means exactly will become clear later. Through all of this the basic on/off functionality will continue to work, even if your MQTT controller or complete local network are down.You don't need someone else's cloud and a flaky app on your iPhone. Anything that runs a web browser inside your network can control it. So even without your own additional code, it already does a lot.

Once you start adding your own code, the world of Home Automation and IOT is your oyster. In the example that follows, the device is running mu own Chez Toi ioT "truc" firmware, which provides a common interface for a number of sensors (PIR, light, sound, temperature) adds RF433MHz "dumb" switch control and stores the location and function of the device so that the NODE-RED controller can present a smooth UI for the overall management of the whole system.

Basic Esparto:

From the top: Device name, SSID+password change. No reboot, it just reconnects to the new settings. You will need to browse to another "devicename.local" page of course if you change the name or another IP page if the new SSID has a different range.

We'll come back to the "settings" cog icon in a moment, but the yellow circular arrow reboots the device and the factory icon resets it to as close to its original state as possible.

The next panel shows the current state of all the GPIOs, each labelled appropriately for the type of hardware its running on. The top row is the "raw" state of the pin, i.e. its instantaneous value, irrespective of what is connected to it. If it is configured as an output (e.g. the LED or RELAY in a Sonoff device) clicking the top row pin will toggle the state and switch the GPIO on or off. The bottom row will make more sense in the next article when we look "inside" at the code, but for now, just know that these are what Esparto calls the "cooked" state.

Esparto is based on the SmartPins library, which is itself based on the H4 timer / scheduler library. (See the side panel links to these for full documentation - all the functionality will be available in Esparto V2.0, plus a lot more of course. Or see http://github.com/philbowles/smartpins and http://github.com/philbowles/h4) . I'll also cover in the next post why those are both "good things" and will save you months of pain. SmartPins defines number of useful input types such as "debounced", "latching", "encoder" and several more. It does all the work to manage the pin, keep those lights updated and call your code back when the pin changes. So a "raw" button which is bouncy will flicker on and off as it bounces, but once SmartPins has done all the hard work and debounced the pin it puts the corresponding bottom line "cooked" pin on and calls your code. 

The next panel shows some vaguely-useful static information and the final panel shows dynamic information - including MQTT connection status - which is useful when you start adding your own code.

Back to the "setting" gear wheel. Out-of-the-box it will give you a "404" error. That is because it links to a "config" page that you add for the extra functionality that your code provides. Esparto has been specifically designed to make this easy for you, and we'll see just how much so in the next article. For the moment let's look at what Chez Toi ioT "truc" firmware adds:

Chez Toi needs to know where each device lives, to "wire together" important sequences, e.g. "Going to bed" turns on the bedroom light and the landing light and maybe later, the heater if the temperature is below 14degrees. Much of that added "intelligence" is in the raspberry Pi NODE-RED controller in order to keep the code "footprint" in each device as small as possible.

By knowing the location and function, I can add MQTT commands to turn on all devices in a zone, or all floor lamps in the whole system. "Restart" determines if the device is on , off or whatever it was before in the (rare) case of reboot or power outage. A safety nightlight for example would be (as before) but generally "off" is safest.

The numbers (all in milliseconds) are the configuration values for the sensors. PIR(t) is the PIR timeout and PIR(h) is its "hysteresis", meaning the "dead time" between two movement events to prevent it flicking on and off too rapidly. Sound is the timeout value for the re-triggering sound sensor, and light and temp are the frequency to poll those two sensors, since they change slowly.

This particular "truc" is also an RF master controller. When it switches on, those three 433MHz switches will also go on. Or off, of course. Clicking on any will toggle its current state to allow  degree of "manual" control. No more searching for the handset - any browser-capable device will do, tablet, phone, PC etc.