Esparto v3.3 released at last

AT last! A "quick fix " to patch up a problem in someone else's library has just ended up being a 2-month ground-up rewrite. I am so fed up with the whole thing, I hope you lot all like it enough to cheer me up: Watch the video then go and have a play with it. Esparto v3.3 released at last. Still some documentation being "backfilled" but the code should be ok and with 61 examples, you should be fine....

Esparto 3.3v feature preview

Esparto v3 finally released!

Blimey, it's finally done. Esparto v3.0.0 released! (subject to some documentation additions and tweaks) Some fun facts:
281 files in 72 folders totalling 11.8MB
Main code is 16 source files totalling 3742 lines of code
Includes 47 example sketches totalling 3704 lines of code..I'm happy to call it 7500 lines of code...

https://github.com/philbowles/esparto

Esparto V2 almost ready! The new web UI part 4

The lower panel(s)

Info

This is mostly self-explanatory but also raises a few points that are well worth knowing if you intend to become an "Esparto Expert".

These are all static values at the heart of the system, some of them permanently, some for example the IP address for this boot of the system only. They cannot be directly changed by the user. For some that can, see the next section.

Hardware Type:

This lets you know what's running "under the hood". It can be any one of these:

• ESP-01 (but why would you bother when there’s…)
• ESP-01S
• Wemos D1
• Wemos D1 mini
• Wemos D1 lite (and thus probably any other ESP8285 device)
• Wemos D1 pro
• NodeMCU 0.9
• SONOFF Basic
• SONOFF S20
• SONOFF SV

Esparto has been tested on all of the above. It will probably run on anything with an ESP-12 in it, but obviously I can't test every single device on the market/ If you want to send me one to try it out and modify if necessary...

I will be very interested to hear of anybody getting it running on any other platform e.g. NodeMCU 1.0 will probably work, as will (I expect) other SONOFFs

Unique Hardware ID:

This is the last 6 digits of the MAC address and is commonly used in new-out-of-the-box scenarios as a default name before choosing your own and setting your SSID / Password credentials the default name of the demo device would be ESPARTO-17D383. See part 2 earlier for more detail on this value and advice renaming your device to replace it.

IP Address:

Need I say more?

Flash Memory Size:

Same as above, except the answer is "Yes". 

Esparto weighs in at  over 410k - it has a lot of functionality and features. To support OTA updating (and who wouldn't want that?) a "sketch" (app) has to be able to fit into half the available flash size. On smaller devices e.g. the SONOFFs you get 1MB thus 512k is usable if you want OTA (and you do!).

As you can see things are already starting to squeak, so you need to keep your own additional code
small, efficient and light-weight. There is also a very limited amount of heap left. Esparto starts up with about 27k free, and that can go up and down rapidly - see the graphs in part 3 of this series for an example. Keep your heap use to a minimum and guard any heap-using routines with a check in what's free first if you want to avoid crashing (again, you do!).

The good news is that Esparto does so much for you that your own code will be small and consist mostly of short callback routines that Esparto will execute at the relevant time on your behalf. There is no loop() function and no setup() function. You will rapidly get used to doing things the "Esparto Way" once you see how easy it is.

H4 library version number:

If you want to get further than a simple "Blinky" it helps to understand the structure of Esparto. It is built from 3 main libraries, H4, SmartPins and Esparto itself.

H4 which handles all the timer functions, scheduling, task separation and "slip streaming" of asynchronous functions into the synchronous task queue which runs on the main loop. No more WDT resets, no more "volatile"s. When your task runs, it is (almost) the only player in town and the H4 library makes sure you have to try really really hard to break things or upset other tasks.

It comes with 7 of its own examples demonstrating how each and all of its functions work. Esparto "encapsulates" H4, so all of the H4 functions will appear to you as identical Esparto functions, so you do need to understand these first.

SmartPins library version number:

See above. Note the version shown is incorrect - by the time of  release it will also be 2.0.0 (actually it is, but I forgot to update the version number field before the demo - my bad!)

SmartPins as its name suggests manages all the input and output pins for you. It is what enables Esparto to give you the fancy real-time flashing LED display for all the pins. It also does everything you could ever want to do with a pin, including debouncing, interrupt handling (although there are good reasons why you would probably never need to use it), rotary decoding and much more.

The Encoder input type lets you manage a rotary with a single line of code - you tell it the name of a variable, and whenever you access the variable it will automagically have the current decoder value in it. One line of code! It's my favourite Esparto feature: most of my own mini-apps have some kind of "tweak" factor using a rotary, it's so easy. Some even have two...at the extra expense of one more line of code...I'll stop now, I think you have got the point.

SmartPins comes with nineteen sample program covering every in and out (literally!) of the many types of input modes it supports. It also has access to all H4's functions and relies on it 100% to function. Pretty much every example has at least one or two H4 features though of course they appear seamlessly as identical SmartPins features.

It is important then to work through the examples in order to fully understand the power and flexibility of Esparto, because in the same way, all Esparto functions are automatically the same as all SmartPins functions.

Even seasoned programmers will benefit, as Esparto works in a very different way from 99.235% of all the thousands of sample sketches you will find online. You need to learn the "Esparto Way", but for those with experience it won't take long at all.

Just as an example, here's the code (with comments removed for brevity) for the simple blinky. "Simple" includes having a fully debounced on/off switch unlike 99.476% of other blinkies.

NOTE: 

While H4 and SmartPins both have visible setup() and loop() functions, Esparto does not. There are two reasons for keeping them in:

1. To make the early examples more readily recognisable and ease you in to the "Esparto Way" and "chunk up" the amount of learning at each stage into bite-sized pieces.
2. To enable you to use them on their own without the full Esparto, although I can't think of any reason why you would want to unless you are the kind that likes to make things deliberately hard for themselves

#include <SmartPins.h>

SmartPins smartPins;

void buttonPress(bool hilo){

  if(!hilo) smartPins.flashLED(250);

  else smartPins.stopLED();

}

void setup(){

  Serial.begin(74880);

  Serial.println("LED will change state (flashing/off) on each separate button up/down press");

  smartPins.Output(BUILTIN_LED);

  smartPins.Latching(0,INPUT,15,buttonPress); // GPIO 0 + 15ms of debouncing

}

void loop(){

  smartPins.loop();

}

I hope you will agree both that it's pretty easy and also that you get "a lot for your money" for very little coding effort. That principle underlies the whole of the "Esparto Way": Esparto does 90% of the "heavy lifting", you plug in the remaining 10% which is specific to your IOT / home automation app. Esparto allows you to concentrate on just the code that's important to you - all the hard stuff "just works"

NBoot & Code:

These may be the first indication of a (hopefully very rare) problem. NBoot is the number of times this device has been rebooted and "Code" is the reason why. If it has just been freshly programmed then (as has the demo device) then it will read ESPARTO_BOOT_UNCONTROLLED.

What this means is that it was not shut down by user action, but forcibly rebooted, as the IDE does. You will also see this code if the device crashes for any reason.

If you click the Reboot button,. the code will become  ESPARTO_BOOT_UI. If you send an MQTT command e.g. testbed/cmd/reboot the code becomes ESPARTO_BOOT_MQTT and son on, although obviously you will replace "testbed" with your own device name first.

If you see an increased boot count and a reason you don't expect - something has gone wrong!

The "tXXX" values:

These measure the amount of milliseconds since boot up when:

tHW: 

The time after which your sensors, buttons, relays, remote controlled Gatling guns etc become ready to run. One of the fundamental design goals of Esparto is that your hardware should operate a) as early as possible b) whether you have a WiFi connection or not c) all the time, always.

Even if - as happens in the real world - bugs occur and the occasional crash occurs, your hardware will be back up ready to go in about 125 milliseconds. Impressive, non? It's one of the reason behind why Esparto won't let you play with setup() and loop(): it has quite a bit of complex setup of its own to do, and it has to happen fast, and in a very specific order.

tWiFi:

The time after which you can load up the web UI because your device now has a valid IP address.

tMQTT: 

Similarly, the time after which Esparto is actively listening for MQTT commands, both its own any any that you choose also to listen for. All Esparto command start with "cmd", so you must not use this in any of your own topics, or who knows when that Gatling gun may go off in error?

High values of either tWiFi or tMQTT may be early indications of problems with your router, network or MQTT broker. Or they may just be a sign of a slow network - only you will know. Personally, I'd worry about anything much more than the demo values. Again, I think 3.2 secs from power on to receiving MQTT commands is "in the zone".

Automatic unattended update of ESP8266 firmware using http server

One of the design goals for my home automation system was to be able to performs OTA (over-the-air) updates. The Arduino IDE makes this very easy, but when you have many devices deployed, it becomes a trifle tedious.

An easier way is to run and "update server" on your network and have each device check at boot time (or via an MQTT message) whether a newer version exists and if so, update itself automatically.

The central issue then is building the update server. The server software, language etc are unimportant:  it's the logic of how it responds to the request that matters. The excellent ESP8266httpUpdate class makes it very simple.

My own server is a "flow" in my NODE-RED controller on a raspberry Pi and - as you will see shortly - the logic is actually very simple.

What makes this all possible is the extra "headers" that the ESP8266HTTPUpdate class adds to the outgoing http request. They allow a great deal of control (should you need it) over how each device is updated.

x-ESP8266-STA-MAC
x-ESP8266-AP-MAC
x-ESP8266-free-space
x-ESP8266-sketch-size
x-ESP8266-sketch-md5
x-ESP8266-chip-size
x-ESP8266-sdk-version
x-ESP8266-mode set to either "spiffs" or "sketch"
x-ESP8266-version

This information allows the server to check that the upload will fit into the requesting device's memory, or provide a specific binary for a specific chip using the unique MAC address. Note also that you can also update a SPIFFS image (e.g. you inbuilt web pages / images scripts etc). The most important item is the version. This can be anything you decide.

The simple protocol and logic is this:

Device: "Hi, I'm firmware version a.b.c, do you have a newer version?"
Server: "Let me see...ah yes I have a.b.d - here it comes..."
If a newer version does exists the server just sends it as a load of binary data in the http reply. The ESP8266httpUpdate class does the tricky part of copying the binary into memory, changing the firmware boot address to the new code than (if requested) rebooting the device to run the new coded

If on the other hand there is no higher version, it replies with a http 304 error which effectively says: "I have nothing for you" and your code continues to run as normal.

Here's my NODE-RED update flow which shows the simplicity of it:

The salmon pink "nodes" are functions written in javascript - before delving into the actual code (which is minimal) we need to talk a little more about my setup:

On my server, I have a folder called /home/pi/trucFirmware which contains:

-rw-r--r-- 1 pi pi   65536 Nov  7 10:22 spiffs_0_4_3_1M.bin
-rw-r--r-- 1 pi pi 1028096 Nov  7 10:22 spiffs_0_4_3_4M.bin
-rw-r--r-- 1 pi pi  352192 Oct 17 13:37 truc_0_4_0.ino.d1_mini.bin
-rw-r--r-- 1 pi pi  344336 Oct 17 13:34 truc_0_4_0.ino.esp01s.bin
-rw-r--r-- 1 pi pi  352208 Oct 17 13:39 truc_0_4_0.ino.nodemcu.bin
-rw-r--r-- 1 pi pi  344528 Oct 17 13:35 truc_0_4_0.ino.sonoff_basic.bin
-rw-r--r-- 1 pi pi  347552 Oct 17 13:36 truc_0_4_0.ino.sonoff_sv.bin
-rw-r--r-- 1 pi pi  348144 Nov  9 17:46 truc_0_4_3.ino.sonoff_basic.bin

I maintain a separate binary for each hardware type (from a single source file with a few #defines) and when a new release is ready I use the Arduino IDE "sketch/Export compiled Binary" menu command for each target device.

Note that even though there are 5 different hardware types, there are only two SPIFFS binaries: a 1M and a 4M version - constructed with the mkspiffs tool - since all the devices have either 1M or 4M flash.

Once you have the mkspiffs tool, building the spiffs is simple. I have a one-line batch file for the 1M version which takes the version number as a parameter (%1)

mkspiffs -c data/ spiffs_%1_1M.bin

and another for the 4M version:

mkspiffs -p 256 -b 8192 -s 0x0FB000 -c data/ spiffs_%1_4M.bin

I then copy all the compile binaries and the SPIFFS .bin files over to /home/pi/trucFirmware

The whole process then, goes like this:

The sketch contains the following code:

#include<esp8266httpupdate.h>
...
#define TRUC_VERSION "0_4_99"
// THIS_DEVICE is set earlier depending on various compile-time defines 
// which eventually define the hw type, e.g. #define THIS_DEVICE "d1_mini"
const char * updateUrl="http://192.168.1.4:1880/update/"THIS_DEVICE;
// this is my raspberry Pi server, the 1880 is the default NODE-RED port
// /update is the url I chose for the server to "listen" for, followed by the device type
...
bool ICACHE_FLASH_ATTR  _actualUpdate(bool sketch=false){
    String msg;
    t_httpUpdate_return ret;
     
    ESPhttpUpdate.rebootOnUpdate(false);
    if(sketch){
      msg="sketch";
      ret=ESPhttpUpdate.update(updateUrl,TRUC_VERSION);     // **************** This is the line that "does the business"   
    }
    else {
      msg="SPIFFS";
      ret=ESPhttpUpdate.updateSpiffs(updateUrl,CSTR(E.getSPIFFSVersion()));
    }
    if(ret!=HTTP_UPDATE_NO_UPDATES){
      if(ret==HTTP_UPDATE_OK){
        msg.concat(" upgraded");
        Esparto.publish("update",CSTR(msg));
        return true;
        }
      else {
        if(ret==HTTP_UPDATE_FAILED){
          msg.concat(" upgrade FAILED code:");
          msg.concat(ESPhttpUpdate.getLastError());
          msg.concat(" reason:");
          msg.concat(CSTR(ESPhttpUpdate.getLastErrorString()));
          Esparto.publish("update",CSTR(msg));
          }
        }
      }
  return false;
}

When I'm ready, I call _actualUpdate(true); and the action then moves to the server:

The first node in the diagram above "listens" for an http request to url http://192.168.1.4:1880/update with the device type appended. It passes this to "Construct search path" function node which has the following javascript code:

msg.type=msg.req.params.type;
var h=msg.req.headers;
msg.version=h["x-esp8266-version"];

msg.mode=h["x-esp8266-mode"];
if(msg.mode=="sketch"){
    msg.payload="/home/pi/trucFirmware/*.ino."+msg.type+".bin";
}
else {
    var sz=h['x-esp8266-chip-size'];
    msg.payload="/home/pi/trucFirmware/spiffs_*_"+(sz/1048576)+"M.bin";
}
return msg;

This just sets up the appropriate path with wildcard for the sys function which follows, which simply runs ls - r <msg.payload>

The output is then fed to the "Compare versions" function node:

var f=msg.payload.split("\n")[0];
msg.filename=f;

if(msg.mode=="sketch"){
    f=f.replace("/home/pi/trucFirmware/truc_","");
    f=f.replace(".ino."+msg.type+".bin","");
}
else {
    f=f.replace("/home/pi/trucFirmware/spiffs_","");
    f=f.replace(/_\dM\.bin/,"");
}

if(msg.version<f){
    node.warn("upgrade required");
    node.warn("will return "+msg.filename);
    return msg;
}
node.warn("no upgrade");
msg.statusCode=304;
msg.payload=[];

return msg;

The switch node then ensures that either the 304 "no update needed" message is sent or the actual new binary is returned and sent back to the device.

There you have it: automatic unattended updates. All you need to do is copy the new binaries to your sever and - voila! - the rest "just happens".  It certainly makes my life easier with 8x ESP-01S, 6x Wemos D1, 4x Sonoff Basic 10x Sonoff S20, 2x Sonoff SV and a NodeMCU - 31 devices in all to update when I make a code change.

Doing is this way is almost effortless.

"A bit cheeky..." 2

The SMD soldering went..er...let's say it just went. I don't usually enjoy it, but the results didn't look too shabby. After a quick "smoke test" with the meter, I checked for continuity where expected, and not where it was not. All good.

Then - just to show how paranoid (read "safety conscious") I am with mains electricity - I built this little rig so I don't have to go anywhere near it while its under test and can be done fully hands-off. So much so, that the mains plug is into one of my WiFi truc devices and switch remotely from the bench! "The management eats here".

The PCB is a 1" square type that I came across on ebay some while ago - they are absolutely indispensable for knocking together little bits like this: buy them here. The came as 24pcs for eu6.62 or a measly 28c each - my kind of component. I always keep plenty in stock.

Anyway, once the power went on:

I would have hoped for a little closer to 3.3v but a) its all cheap parts b) its a cheap meter c) I'm sure it will be fine d) I'm over the moon it worked at all!

Find out if I'm right when I start the first "production" build of my ESP-01S powered wall switches which this PSU is for!

A bit cheeky...

I recently bought some of these AC-DC converters from ITEAD to run my truc devices from the mains. Mainly they are Wemos D1s but I also decided to build smart wall switches using ESP-01S as they are very small - but they run on 3.3v.

Click here to buy them yourself

"No problem...", I thought - "...looks like they do 3.3v too - bonus!". If you follow the link above to buy one, you'll see that ITEAD's latest photo has no 3.3v pin any more and on closer inspection of my own units, I can see why:

Seems that they are economising the few extra cents per unit of a 3.3v LDO...To be fair to them, the unit is clearly advertised as 5v and that's what I bought it for, but I couldn't help thinking "a ha'porth of tar..."

I also couldn't help thinking that if I could buy a few cheap LDOs in SOT-89 format with the right pinout, then I could right this "wrong". Don't misunderstand me, I'm very happy with ITEAD products and I have bought a lot of them - I trust them and I highly recommend this one, but...you know what I mean?

Anyways, it took a little research to find the right part, and as usual (I get the majority of my components from ebay.com) I found a seller doing five AP1117Y33L (here) They are incorrectly tagged as "Diodes" - but that's half the fun of ebay isn't it? Either way, they are what I needed.

There's a whacking great cap on the input side (470uF) but - as you can see - none on the output side. The datasheet for the AP1117Y33L states it needs a 22uF cap on the output for stability. So as long as I hook one up there on the final board, I should be "quids in".

Now all I have to do is a) bemoan my failing eyesight and b) get the fine tip on my soldering iron and "bite the bullet". If you aren't a big fan of soldering SMD parts ("fiddly" ain't the word), look away now...

I shall report back in due course.

Esparto v2.0 - sneak preview: Inside

"A picture is worth..." as they say:

The following 21 lines (one of which is a comment...) are all you need to turn a Sonoff Basic, S20 or SV into an MQTT device with a web interface...etc etc as described in the previous post "...outside". If you don't want diagnostics, you can lose the Serial,begin and cut another line.

The Sonoffs have a push button on GPIO0 and a mains relay on GPIO12. That's all they have, hardware-wise

#include <ESPArto.h>
// ToiioT-Etage is my SSID, pw="" (I live in the forest) my raspi mosquitto is on 192.168.1.4
ESPArto Esparto("ToiioT-Etage", "", "esparto666", "192.168.1.4", 1883); 
void buttonPressed(bool hilo){
  if(hilo) toggleRelay();
}
void mqttSwitch(String topic,String payload){
  toggleRelay();
  Esparto.publish("state",digitalRead(12) ? "ON":"OFF");
}

void setupHardware(){
   Serial.begin(74880);
   Esparto.Debounced(0,INPUT,15,buttonPressed); // 15 = ms debounce time
   pinMode(12,OUTPUT); // relay / switch
}
void onMqttConnect(){
  Esparto.subscribe("switch",mqttSwitch);
}
void toggleRelay(){
  digitalWrite(12,!digitalRead(12));
}

And the code above is all they need, and I ask you: "What could be simpler?"

True, you will have to physically FLASH upgrade it first time with a FTDI adapter, but after that, Esparto will update itself automatically as needed. It will appear on your WiFi network as esparto666.local and respond to an MQTT "switch" command, by toggling the power relay and will reply with an MQTT "state" message with a payload of "ON" or OFF". It will reconnect after any network failure and all the while, the manual button will still turn it on an off.

Plus it's inside your own network. No snazzy (but often rubbish) App to download. No security problems. No worrying if XYZ corp go out of business and close their cloud, that your lights will never work again...If you can use a web browser, you can control it. If you have an MQTT server, you can control it in much more detail. If you have a NODE-RED server, you can start to do really clever things with your whole house.

Let's look inti the code in more detail (shouldn't take long)

It doesn't look much like a typical Arduino sketch. There is no setup() function and no loop function. Esparto takes care of both, to make sure things are done in the "right" order and to prevent your code from accidentally breaking things or stopping it working.

Your code is all driven asynchronously by Esparto using callbacks. If you don't know what that means, you need to read the sidebar articles under "Essential Information". It starts with setupHardware. This is where you do what you'd normally do in setup. Having said that, much of what you'd "normally do" isn't needed any more.

Esparto.Debounced(0,INPUT,15,buttonPressed);

Tells Esparto that you want the button on GPIO0 debounced (for 15ms) and to call buttonPressed when someone pushes it or lets it go - i.e. when it changes. When it goes HIGH (the button on a Sonoff is "reversed" in sense: it goes LOW when you press it and HIGH when released) the relay is set to the opposite of what it is now. If its already on, it goes off etc - and that is the same as the standard firmware that it comes with when you buy it.

When Esparto has established a valid MQTT connection it calls onMqttConnect. Here, your code tells Esparto you want to receive "switch" topic message and when it gets one, it will call your code in mqttSwitch. As for a button press, you call toggleRelay which then publishes the current switch state to MQTT.

"And that's that"...as they also say.

Adding sensors to a "homebrew" board and adding lots of functionality on top of this is going to get more complex of course, but Esparto is designed to take a lot of the hard work out of that process too. It has a lot of "Hooks" where you can add callbacks in exactly the places you need to create a new "layer" of your own HA system on top of Esparto. That's how my own Chez Toi ioT system works: 90% of the code in each device is Esparto. Esparto has 9 different types of input pin it can manage for you, including rotary encoders and each of those only require one or two lines of code.

As an example my truc firmware when I fisrt wrote it was about 1200 lines of (pretty hairy) code and a lot of bugs. Now it handles GPIO on every pin of a WemosD1 and runs temperature PIR, sound, light, button and touch sensors. It also controls 433MHz RF switches, and it auto-updates itself. It's far more robust, easier to control and has two web pages (one of which is a live GPIO view) when the old one had only one very basic config page. Using Esparto, its only about 300 lines long and most of those 300 lines are a lot simpler and a lot more obvious to read.

But the biggest "gift" it brings is this: it runs all your code on the main loop thread, in a non-overlapping "job queue". All asynchronous events are "serialised" into the queue so no more problems of resource clashes, hangups, WDT resets and a hundred other headaches. It also provides tools for you to do the same from your own code. Each task runs separately in turn and can't interfere with/break/stop any other task, unless you deliberately make it do so. Again, if you don't know what all that means, read the "Essential Information" but what it translates to is: It prevents you from accidentally falling into about 90% of the common traps that newcomers fall into - and not all of them are obvious even to some experts. Some of them confuse experienced programmers for days and make grown men weep. Kiss 'em all goodbye.

It's fair to say that some of the complexity that Esparto hides (by deliberate design) would easily put off a lot of beginners, so having "MQTT in a box" is a huge help to getting started in the world of Home Automation and IOT. Now its absolutely true that if you can write a simple sketch to flash an LED you can also write one to produce your own Sonoff firmware. How's that sound for starters?

Of course Sonoff aren't the only player in town: that exact same sketch above will compile and run on Wemos D1, NodeMCU and (with a touch of "fettling" and shifting pin 12 to e.g. GPIO2) even an ESP-01 or ESP-01S.

Esparto is the result of 2years' worth of  thousands of mistakes, false starts, burned fingers, frustration and swearing - so that you don't have to go through it all again yourself.

Testbed part 2 - using ESP-01S

Here's the "mothership" with the optional 3.3v LDO plugged in to supply the ESP-01S with the 3.3v it needs:

Note, that's an "ESP-01S" and not the much more common "ESP-01". The S has 1M FLASH which makes OTA updates possible, and that's why there's no USB or FTDI on the board - thing of the past!

The touch sensor is a little gem - firstly, they are bounce-free. They are either on or off folks, which makes the very easy to program. I don't think I'm ever going to use a push button again! They are also sensitive enough to react from inside a plastic project case. I shall be using them inside all my IOT wall switches, hot-glued to the inside of the case front. Their best feature though is if you look around you can get 10 or 20 (including header pins) for a couple of bucks / euros. I got mine from ebay: 10 of them for eu1.39 - mad!

For the more adventurous, they can be "reprogrammed" to act in the reverse sense: as they come they go HIGH when touched, thus sit at LOW until then. That's fine unless you want to wire it to GPIO2 of an ESP8266 which needs to be HIGH at power-on...You just have to "short" a solder bridge or solder in an SMD zero-ohm resistor.

The 3.3v LDO (actually and AMS1117) breakout was also from ebay and only a few cents / pence etc each.

The best gadget though is the OpenSmart adapter. Again ebay, again very cheap. It allows the horribly unfriendly (who in god's name ever thought of it?) pinout of the -01/S modules to fit into a breadboard, with signal names conveniently silkscreened on it.

Take my word - if you are experimenting with or programming either variant, you need one of these!

Testbed part 1

Here is the testbed I built to cope with both ESP-01S and Wemos D1 Mini devices.

It has mains input which is fed to a relay board and then on to the actual appliance (usually a lamp) - this is used for final testing, which is all remote (via MQTT) as I don't like going near bare mains wires - and neither should you!

Most of the "bench" testing is done with 12v input from a PSU which is dropped to 5v by an L7805. That's fine for the Wemos D1, as it has an oboard 3.3v LDO, but when I'm testing ESP-01S, it doesn't, so there's a socket to take an external 3.3v LDO (an AMS1117) on a 3-pin breakout board.

The baseboard itself is from a company that doesn't seem to make them anymore, which is a shame as they are solid, rigid and very well-made.

Next comes the Wemos "tripler" base - a lovely piece of kit, but with a serious "gotcha" (see later post)

And then the separate sub-assemblies each of which I will detail in a separate post

Getting started

What you will need:

• ESP8266-based hardware
• Arduino IDE + Arduino ESP8266 core

Hardware:

The code examples in this blog are tested on:

• ESP-01S
• Sonoff Basic
• Sonoff S20
• Sonoff SV
• Wemos D1
• NodeMCU

They will almost certainly run on a number of other devices / dev boards, but the ones above are "guranteed"

ESP-01S

To be honest, I would avoid using any ESP-01 board until you are completely familiar with programming the ESP8266. They have extremely limited GPIO, are breadboard unfriendly and the non-S variants have insufficient RAM for OTA updates. They are also a bit of a nightmare to upload and you need an additional adapter. Having said all that, they are cheap cheap cheap and tiny so there are good reasons why I use the ESP-01S but until you've learned what they are and feel a bit more confident - avoid.

Where to buy: TBA

Sonoff

I love these things! They are "as cheap as chips" and wonderfully useful. A lot of people spend a lot of time and money trying hack together their own equivalents, but - trust me - save yourself a lot of time / effort / pain / money by going for these first if all you need is a simple WiFi-controlled switch.

Where to buy: TBA

Wemos D1

If you need more than a just a switch or plenty of GPIOs for sensors etc then the Wemos D1 is the dev board of choice. It's smaller and cheaper then the NodeMCU and does everything that 'MCU can do. I honestly don't know why people buy the 'MCU* when the Wemos D1 is so much better in many ways. (*I actually do - but for this blog and my own home automation system, the D1 wins hands-down)

Where to buy: TBA

NodeMCU (v0.9)

I keep one "in stock" to make sure all the code runs on it, but that's pretty much all I do with it - the Wemos D1 is so much better!

Where to buy: TBA

Chez Toi IOT design goals 2

Design Goals 2:

• OTA updates
• Compatibility with a variety of hardware
• Common fundamental interface for all devices
• Minimal functionality per device - "intelligence" added in central controller
• Control of "slave" RF 433MHz devices

OTA updates

A "no-brainer" - I can't climb ladders or rip open walls to press flash upload buttons. It's as simple as that. Some of devices will be on barn roofs and/or 100m from the house and it gets cold and wet in winter here...

There's a conflict here with the next goal, as the cheapest and smallest ESP8266 device (the ESP-01) doesn't come with enough flash RAM to support OTA updates. This is because to do so, a device has to have RAM at least 2x the size of the upload. ToiioT firmware weighs in at 300k+ and the '01 has 512k RAM - "do the maths" as they say. Happily the more recent ESP-01S variant has 1M - now we are cooking again, except...(It also has a couple of very subtle other differences that will bite you...)

Compatibility with a variety of hardware

Once I had discovered how easy and cheap the Sonoff range of devices are to re-flash with your own firmware, I thought "why use anything else?". Well, because they don't do much other than switch things on and off. That's great if that's all you want and in many cases, I do. But once you need to add PIR sensors etc etc etc. you need more pins.

Hence a heavy reliance on the Wemos D1, whose praises I cannot sing loudly enough. What a gem of a device and - again - an ever-present deign goal for me: cheap as chips.

Common fundamental interface for all devices

All "trucs" have a push button and an LED, whether they came built with it or not. In all of them, the action is the same:

Short press: device on/off

Medium press: LED flashes rapidly, device reboots on release

Long press: LED flashes high-speed, device "factory resets" on release

...all "just in case" and utterly, utterly essential during development.

Minimal functionality per device - "intelligence" added in central controller

The layout of my home is such that I want the landing light to come on when motion is detected. There are 5 choices: The 3 bedrooms, the bathroom or someone coming up the stairs.

I don't want "bedroom 1" controller to be "linked" by its inbuilt logic to the landing light, because tomorrow it may actually be "outside barn no 2" controller. So each sensor device is just a "dumb" box: "something moved". "I heard a noise". "It just went dark" etc.

The raspberryPi controller is what "wires" the devices together. It's got a lot more "grunt" and is a lot easier to program if you use the wonderful NODE-RED software. I have to have one anyway as my own "cloud" / MQTT controller. So NODE-RED receives the MQTT message "I'm truc27, something just moved" and says, "OK then, lets tell trucs11, 13 and 15 to switch on, and 7 to go off" - all by virtue of visually dragging and dropping a few virtual "wires".

When I move the same device somewhere else, I just "re-wire" NODE-RED in a matter of seconds.

Control of "slave" RF 433MHz devices

This one came very recently to the party. I have no overhead lights upstairs. When I moved in 3 years ago, I purchased some very cheap 433MHz RF sockets. I think I paid 14eu for three of them and a controller. I hate waste, so I thought it would be a wonderful and cheap way of adding to the ToiioT ecosystem to have the "big box of sensors" each be able to have a number of "slave" units.

So when truc9 says "someone touched me", NODE-RED tells truc14 to switch on. truc14 then autonomously happens to also send RF codes to its two floor lamps and bingo: Main light on, incidental lighting on - all for the price of one complex box and two cheapies I already owned.