Search This Blog

Showing posts with label RF433. Show all posts
Showing posts with label RF433. Show all posts

Wednesday, 1 November 2017

Esparto v2.0 - sneak preview: Inside

"A picture is worth..." as they say:

The following 21 lines (one of which is a comment...) are all you need to turn a Sonoff Basic, S20 or SV into an MQTT device with a web interface...etc etc as described in the previous post "...outside". If you don't want diagnostics, you can lose the Serial,begin and cut another line.

The Sonoffs have a push button on GPIO0 and a mains relay on GPIO12. That's all they have, hardware-wise

#include <ESPArto.h>
// ToiioT-Etage is my SSID, pw="" (I live in the forest) my raspi mosquitto is on 192.168.1.4
ESPArto Esparto("ToiioT-Etage", "", "esparto666", "192.168.1.4", 1883); 
void buttonPressed(bool hilo){
  if(hilo) toggleRelay();
}
void mqttSwitch(String topic,String payload){
  toggleRelay();
  Esparto.publish("state",digitalRead(12) ? "ON":"OFF");
}

void setupHardware(){
   Serial.begin(74880);
   Esparto.Debounced(0,INPUT,15,buttonPressed); // 15 = ms debounce time
   pinMode(12,OUTPUT); // relay / switch
}
void onMqttConnect(){
  Esparto.subscribe("switch",mqttSwitch);
}
void toggleRelay(){
  digitalWrite(12,!digitalRead(12));
}

And the code above is all they need, and I ask you: "What could be simpler?"

True, you will have to physically FLASH upgrade it first time with a FTDI adapter, but after that, Esparto will update itself automatically as needed. It will appear on your WiFi network as esparto666.local and respond to an MQTT "switch" command, by toggling the power relay and will reply with an MQTT "state" message with a payload of "ON" or OFF". It will reconnect after any network failure and all the while, the manual button will still turn it on an off.

Plus it's inside your own network. No snazzy (but often rubbish) App to download. No security problems. No worrying if XYZ corp go out of business and close their cloud, that your lights will never work again...If you can use a web browser, you can control it. If you have an MQTT server, you can control it in much more detail. If you have a NODE-RED server, you can start to do really clever things with your whole house.

Let's look inti the code in more detail (shouldn't take long)

It doesn't look much like a typical Arduino sketch. There is no setup() function and no loop function. Esparto takes care of both, to make sure things are done in the "right" order and to prevent your code from accidentally breaking things or stopping it working.

Your code is all driven asynchronously by Esparto using callbacks. If you don't know what that means, you need to read the sidebar articles under "Essential Information". It starts with setupHardware. This is where you do what you'd normally do in setup. Having said that, much of what you'd "normally do" isn't needed any more.

Esparto.Debounced(0,INPUT,15,buttonPressed);

Tells Esparto that you want the button on GPIO0 debounced (for 15ms) and to call buttonPressed when someone pushes it or lets it go - i.e. when it changes. When it goes HIGH (the button on a Sonoff is "reversed" in sense: it goes LOW when you press it and HIGH when released) the relay is set to the opposite of what it is now. If its already on, it goes off etc - and that is the same as the standard firmware that it comes with when you buy it.

When Esparto has established a valid MQTT connection it calls onMqttConnect. Here, your code tells Esparto you want to receive "switch" topic message and when it gets one, it will call your code in mqttSwitch. As for a button press, you call toggleRelay which then publishes the current switch state to MQTT.

"And that's that"...as they also say.

Adding sensors to a "homebrew" board and adding lots of functionality on top of this is going to get more complex of course, but Esparto is designed to take a lot of the hard work out of that process too. It has a lot of "Hooks" where you can add callbacks in exactly the places you need to create a new "layer" of your own HA system on top of Esparto. That's how my own Chez Toi ioT system works: 90% of the code in each device is Esparto. Esparto has 9 different types of input pin it can manage for you, including rotary encoders and each of those only require one or two lines of code.

As an example my truc firmware when I fisrt wrote it was about 1200 lines of (pretty hairy) code and a lot of bugs. Now it handles GPIO on every pin of a WemosD1 and runs temperature PIR, sound, light, button and touch sensors. It also controls 433MHz RF switches, and it auto-updates itself. It's far more robust, easier to control and has two web pages (one of which is a live GPIO view) when the old one had only one very basic config page. Using Esparto, its only about 300 lines long and most of those 300 lines are a lot simpler and a lot more obvious to read.

But the biggest "gift" it brings is this: it runs all your code on the main loop thread, in a non-overlapping "job queue". All asynchronous events are "serialised" into the queue so no more problems of resource clashes, hangups, WDT resets and a hundred other headaches. It also provides tools for you to do the same from your own code. Each task runs separately in turn and can't interfere with/break/stop any other task, unless you deliberately make it do so. Again, if you don't know what all that means, read the "Essential Information" but what it translates to is: It prevents you from accidentally falling into about 90% of the common traps that newcomers fall into - and not all of them are obvious even to some experts. Some of them confuse experienced programmers for days and make grown men weep. Kiss 'em all goodbye.

It's fair to say that some of the complexity that Esparto hides (by deliberate design) would easily put off a lot of beginners, so having "MQTT in a box" is a huge help to getting started in the world of Home Automation and IOT. Now its absolutely true that if you can write a simple sketch to flash an LED you can also write one to produce your own Sonoff firmware. How's that sound for starters?

Of course Sonoff aren't the only player in town: that exact same sketch above will compile and run on Wemos D1, NodeMCU and (with a touch of "fettling" and shifting pin 12 to e.g. GPIO2) even an ESP-01 or ESP-01S.

Esparto is the result of 2years' worth of  thousands of mistakes, false starts, burned fingers, frustration and swearing - so that you don't have to go through it all again yourself.


Esparto v2.0 - sneak preview: Outside

"Esparto" is a type of coarse grass. That's neither here nor there really but it's a nice name for this Arduino ESP8266 library. Originally it stood for ESP All-purpose Run-Time Object, but it could just as well be "ESP All Ready To Operate" because that's what it is.

It is a C++ class object which does almost everything you need to do to turn your Wemos D1, NodeMCU, ESP-01S or Sonoff device from a pretty brick into an MQTT-controlled IOT device. Some folk would call it "firmware" and indeed you can upload a binary if you choose, but the real power comes when you extend it yourself with your own code.

As it stands, it will connect to your WiFi and MQTT controller and automatically reconnect after they fail - without rebooting. It will give you a visual (near*) real-time view of all the GPIO pins, with the correct labels on each depending on the device its deployed on. It will accept OTA updates and runs a web server that allows simple reconfiguration.

Using MQTT you can switch it on of off, reboot it (though you shouldn't need to) or factory-reset it. You can even dynamically change its deviceID and SSID+Password. You can read or write any IO pin and/or reconfigure them. What that last one means exactly will become clear later. Through all of this the basic on/off functionality will continue to work, even if your MQTT controller or complete local network are down.You don't need someone else's cloud and a flaky app on your iPhone. Anything that runs a web browser inside your network can control it. So even without your own additional code, it already does a lot.

Once you start adding your own code, the world of Home Automation and IOT is your oyster. In the example that follows, the device is running mu own Chez Toi ioT "truc" firmware, which provides a common interface for a number of sensors (PIR, light, sound, temperature) adds RF433MHz "dumb" switch control and stores the location and function of the device so that the NODE-RED controller can present a smooth UI for the overall management of the whole system.

Basic Esparto:


From the top: Device name, SSID+password change. No reboot, it just reconnects to the new settings. You will need to browse to another "devicename.local" page of course if you change the name or another IP page if the new SSID has a different range.

We'll come back to the "settings" cog icon in a moment, but the yellow circular arrow reboots the device and the factory icon resets it to as close to its original state as possible.

The next panel shows the current state of all the GPIOs, each labelled appropriately for the type of hardware its running on. The top row is the "raw" state of the pin, i.e. its instantaneous value, irrespective of what is connected to it. If it is configured as an output (e.g. the LED or RELAY in a Sonoff device) clicking the top row pin will toggle the state and switch the GPIO on or off. The bottom row will make more sense in the next article when we look "inside" at the code, but for now, just know that these are what Esparto calls the "cooked" state.

Esparto is based on the SmartPins library, which is itself based on the H4 timer / scheduler library. (See the side panel links to these for full documentation - all the functionality will be available in Esparto V2.0, plus a lot more of course. Or see http://github.com/philbowles/smartpins and http://github.com/philbowles/h4) . I'll also cover in the next post why those are both "good things" and will save you months of pain. SmartPins defines number of useful input types such as "debounced", "latching", "encoder" and several more. It does all the work to manage the pin, keep those lights updated and call your code back when the pin changes. So a "raw" button which is bouncy will flicker on and off as it bounces, but once SmartPins has done all the hard work and debounced the pin it puts the corresponding bottom line "cooked" pin on and calls your code. 

The next panel shows some vaguely-useful static information and the final panel shows dynamic information - including MQTT connection status - which is useful when you start adding your own code.

Back to the "setting" gear wheel. Out-of-the-box it will give you a "404" error. That is because it links to a "config" page that you add for the extra functionality that your code provides. Esparto has been specifically designed to make this easy for you, and we'll see just how much so in the next article. For the moment let's look at what Chez Toi ioT "truc" firmware adds:



Chez Toi needs to know where each device lives, to "wire together" important sequences, e.g. "Going to bed" turns on the bedroom light and the landing light and maybe later, the heater if the temperature is below 14degrees. Much of that added "intelligence" is in the raspberry Pi NODE-RED controller in order to keep the code "footprint" in each device as small as possible.

By knowing the location and function, I can add MQTT commands to turn on all devices in a zone, or all floor lamps in the whole system. "Restart" determines if the device is on , off or whatever it was before in the (rare) case of reboot or power outage. A safety nightlight for example would be (as before) but generally "off" is safest.

The numbers (all in milliseconds) are the configuration values for the sensors. PIR(t) is the PIR timeout and PIR(h) is its "hysteresis", meaning the "dead time" between two movement events to prevent it flicking on and off too rapidly. Sound is the timeout value for the re-triggering sound sensor, and light and temp are the frequency to poll those two sensors, since they change slowly.

This particular "truc" is also an RF master controller. When it switches on, those three 433MHz switches will also go on. Or off, of course. Clicking on any will toggle its current state to allow  degree of "manual" control. No more searching for the handset - any browser-capable device will do, tablet, phone, PC etc.



Testbed 4 - more sub-assemblies: temp and rf433

This one has the (very "bouncy") tact button that all my IOT devices have on GPIO0
  • quick press: switch on / off
  • medium: reboot
  • long: factory reset


It also has the TMP36 temperature sensor and the output module to transmit RF codes to my "el cheapo" sockets: These are what I bought for the 300+ year old farmhouse before I had the (insane) idea to replace everything with a fully-fledged WiFi IOT Home automation system.


Being one who doesn't have a lot of cash to splash, I wondered if I could re-use them...Then I found the wonderful rcswitch library https://github.com/sui77/rc-switch and I found some ridiculously cheap RX/TX pairs on ebay (buy similar here) 5 sets for eu2.79 or 56c a pair. So the TX you see here set me back - what - 28c? I love cheap.

Setting it up was easy: The rcswitch lib has an example that uses the RX to read the codes of the handsets. I was short of Wemos D1s at the time, so I dragged a dusty old Arduino UNO out of a drawer and had the code running in about five minutes flat. I captured all the codes from my four handsets for on and off. I then knocked a little transmit library for the Wemos with a few "hard-coded" values and no-one was more surprised than me to hear the happy sound of clicking less than an hour or so after I started! Tweaking the code, removing the hard-coded values and setting up the automatic re-configuration then occupied me for the next few days, of course...

Now my system uses these as "slaves". Each "truc" or Wemos-controlled "big box of sensors" reads MQTT configuration data at startup to see which RF switches are linked to it. When it next switches on or off, it automatically sends the RF codes to make all its "slaves" match. In the real world, the main box would be wired in with the overhead light in each room, and the el-cheapos would be floor lamps, or corner lamps. When commanded either by touch or remote MQTT, all go on/off at once.





Monday, 30 October 2017

Chez Toi IOT design goals 2

Design Goals 2:
  • OTA updates
  • Compatibility with a variety of hardware
  • Common fundamental interface for all devices
  • Minimal functionality per device - "intelligence" added in central controller
  • Control of "slave" RF 433MHz devices
OTA updates

A "no-brainer" - I can't climb ladders or rip open walls to press flash upload buttons. It's as simple as that. Some of devices will be on barn roofs and/or 100m from the house and it gets cold and wet in winter here...

There's a conflict here with the next goal, as the cheapest and smallest ESP8266 device (the ESP-01) doesn't come with enough flash RAM to support OTA updates. This is because to do so, a device has to have RAM at least 2x the size of the upload. ToiioT firmware weighs in at 300k+ and the '01 has 512k RAM - "do the maths" as they say. Happily the more recent ESP-01S variant has 1M - now we are cooking again, except...(It also has a couple of very subtle other differences that will bite you...)

Compatibility with a variety of hardware

Once I had discovered how easy and cheap the Sonoff range of devices are to re-flash with your own firmware, I thought "why use anything else?". Well, because they don't do much other than switch things on and off. That's great if that's all you want and in many cases, I do. But once you need to add PIR sensors etc etc etc. you need more pins.

Hence a heavy reliance on the Wemos D1, whose praises I cannot sing loudly enough. What a gem of a device and - again - an ever-present deign goal for me: cheap as chips.

Common fundamental interface for all devices

All "trucs" have a push button and an LED, whether they came built with it or not. In all of them, the action is the same:

Short press: device on/off
Medium press: LED flashes rapidly, device reboots on release
Long press: LED flashes high-speed, device "factory resets" on release

...all "just in case" and utterly, utterly essential during development.

Minimal functionality per device - "intelligence" added in central controller

The layout of my home is such that I want the landing light to come on when motion is detected. There are 5 choices: The 3 bedrooms, the bathroom or someone coming up the stairs.

I don't want "bedroom 1" controller to be "linked" by its inbuilt logic to the landing light, because tomorrow it may actually be "outside barn no 2" controller. So each sensor device is just a "dumb" box: "something moved". "I heard a noise". "It just went dark" etc.

The raspberryPi controller is what "wires" the devices together. It's got a lot more "grunt" and is a lot easier to program if you use the wonderful NODE-RED software. I have to have one anyway as my own "cloud" / MQTT controller. So NODE-RED receives the MQTT message "I'm truc27, something just moved" and says, "OK then, lets tell trucs11, 13 and 15 to switch on, and 7 to go off" - all by virtue of visually dragging and dropping a few virtual "wires".

When I move the same device somewhere else, I just "re-wire" NODE-RED in a matter of seconds.

Control of "slave" RF 433MHz devices

This one came very recently to the party. I have no overhead lights upstairs. When I moved in 3 years ago, I purchased some very cheap 433MHz RF sockets. I think I paid 14eu for three of them and a controller. I hate waste, so I thought it would be a wonderful and cheap way of adding to the ToiioT ecosystem to have the "big box of sensors" each be able to have a number of "slave" units.

So when truc9 says "someone touched me", NODE-RED tells truc14 to switch on. truc14 then autonomously happens to also send RF codes to its two floor lamps and bingo: Main light on, incidental lighting on - all for the price of one complex box and two cheapies I already owned.  




"Chez Toi" IOT - the need and the naming

In France where I live, "toi" means "you" and "Chez toi" means "your home". Its handy that "toi" is IOT backwards and so I jumped for the obvious name of my still-under-construction home automation system to be "Chez Toi ioT" or sometimes when I'm lazy - which is quite often - "ToiioT" for short.

Also, a "thing" in French when used to mean doodad / widget / gadget / thingamajig is a "truc"hence all the "things" in my Internet of things are called truc1, truc42, truc666 etc. Just so's you know.

The plan was to use the smallest devices possible for the various components, hence I use ESP-01S (the extra "S" is important, you'll learn why later) for wall switches, ITEAD Sonoff devices for the sockets and a few hard-to-get-at places and "homebrew" Wemos D1s for my "big box of sensors" devices which monitor daylight, movement, sound and control the el-cheap RF433 sockets I owned before I started building the system.

The overall aim is to make life easier in a house + barns that was built when candles were the only light source and provide controllable and informative outdoor devices (webcam, PIR lights) for when visitors arrive or wild boar rampage through my garden.

The main design goal was robustness. The Internet here is flaky beyond belief. I can only get a half-decent speed via a satellite dish. I don't want the lights / power to become inoperative every time there is a storm and the Internet goes down (which it invariably does). Secondly I want practicability / ease of maintenance, so all devices have to be upgradeable remotely, via "OTA" (over-the-air): Once I have a solar-powered Sonoff SV-controlled PIR light device 35feet up on the roof of "the big barn", I don't want to have to climb a ladder to reboot it!

Putting all that together, I realised I might need a lot of different bits of code for all the different devices / functions, so I took a decision early on that all the code must run on all the devices, so I could mix-n-match during the early design and experimentation stages.

After a steep learning curve - and several changes of heart - the system fell slowly into place. It started with a monolithic piece of code called "truc". It developed into a series of external libraries which break the functionality down into manageable sizes chunks and allow others to benefit from the learning curve by encapsulating all the difficult bits. (See the right hand links to github where these are published for anybody to use)

"Truc" became "Esparto" which I will talk about in a later post but is essentially all the general-purpose bits of the system that are needed by anybody doing this. All my own specific bits are a "layer" on top of that, still called "truc", but much, much smaller and lighter.