Beware of the (watch)dog!

Your house is full of smoke. The smoke alarm is beeping. Do you a) Turn of the smoke alarm and burn alive or b) find the source of the smoke and put out the fire?

I'm sure most folk would opt for b) but the most common problem I see when helping newbies in various esp8266 forums is the programmming equivalent of a). For reasons that are explained elsewhere on this blog LINK programming the ESP8266 isn't the same as programming  a "simple" AVR / Arduino etc and part of that difference frequently causes a "watchdog" timer reset - essentially a "crash" followed by a reboot.

These things generally only happen when the programmer doesn't fully grasp all the issues mentioned in the above LINK, but their first attempts to "fix the problem" usually involves "shooting the meesenger" and turning off the smoke alarm...

If you already know what a WDT is, how it works and why, then you will probably disagree with some aspects of my next statement...in which case, pop off somewhere else and let those who don't yet know those answers to allow this to sink in:

DO NOT TOUCH THE WATCHDOG TIMER. YOU DON'T NEED IT. FORGET IT EVEN EXSISTS! WHATEVER YOU THINK THE PROBLEM IS, IT IS ABSOLUTELY NOT THE WATCHDOG TIMER! DON'T FEED IT. DON'T DISABLE IT. 

D O N ' T   T O U C H  I T!!!

The “watchdog” timer (WDT) is the ESP8266’s smoke alarm. It goes off when there is a fundamental problem with your code. You need to find and fix that problem, not mess around with the WDT.

Embedded systems often don’t have the luxury of a screen and/or keyboard and are frequently fitted in difficult-to-access places where they are never seen by the human eye such as behind your living room wall or under the hood of your car - or in my case - 25feet up on a barn roof... When something fatal occurs, they have little option but to automatically reset themselves, thus many such devices have a WDT built into the hardware. This monitors the state of the system and if it freezes, locks / up or loops indefinitely for more than an “acceptable” amount of time, the WDT will reboot the device. After all, an occasionally faulty device is better than no device at all - especially if it controls your brakes.

I see many forum posts where the programmer says one of:

•        “I need to understand how the WDT works”
•        “There is something wrong with the WDT”
•        “My code runs fine on xxxx , but when I run it on the ESP8266, I get a WDT reset”
•      “Every time I run my code, I see: WDT reset, please help”.

My answers usually are:

• Oh no you don't (see above)
• Oh no there isn't
• So what?
• Read this blog

It really helps if you have already read the article on "Asynchronous programming". If you haven't, then you need to, because WDT problems are the tip of an iceberg and you need to understand the whole iceberg to get the best out of your ESP8266.

The usual cause of a WDT reset is that your code is “blocking” which means its stopping other processes or "threads" from running. This is often caused by taking too long to do what you think it needs to do. The most common causes I see are indiscriminate use of  delay() calls and/or waiting in a loop for an external resource e.g. a remote website. 

So how long is “too long” and what is an “acceptable” period of time, when your code already runs fine on an Arduino / stm32 / cray 1 / HP pocket calculator? Perhaps more importantly - why? 

ESP8266 is a WiFi capable device – that’s why you bought it, right? Connecting to, disconnecting from,  and – more importantly - maintaining a WiFi link os not magic - it takes processing time. There is only one CPU. The most important thing to grasp is that the code you write is not the only code running in the chip. About 200k+ of ESP code is loaded in before you even get to think about blinking an LED. And when does that code run? All the time. It runs “in the background” and you cannot easily see it or find out exactly what it’s doing and when. It just does its thing. Untill you interfere with it and stop it doing its thing. Then the WDT kicks in. It's really quite simple.

If your code stops the WiFi code from running for more than a very short period of time, the WDT says “oops! System has locked up, reboot!”. There is a reason why I have left you thinking "what does 'very short' mean? How long exactly is it?" and the reason is because if you write your programs correctly, you don't need to know. If you really want to, google it.

Yes, you can try to turn off  the WDT to “fix” the problem, but like the smoke alarm, it doesn’t remove the source of the fire, it just delays the inevitable. You can turn off the smoke alarm too, but if that is your preferred solution, I won’t be staying at your house, thank you. Even if you turn it off but still don't fix your code, the hardware WDT will probably kick in after a few seconds -and you can't turn that one off, so you are still going to crash - just several seconds later than if you hadn't turned off the software WDT.

Yes, there are ways you can "cheat" and "feed" the watchdog, but all you are doing is putting a blanket over the beeping smoke alarm to obscure the problem and hiding your bad code. Bad code generally finds a way to bite you in the ass no matter what you do, so it's best to find it and get rid of it, don't you think?

The only solution is to find the part of your code which blocks the background processing and then change it so that it doesn't. How to change it is a whole other (complex) story and for that, you definitely need to understand the link you haven't read yet...How do I know you haven't read it? Easy - because if you had, you wouldn't need to be reading this. Now go and read it.

The only way to absolutely guarantee no WDT resets is to write your code so that it can run asynchronously, co-operate fully with other processes and obey all the rules that multitasking requires. Unfortunately, that is a) a whole new way of thinking b) can be quite complex. With some basic rules, you can avoid most of the problems, but don't forget: we are talking about the tip of an iceberg here.

Until you get more experienced and fully understand the above paragraph, try to stick to the following:

1.       Never forget that yours is not the only code running.

2.       The problem is in your code. Messing with the WDT won’t fix that.

3.       Try to avoid delay() if at all possible. Only ever include delay() if it is absolutely needed and you truly understand why it is needed. If both of those aren't true, take it out.

4.       Never sit in a loop waiting for an external event to happen. Instead, set a volatile global, test and reset the global in the main loop. The same goes for callbacks and timer events. Or, write your code properly (see above link)

5.       Yield() in your main loop.

6.       If a library has a “run” or “handle” or “loop” method, always call it, it’s there for a reason!  This is usually the way library code does what your code also needs to do: co-operate with all other code running in the CPU. The best place is in your main loop.

7.       Never disable the WDT, it’s there for a reason!

Event-driven programming with callbacks

The previous article (which you should read now, before you continue) how "callbacks" made programming the ESP8266 easier and less error-prone - but what do they look like and how do they work?

Let's take the case where you want to read a sensor every minute. I've seen a lot of code like this around (or variations of it)

#define SENSOR 5

void setup(){
  Serial.begin(74880);
  pinMode(SENSOR,INPUT);

}

void loop(){
  if (millis()%60000){
    Serial.println("do something");
  }
}

Looks OK? Often the if(millis()... will be taking the current time, subtracting the previous value and checking if it == 1000 which of course requires a global variable for the previous value and some extra code, but the principle is the same - and it doesn't work!

loop() gets called about 40,000 times a second. So you are likely to "do something" up to 40 times because the value of millis() will be the same until another millisecond elapses! So, depending on how long "do something" takes, depends on how often it will be called, which is nothing like what you think you were doing, and if "do something" relies on accurate timing, your program will not work.

"Easy!" you think, "I'll set another global variable while doing something, then check it in loop and make sure I only do something once per loop".

"or, I can put delay(60000) inside the loop and then my timing will be accurate!"

The first option adds more code, more complexity (none of which is necessary and usually frowned upon - for plenty of good reasons - by experienced programmers) and the second just won't work.

No, the solution is to use the Ticker library which runs a highly accurate timer and calls back your code when the timer expires:

#include<Ticker.h>
#define SENSOR 5

Ticker  everyMinute; 

void doSomething(){                        // this is your callback function
  Serial.println("do something");
}

void setup(){
  Serial.begin(74880);
  pinMode(SENSOR,INPUT);
  everyMinute.attach_ms(60000,doSomething); // "register" your callback
}

void loop(){
}

The most important thing to realise here is that doSomething does not get called by everyMinute.attach_ms(60000,doSomething) in setup...all you are doing here is telling the Ticker library the name of your function - "registering" it - which will then be called every minute.

In a nutshell, that's how callbacks work. They are lot simpler, a lot cleaner and prevent you from re-inventing the wheel every time you write a sketch. But the most important  thing, is that they "just work" and they avoid numerous common problems.

Imagine if you had three or four sensors which need reading at different times...the loop code would very soon start to get complicated...using Ticker, you just have three or four tickers going off at different times, each with its own separate (obvious) callback which does just what that sensor needs. It's a lot more obvious and easier to read as well as being a lot less error-prone.

If you use "lambda" functions (and if you don't, you should - google them now) it's  even easier:

#include<Ticker.h>
#define SENSOR 5

Ticker  everyMinute;  

void setup(){
  Serial.begin(74880);
  pinMode(SENSOR,INPUT);
  everyMinute.attach_ms(60000,[](){ Serial.println("do something"); });
}

void loop(){
}

The "callback" is defined "inline" with the thing that will call it and saves having a separately defined function.

The Ticker library also allows you to pass a single (32-bit) parameter to your callback function, which is extremely useful and solves a lot of additional issues in the majority of cases. If however you want to pass two parameters, or call a class method when the timer "fires" - you are in for a lot of "fun" - unless you look at the author's "H4" library github.com/philbowles/h4 which is specifically designed to do just those things. It also adds more creative timer functions, such as calling back at random times or calling back a fixed number of times. Finally, it allows you to "chain" functions, i.e. call one after another has just finished. This allows some quite complex sequences to be built in to your code very simply indeed.

If the H4 library is used correctly, you will never need to call delay()...nor will ever need to know (far less need to muck about with) the "watchdog timer" and if you don't yet know why those are good things, read the next two articles!

It also does something much more important to prevent common errors, but I'll explain that later, once you are more familiar with this new "event-driven" style.

Even-driven programming - why you need it on ESP8266

Remember when you were a kid and your dad sat you down for that difficult "birds and bees" talk? Well, there's something we need to get out of the way now:

Programming the ESP8266 properly is not easy. In fact it's pretty tricky.

Sure, if all you want to do is flash an LED a la "blinky" sketch - that's ridiculously easy: pre-teenage kids can do it. But building an IOT infrastructure, writing firmware that will run on ESP-01, Sonoff, Wemos and NodeMCU that will never crash or reboot and will seamlessly reconnect after any network problem and never stop the attached hardware working...i.e. something actually useful, well that's harder. And it's virtually impossible unless you adopt the "event-driven programming" style.

Sure that are a lot of simple examples out there that don't do this - they look just like the code you are already used to writing - but the key is in the word "simple". They are examples to introduce you to a new concept: they are deliberately stripped back to the bare bones help you learn. They cannot hope to also teach you in a few lines of demo code the best way to use the new idea in the real world.

Rocket Science 101 is probably taught using a lot of fireworks, but no-one goes to the moon on one...

So why do you need to start adapting to the event-driven style?

Whatever device you are reading this on will be doing at least a dozen other things too: receiving an incoming SMS, updating your GPS location and playing your favourite tune. We live in an age where we are so used to such things, that we don't even notice it any more, we expect it.

The reason that your device is able to do all of those things apparently at the same time, is that for the last 40 or 50 years, hordes of programmers have learned the techniques of "multitasking" and built systems such as Windows, Linux, IOS, Android which are operating systems (OSs) that allow many things to happen as if at once, so that you don't have to understand muiltitasking when you write programs on those devices. But the ESP8266 doesn't have an operating system* so if you want to do more than one thing at a time, you do need to understand multitasking.

"But" - I hear you say - "I only want to do one thing at a time with my sketch! That's what I've always done with Arduino / AVR / STM32 etc and I didn't have to learn this "multitasking" thing!"

And I reply "Maybe so, but they didn't have built-in WiFi, did they? The ESP gives you no choice: it comes with WiFi built-in, and that's why you bought it. Having that WiFi changes everything - it's not "free" and it has important consequences. The first is that it doesn't operate by magic, it needs CPU time just like your sketch does.

The WiFi code in the ESP8266 needs to run all the time to keep the connection alive, as well as being ready to send and recieve data when your code needs it. So the ESP is partitioned into two sets of code: the WiFi code and your sketch. It is designed to run both at the same time. The main point here is that - unlike many other systems you may be used to - yours is not the only code running. Failing to adapt your coding style to these consequences - in anything but those simple examples - often leads to crashes, exceptions, "random" failures, "watchdog timer" resets and many other forms or programming pain. Event-driven programming is the easiest way to avoid that pain.

Instead of your code saying "do this, do that, then do the other" as you are used to doing, it now needs to say "tell me when X happens, tell me when Y happens and I'll sit here quietly - doing nothing - till you actually do". Buzzword time: The "old" way is called "synchronous" the "new" way is called "asynchronous" as well as "event-driven". Synchronous = you control when things happen, Asynchronous=you don't  - someone else does. In our case, "someone else" is the 200k+ of ESP firmware that manages the WiFi (amongst other things)

Your code has to co-operate and be ready to do what you need when the firmware says its OK. If your code runs in a tight loop or waits for a long time for an external resource (hardware, remote web page etc) and "blocks" the other code from running, bad things happen. This is why sometimes you will also hear about "blocking" (synchronous) and "non-blocking" (asynchronous) code.

To get the best out the ESP8266, you need to write "non-blocking" code.

The most common way of doing that is with "callbacks". A callback is a function that you write that gets called by some other piece of code, when that other code knows it is a good time to do so. You tell the "other" code what you are interested in, and it "calls you back" when the interesting thing happens. Till then you just twiddle your thumbs in the main loop. Telling the other code the name of your function is known as "registering a callback". Luckily for us, many of the libraries that come with the Arduino ESP8266 add-ons are designed this way.

Before we get into more details, let's get some other buzzwords out of the way. Those two "partitions" (your sketch and the WiFi firmware) are sometimes called "threads". Often when a system has only two threads - as in the ESP - they are known as the "background thread" (WiFi) and the "foreground thread" (Your sketch). After "setup" completes, most of your code starts from the "loop" function. So now its easy to understand why your code is sometimes said to "run on the main loop thread". Once the two threads start trying to "talk to each other" is when the fun begins...

The new way of thinking that comes along with the event-driven style is that events can happen at any time and often in a different order from what you might expect. Your code has to be ready. It needs to cope with suddenly being called "out of the blue" and it has to what it needs to do quickly, so that it doesn't block other (more important) code from running.

So now you know why, the next article will start looking into how, but before it does you need to realise that this isn't the end of the story...There are numerous (often complex) other consequences of two pieces of code both wanting to access a single resource (e.g. the CPU, a shared "flag") at the same time and there will be a lot more new buzzwords flying around when we meet them and learn how to deal with them.