sande.ca

You've probably read the story about the remote rebooting device I made out of an old computer, a set of speakers, and a “Clapper”. This allowed me to monitor the webcams I have installed about 20km from my home and reboot them if any of them locked up, requiring a reset.

This worked well enough last fall while the temperatures were above freezing – as the cameras are fairly reliable in that state. However, in December, the temperatures here dropped to around -10 or -15C, which caused the cameras to lock up fairly regularly – and it seemed that every time I checked, I needed to take over the remote PC and manually run the “clapper”. I thought there had to be a better way, so I put on my thinking cap and tried to figure out a way that I could have the computer do all of the work with a minimum of manual intervention from me.

20 years ago or so, I used to run a BBS that functioned fairly reliably mainly because of fairly complex DOS batch files that I wrote (with many concepts borrowed from other Sysops who had solved similar problems before me). For example, one time while I was away on a trip, my system quit operating because it had run out of disk space while importing Usenet Newgroups – so I wrote a batch file to check for free disk space, and less than a certain amount of space was available, it would branch off to run a “message base pack utility” that would delete older messages and make room for new ones. The whole system worked off of “events” that the PCBoard BBS software used to run batch files containing various commands at pre-determined times.

A comment my brother-in-law made in early December when we were talking about something computer-related provided me with a clue that really opened the door to doing something similar with a Windows-based PC (thanks, Bruce!). For some reason, I had overlooked the fact that Windows has a built-in scheduler that could run commands (including batch files) at certain times. A little experimentation with the scheduler on one of my Windows-based home computers demonstrated that batch files could be run when scheduled and could be set to run repeatedly at pre-determined intervals. So, eventually I ended up with a bunch of batch files, all run through the scheduler to automate the monitoring process.

So, I had a plan to implement. But, of course, as with any technical projects, there were a few hurdles to overcome. One of the first was to try to figure out how to see if the webcams were working. On the surface, this seemed easy enough – I could “ping” them from a command line and wait for a response. A little “googling” told me that ping could set an errorlevel based on its success in reaching a host – at least on Linux, which is my primary operating environment at home. The importance of returning an errorlevel is this: a batch file can branch to run certain commands based on the value of an errorlevel – so if the errorlevel corresponding to a “locked up” webcam (no reply) was returned, I could have the batch file run my “clapper” command.

But of course, it's never that easy – as I soon discovered that the Windows implementation of ping doesn't return errorlevels – so I had to figure out a different way to do it. Again, a little searching quickly turned up the beginnings of a solution – ping wouldn't return an errorlevel, but it does return a unique text string depending on whether or not the host being pinged responds or not.

So, I came up with this tidy little solution to put into my batch file. One of my cameras has an internal IP address of 192.168.0.100 – so the following example will use that address.

Ping 192.168.0.100 -n 3 | find “Reply from” > NUL

if not errorlevel 1 got CAM14W

goto RESET

So, although ping doesn't return an errorlevel, but the DOS “find” command does. The logic used is this: If errorlevel 0 is returned, it means that the text has been found. If the errorlevel 1 is returned, the text has not been found. The “Reply from” text is only returned when the device responds – so when that's seen in the response, the batch file just goes to the next camera and tests it. If “Reply from” is not seen, the assumption is that the camera isn't working, and the Reset (Clap) command is run.

Here's the batch file as it exists today.

There are a couple of other things in here that were implemented as the result of trial and error testing – for example, in the :RESET section, you can see that it uses the “mplayerc.exe” command (Media Player Classic) to play the “clapper.mp3” file. The audio file clapper.mp3 plays three hand claps to turn the power off, waits about 10 seconds, and then plays another three claps to turn the power back on again. The batch file also uses command line switches /play and /close to run mplayerc and play the audio file through the speakers, exiting when complete. The play/exit sequence is important since this is all done from the command line. Once a batch file passes control to an external program, especially one that is Windows and not command-line based, it waits and gives up control until the program exits. If the external process wasn't able to be stopped, the batch file would pause indefinitely waiting for the external program to return control to it. Another simple solution found with a little research...

I also wanted to know what had happened during the day (or night, as the case may be), so the batch file also updates a log file (camreset log) when it does certain things. The first line after “@echo off” updates the log file whenever the batch file runs – and if a camera is checked without responding, another line is written to camreset.log indicating that a particular camera was responsible for the reset – and finally, the time of the reset is recorded. The result is a log file that looks something like this:

So, by analyzing the log file, you can tell that camera 17 hung a couple of times on this day – at 7:00 am and again at 2:00 pm (14:00 with the 24 hour clock). Look closely at the line right after the 14:00 reset – and you'll see that there's a different entry in the log file: “Changing clapper state...” Over the past 6 weeks or so, I've found that occasionally the clapper leaves the system with the power off when it should be on, and vice versa. This is a function of the fact that this is a “sound-based” system – if there happens to be a bit of ambient noise in “mid-clap”, the device ignores the command – so it's possible to get things reversed. After a couple of minor system malfunctions because of this, I added a check to the most reliable camera (Camera 15 almost never hangs!) - and if it's off, then I make the assumption that the power state has been switched – and send a single clap sequence to change from the power-off state to the power-on state. Here's that batch file:

So, everything was working well enough – but of course, I couldn't stop fiddling with things – so I found one more (minor) problem to solve. I use VNC and the Hamachi VPN software to remotely control this PC over my satellite Internet connection – which is fairly slow (512 kbps down and 128 kbps up, with about 150ms of latency). As the log file started to get a little larger, it became awkward to view – so I wrote another batch file for the scheduler to run just before midnight to rename that day's log file to the current date to shorten things up a bit and make it easier to view on the slow connection.

The result is a nice list of log files by date, which I manually compress into a ZIP archive once a month. It's very easy to now quickly remotely control this PC and check any day's log file to see how reliable the webcams have been at any given time.

So, that's my solution to keep my remotely-located webcams operating normally and (hopefully) reliably.

What's on the future horizon for this? If I stick with this solution, I have been thinking about adding a command-line Twitter update to the batch file which would let me check the status from any web browser without having to “remote deskop” into the on-site PC. However, a more likely solution would be to switch over to a remotely accessible power bar system that auto-pings the cameras itself, but is also controllable with a web interface. I think I've found one that's available at a reasonable price – but more on that later as I check out some of the details...