Augmenting Nest's Auto Away

Nest Thermostat in the hallway

In hindsight, the placement of my Nest thermostat is less than ideal.  It's in the hallway on the ground floor, pretty much the middle of my house.  The Nest sees me walk past it when I wake up, when I get home from work, and when I have to go and wave at it to turn the heating on.  You see, we spend most of our time between the kitchen, the office and the TV room, all of which are below ground floor, and so can't been seen by the Nest.  The Nest will go into auto away mode while we are home, something we only notice when things get a little chilly.

I wanted to improve the situation with technology, so I needed a proxy for my presence in the house.  I always carry my iPhone with me when I leave the house, so I set out to detect the presence of the phone on my wifi network, and use this to set the away status of the thermostat.  Unfortunately the iPhone’s behaviour on the wifi network is unpredictable - the phone sleeps (does not respond to a ping) for long periods of time.  Other projects use knowledge about devices on the network gleaned from their particular brand of wifi router, but I wanted a router agnostic approach.

I also wanted the system to be easy to configure - I wanted to show a list of devices seen on the network to the user and have them pick their phone.  To this end, I began by not targeting a single MAC address, but all devices on the network.  I initially started monitoring the arp table on my Raspberry Pi for the presence of my phone’s MAC address using the ‘arp -a’ command.  Unfortunately, entries seem to hang around in the arp cache for a very long time - and trying to understand the linux arp cache was difficult.  

I switched to using the command from above stackoverflow answer (‘ip -s neighbor list’) to expose the freshness of entries in the cache, but even this again wasn’t a good enough signal, as the iPhone never really spoke to the Raspberry Pi.  I had to start pinging stale entries in the list - this worked well, but eventually the iPhone MAC address would drop out of the list.  To fix this I added a periodic broadcast ping to the entire subnet.  I combined this with some simple timeout logic produce a less noisy signal, but since then I’ve migrated to using a more sophisticated failure detector.

UI tiles show presence of phones

This system is reasonably effective - it can detect when I am home and hold the heating on.  A background cron job periodically polls the Nest API and sets the away status correctly, just in case it goes into away mode.

Since initially building this I’ve added motion sensors in the aforementioned rooms - used to control the lighting.  I plan to incorporate this signal into the logic for controlling the Nest, improving the responsiveness of the system.  What's more, I’ve begun experimenting with an iPhone app for geofencing and iBeacons.  Hopefully this will allow me to know who is in a room, not just that the room is occupied.  But that’s a topic for another post...

Inexpensive Automatic Home Lighting

If you're like me, you prefer home lighting that uses standing lamps and desk lamps (as opposed to overhead lights). And you find it bothersome to have to turn on 4 lamps every time you walk into a room. I therefore decided to automate the lighting in rooms I most commonly use, using a mix of motion sensors, Philips Hue and wireless plug in mains switches.

A Wemo switch, a TKB Z Wave switch
and a Brennenstuhl 433Mhz switch

There are a bunch of wireless mains switches you can use: the Belkin Wemo switch is a popular choice, as are the various Z Wave dimmers. But much simpler and cheaper alternatives are the numerous 433mhz mains switches, such as the Brennenstuhl remote control mains sockets I have. These can be bought for as low as £5 a piece (when bought in packs of 3), which makes automating a room with 6 lamps the same price as using a single Wemo or Z Wave switch.

A 433Mhz transmitter wired up
for the Pi's header

But these switches come with a single, proprietary remote, and don't integrate well with existing home automation systems (that I know of).  Luckily, inexpensive 433mhz transmitters can be found on eBay, and someone has already reverse engineered the protocol used to communicate with these devices (writing an excellent forum post about it).  All that was needed was a quick a bit of soldering to connect the transmitter to the GPIO pins on a Raspberry Pi, and some swigging of the libraries so I could talk to the it from Python. These were the first devices integrated with Awesomation, my Python home automation system.

The transmitter mounted on the Pi

The simplest solution was to repeat each command to each switch multiple times; however done naively I found this causes very slow response times when turning on rooms with many lights.  I settled on a weird little queue and background thread in my control script; when commands are sent they are stuck on the queue for the thread to process; the thread reinserts them into the back of the queue to have them repeated without starving the other commands; new commands are given increasing generation numbers to ensure old repeated commands don't compete with new state changes.

The end result has been a super reliable, low latency automated lighting system for many lights, built for less than the price of a single Wemo switch or Hue bulb, all controllable from my phone! I should also say I’m not the only person to think of this - there is an excellent article over on hackaday about a similar project.  I guess Awesomation has the added benefit of having all the devices integrated into a single system - I only need one app (or sensor) to turn all the lights off in a room - be they Wemo switches, Hue bulbs or these cheap 433Mhz ones.  And I can control everything from my mobile when I’m out and about, just incase I forget to turn the lights off!

Using Phi Accural Failure Detectors to Improve Occupancy Sensors

Aeon Labs MultiSensor (left) and
Fibaro Motion Sensor (right)

We’ve got a bunch of Z Wave occupancy sensors (Aeon Labs MultiSensors and Fibaro Motion Sensors - shootout coming soon) to automate the lighting in the office and tv room.  Unfortunately these are both rooms you tend to sit very still in, and having to periodically wave your hands to turn the lights on is seriously damaging the WAF of the system.  I was looking for an approach that wasn’t particularly sensitive to sensor positioning and that didn’t involve fine tuning timeouts.

The approach I’ve gone for is to use a python implementation of the Phi Accural Failure Detector to better interpret the signals from the motion sensors.  These detectors are more commonly used to decide the health of hosts in a distributed system, such as in Apache Cassandra.  These detectors record the intervals observed between heartbeats (pings in the distributed system case), and use this distribution to decide if the target is alive at a given time.

I’m using the detector to determine if a room is occupied (as opposed to a host being alive), but unfortunately the Z Wave motion sensors don’t send heartbeats periodically for me to feed to the detector.  Instead, the sensors send edge-triggered events when they believe the room’s occupancy state has changed, based on their own internal algorithm, which I’m reasonably sure is just a simple timeout.

The initial approach to this was to synthesize heartbeats to the detector periodically, when the sensor says that the room is occupied.   The detector is then periodically used to decide whether to turn the lights on.  The implementation of this can be seen in this changeset.  As the logic for the home automation system runs on App Engine, there is relatively little control over the exact timing of this periodic cron job.  It was very possible that the sensor would detect motion (and the system would heartbeat the detector), then a few seconds later the cron job would run, see the sensor says that the room is occupied, and heartbeat the detector again.  Similarly, the cron job could run, heartbeat the detector, then moments later the sensor could timeout (and the previously synthesized heartbeat could be considered wrong).

As the detector is based on recording heartbeat inter-arrival times, this approach doesn’t work particularly well - the recorded inter-arrival times would be too artificial (based on the cron job timing) and occasionally completely misleading; either too short (at the beginning of the period) or completely wrong (at the end of the period).

The improved approach is based on knowing the internal timeouts the sensors use, which luckily the Z Wave sensors expose as a configuration value.  Now, the system only synthesizes events and feed them to the detector when the sensors “stops” detecting motion - and can therefore avoid the synthesized events being outright wrong (by making the last event happen timeout seconds in the past) and avoid the event inter-arrival times being too artificial (by spacing them timeout seconds apart).   While this approach makes the detector useless while the sensor is “on”, they are not needed at this point - we know the room is occupied!

UI tile showing lighting controls
for a room

You can find the implementation of this approach on Github.  Since implementing this we’ve not had to wave our hands to turn the lights on once.  One downside of this approach is that the lights tend to stay on longer than we would like once we leave the room - about 30 minutes.   This is not a big problem - I’d rather the lights were on for too long rather than occasionally turning off when I’m in the room, and the lights are all low powered LED types (Philips Hue, for example).   I think lowering the timeouts on the sensors will also help here.