Trying to bring some Awesomation to Insteon

Insteon hub next to Netatmo
weather station

I recently got my hands on an Insteon Hub and plug in mains switch, so I set about integrating them with my Awesomation project.  The aim was to have Insteon devices (switch) controlled by other Awesomation devices (such as a Z Wave or Wemo motion sensors).

Insteon recently launched a cloud based API for their hub, which fits perfectly with Awesomation’s cloud based architecture.  On the face of it, their API is a pretty standard RESTful/JSON OAuth2 cloud API.  As Awesomation already supports both the Nest and Netatmo OAuth2 APIs, it should have been trivial to add another…

To begin with, you’ll need a Client ID and Secret - and no automated forms here like the Nest/Netatmo services.  It’s not so bad though, as once you fill out a form they send you the details a few days later.  But then things start to get a little weird.  Insteon’s OAuth implementation does not have anywhere to register the callback URLs, which as I understand leaves them wide open to covert redirect attacks.  What's more, when getting your token from the auth URL you must not provide a redirect URL; otherwise you get ‘401: Unauthorized’.  I wish they gave better error messages, or mentioned this in the documentation: that would have saved me a few hours of trial and error!

I would also be remiss if I didn’t mention their slightly strange scheme for authenticating each request - you are required to pass the following headers:

Authentication: APIKey example_api_key_135fhn80w35hynainrsg0q824hyn
Authorization: Bearer example_access_135fhn80w35hynainrsg0q824hyn

But at least this is documented (although not in the auth section of the docs!).

Once happily authenticated with the service, it was time to fetch a list of devices.  This was relatively straightforward, but nowhere in the API could I find a way to receive the state of the switch!  Hopefully this is an oversight on my part.

Finally, I wanted to wire up my abstract light switch data model such that my motion sensors could control the lights.  For this you need to use the commands API; you create a command object, and then poll the object until it completes (yuk!).  Initially all I got were stack traces, as I forgot to set my request content-type to ‘application/json’ (interestingly the stack traces indicate their stack is all Microsoft .Net and C#).  After fixing this I couldn’t get any further - every command I created was stuck in the pending state.  Debugging this led me further down the the rabbit hole...

Searching around the net I stumbled on this excellent blog post explaining how the outside world communicates with the Insteon hub - the hub relies on UPnP to open a hole in your firewall, and then tells a central service (running on AWS it seems) the IP address of your house.  Of course I have the one ‘obscure’ router which doesn’t support UPnP - an Apple AirPort Extreme!  I don’t understand why Insteon can’t use a websocket to push commands down to hubs, like any other sane home automation hub I’ve seen (Awesomation included!).  Not only does this punch through home routers more successfully, it requires zero configuration and can be much more secure than opening an unencrypted HTTP port to the outside world!  After mapping the port through my router (see this ‘guide’, not for the faint hearted) I confirmed the outside world could talk to my house by tethering my laptop to my phone and using http://connect.insteon.com, but alas, the commands API still did not work.  I’d note that picking apart Insteon’s basic site indicated they don’t use their own API, but rather connect directly to the router and manipulate it that way.

And this is where I ran out of weekend time, the whole Insteon API leaving a sour taste in my mouth.  The API still very much seems in an alpha state - a weak sign up flow, security problems, poor documentation, stack traces and non-functional commands.  I hope other people have had more success than I, and I welcome someone pointing out the mistakes in my code.  I’ve filed bugs with Insteon and will post a follow up when I get this working, but at this point, I can’t recommend anyone attempts to integrate with this API.

Motion Sensor Shootout

Good sensors are key to any automation system; motion sensors are possibly some of the most important.  I couldn’t find a good comparison of different motion sensors when I started building my system, so I’ve gone out and bought one (or two) of pretty much every sensor I could find, and this post shares my experiences with them.

I’m not confining this post to just Z Wave motion sensors - I’ve included the Wemo wifi motion sensor as well.  If there are any sensors I’ve not included, but which you think I should, drop me a comment and I’ll see what I can do.  All these sensors have been setup inside, generally in the top corner of a room.

Belkin Wemo Motion (top), Aeon Labs Multisensor (left), Fibaro Motion Sensor (middle),
Vision Motion Sensor (right).  Banana for scale.

Aeon Labs Multi Sensor (~£45)

Aeon Labs sensor showing the
mounting (next to it is an ADT
motion sensor not reviewed here)

Perhaps the most popular Z Wave motion sensor out there.  I originally purchased two of these, as the first motion sensors for my system - therefore they bore the brunt of the steep learning curve with Z Wave, and my review may be a little jaded by this experience.  As with most (all?) Z Wave motion sensors, the device is asleep and unreachable most of the time - to configure it it needs to be woken up.

It is quite big - easily the size of the base of a pint glass.  The supplied mounting bracket means it sits quite proud of the wall, and doesn’t offer as much flexibility as other sensors.  The sensor takes 6 AAA batteries, which it is supplied with.  When fully populated with batteries, the sensor is quite heavy - my initial attempts to attach it to the wall with double sided tape had me being woken up in the middle of the night by strange noises as it fell from the wall!

Alongside motion, the sensor also measures temperature, humidity and luminance.   There is a firmware bug where these values are wrong when read from the device.  You need to configure the sensor to push the values to your controller periodically - then these values are correct.  The motion sensor has low latency in reporting detected motion to my controller (a Raspberry Pi with the Aeon Labs Z Stick and my Awesomation software).  I have found the device is quite sensitive to positioning - I wrote a whole blog post about this!

Overall this is a capable sensor, which a good price for the feature set.  It is not my favourite though.

Another review worth mentioning is http://zwaveworld.com/2014/revs/923/

Fibaro Motion Sensor (~£50)

Fibaro Motion Sensor showing the
cool mounting system.

The second sensor I purchased, and what a beauty.  By far the smallest and least obtrusive of the sensors in this test, it also feels nicer in the hand - better quality plastics, better fit and finish etc.  The mounting system is also worth mentioning; a single screw, mounts virtually flush against the wall and has complete flexibility on angle.  By far the best here in that respect.

The sensor also measures temperature and luminance (no humidity like the Aeon Labs device).  By default it will send the controller reports when these values change by a configurable threshold, but it can be configured to report them regularly.

Z Wave range on this sensor is good - it is the furthest from the controller and still functions well.  The sensor also appears to have the best latency, although this is purely subjective as I haven’t measured it accurately.

Vision Z Wave Motion Sensor (~£34)

The cheapest sensor here.  No clever mounting system; it is not possible to adjust the position the sensor is pointing in.   The sensor is very light - in the box is a large piece of double sided tape to mount it to the wall, and hasn’t fallen off yet.  The device includes a temperature sensor like the other Z Wave sensors, but not humidity or luminance.   I haven’t yet got it to work consistently, but I suspect like most Z Wave devices it just needs to be ‘healed’ a few times.

Belkin Wemo Motion (£60, with a Wemo switch)
The only mains powered, WiFi based sensor here.  Much larger than the other sensors.  I was hoping this would mean it would be easy to integrate as the device is always powered on and listening, but I’ve found it inconsistently replied to UPnP scans.  When you do finally get registered for events (which are sent as a HTTP NOTIFY back to your application), I found they come thick and fast - constantly toggling between motion and no motion.  The failure detector work I previously blogged about responded quite nicely to this.  A bigger problem is the events stop coming before the subscription timeout - and I haven’t figured out why yet.

Having the motion sensor connected to the wall wart via a thin cable is better than having the sensor on the wall wart (and down near the floor), but it does feel a bit tacky and it is hard to hide the cable.  For a device like this, I would have thought Belkin could have made the sensor itself wirelessly connect to the wall wart.

Conclusion
My favourite of these motion sensors is the Fibaro Motion Sensor.  Looks good, works well and was easy to integrate with Awesomation.  I really wanted to like the Belkin Wemo sensor, but it has too many problems to recommend.

All these sensors are integrated with my Awesomation project, and are used to control devices ranging from Hue bulbs to Z Wave switches.  In the future, I’d like to have a play with the D-Link ‘mydlink Home’ Wi-Fi Motion Sensor but I couldn’t find any python libraries to talk to it.  I guess I’ll have to write my own.

Augmenting Nest's Auto Away

Nest Thermostat in the hallway

In hindsight, the placement of my Nest thermostat is less than ideal.  It's in the hallway on the ground floor, pretty much the middle of my house.  The Nest sees me walk past it when I wake up, when I get home from work, and when I have to go and wave at it to turn the heating on.  You see, we spend most of our time between the kitchen, the office and the TV room, all of which are below ground floor, and so can't been seen by the Nest.  The Nest will go into auto away mode while we are home, something we only notice when things get a little chilly.

I wanted to improve the situation with technology, so I needed a proxy for my presence in the house.  I always carry my iPhone with me when I leave the house, so I set out to detect the presence of the phone on my wifi network, and use this to set the away status of the thermostat.  Unfortunately the iPhone’s behaviour on the wifi network is unpredictable - the phone sleeps (does not respond to a ping) for long periods of time.  Other projects use knowledge about devices on the network gleaned from their particular brand of wifi router, but I wanted a router agnostic approach.

I also wanted the system to be easy to configure - I wanted to show a list of devices seen on the network to the user and have them pick their phone.  To this end, I began by not targeting a single MAC address, but all devices on the network.  I initially started monitoring the arp table on my Raspberry Pi for the presence of my phone’s MAC address using the ‘arp -a’ command.  Unfortunately, entries seem to hang around in the arp cache for a very long time - and trying to understand the linux arp cache was difficult.  

I switched to using the command from above stackoverflow answer (‘ip -s neighbor list’) to expose the freshness of entries in the cache, but even this again wasn’t a good enough signal, as the iPhone never really spoke to the Raspberry Pi.  I had to start pinging stale entries in the list - this worked well, but eventually the iPhone MAC address would drop out of the list.  To fix this I added a periodic broadcast ping to the entire subnet.  I combined this with some simple timeout logic produce a less noisy signal, but since then I’ve migrated to using a more sophisticated failure detector.

UI tiles show presence of phones

This system is reasonably effective - it can detect when I am home and hold the heating on.  A background cron job periodically polls the Nest API and sets the away status correctly, just in case it goes into away mode.

Since initially building this I’ve added motion sensors in the aforementioned rooms - used to control the lighting.  I plan to incorporate this signal into the logic for controlling the Nest, improving the responsiveness of the system.  What's more, I’ve begun experimenting with an iPhone app for geofencing and iBeacons.  Hopefully this will allow me to know who is in a room, not just that the room is occupied.  But that’s a topic for another post...

Inexpensive Automatic Home Lighting

If you're like me, you prefer home lighting that uses standing lamps and desk lamps (as opposed to overhead lights). And you find it bothersome to have to turn on 4 lamps every time you walk into a room. I therefore decided to automate the lighting in rooms I most commonly use, using a mix of motion sensors, Philips Hue and wireless plug in mains switches.

A Wemo switch, a TKB Z Wave switch
and a Brennenstuhl 433Mhz switch

There are a bunch of wireless mains switches you can use: the Belkin Wemo switch is a popular choice, as are the various Z Wave dimmers. But much simpler and cheaper alternatives are the numerous 433mhz mains switches, such as the Brennenstuhl remote control mains sockets I have. These can be bought for as low as £5 a piece (when bought in packs of 3), which makes automating a room with 6 lamps the same price as using a single Wemo or Z Wave switch.

A 433Mhz transmitter wired up
for the Pi's header

But these switches come with a single, proprietary remote, and don't integrate well with existing home automation systems (that I know of).  Luckily, inexpensive 433mhz transmitters can be found on eBay, and someone has already reverse engineered the protocol used to communicate with these devices (writing an excellent forum post about it).  All that was needed was a quick a bit of soldering to connect the transmitter to the GPIO pins on a Raspberry Pi, and some swigging of the libraries so I could talk to the it from Python. These were the first devices integrated with Awesomation, my Python home automation system.

The transmitter mounted on the Pi

The simplest solution was to repeat each command to each switch multiple times; however done naively I found this causes very slow response times when turning on rooms with many lights.  I settled on a weird little queue and background thread in my control script; when commands are sent they are stuck on the queue for the thread to process; the thread reinserts them into the back of the queue to have them repeated without starving the other commands; new commands are given increasing generation numbers to ensure old repeated commands don't compete with new state changes.

The end result has been a super reliable, low latency automated lighting system for many lights, built for less than the price of a single Wemo switch or Hue bulb, all controllable from my phone! I should also say I’m not the only person to think of this - there is an excellent article over on hackaday about a similar project.  I guess Awesomation has the added benefit of having all the devices integrated into a single system - I only need one app (or sensor) to turn all the lights off in a room - be they Wemo switches, Hue bulbs or these cheap 433Mhz ones.  And I can control everything from my mobile when I’m out and about, just incase I forget to turn the lights off!

Using Phi Accural Failure Detectors to Improve Occupancy Sensors

Aeon Labs MultiSensor (left) and
Fibaro Motion Sensor (right)

We’ve got a bunch of Z Wave occupancy sensors (Aeon Labs MultiSensors and Fibaro Motion Sensors - shootout coming soon) to automate the lighting in the office and tv room.  Unfortunately these are both rooms you tend to sit very still in, and having to periodically wave your hands to turn the lights on is seriously damaging the WAF of the system.  I was looking for an approach that wasn’t particularly sensitive to sensor positioning and that didn’t involve fine tuning timeouts.

The approach I’ve gone for is to use a python implementation of the Phi Accural Failure Detector to better interpret the signals from the motion sensors.  These detectors are more commonly used to decide the health of hosts in a distributed system, such as in Apache Cassandra.  These detectors record the intervals observed between heartbeats (pings in the distributed system case), and use this distribution to decide if the target is alive at a given time.

I’m using the detector to determine if a room is occupied (as opposed to a host being alive), but unfortunately the Z Wave motion sensors don’t send heartbeats periodically for me to feed to the detector.  Instead, the sensors send edge-triggered events when they believe the room’s occupancy state has changed, based on their own internal algorithm, which I’m reasonably sure is just a simple timeout.

The initial approach to this was to synthesize heartbeats to the detector periodically, when the sensor says that the room is occupied.   The detector is then periodically used to decide whether to turn the lights on.  The implementation of this can be seen in this changeset.  As the logic for the home automation system runs on App Engine, there is relatively little control over the exact timing of this periodic cron job.  It was very possible that the sensor would detect motion (and the system would heartbeat the detector), then a few seconds later the cron job would run, see the sensor says that the room is occupied, and heartbeat the detector again.  Similarly, the cron job could run, heartbeat the detector, then moments later the sensor could timeout (and the previously synthesized heartbeat could be considered wrong).

As the detector is based on recording heartbeat inter-arrival times, this approach doesn’t work particularly well - the recorded inter-arrival times would be too artificial (based on the cron job timing) and occasionally completely misleading; either too short (at the beginning of the period) or completely wrong (at the end of the period).

The improved approach is based on knowing the internal timeouts the sensors use, which luckily the Z Wave sensors expose as a configuration value.  Now, the system only synthesizes events and feed them to the detector when the sensors “stops” detecting motion - and can therefore avoid the synthesized events being outright wrong (by making the last event happen timeout seconds in the past) and avoid the event inter-arrival times being too artificial (by spacing them timeout seconds apart).   While this approach makes the detector useless while the sensor is “on”, they are not needed at this point - we know the room is occupied!

UI tile showing lighting controls
for a room

You can find the implementation of this approach on Github.  Since implementing this we’ve not had to wave our hands to turn the lights on once.  One downside of this approach is that the lights tend to stay on longer than we would like once we leave the room - about 30 minutes.   This is not a big problem - I’d rather the lights were on for too long rather than occasionally turning off when I’m in the room, and the lights are all low powered LED types (Philips Hue, for example).   I think lowering the timeouts on the sensors will also help here.