Speed of home control protocols . . .

[ChrisWalker]

We decided to do some speed tests with Z-Wave modules today, to demonstrate response times in a typical home (or your not-so-typical light show ).

For Z-Wave, we plugged in 4 Intermatic indoor/outdoor appliance modules. We plugged them into separate outlets around a small (visible/audible) area.

Then, we spent about five minutes writing a short program to turn them on and off rapidly without mercy.

Here are our results:
Test procedure: Turn each device on individually, and then each one off.
Repetition count: 50
Software: Z-Wave PC SDK
Hardware: Intermatic HA02/HA04 plug-in appliance modules

Speed (with full 2-way ACK): 62ms [16 commands per second]
Speed (without 2-way ACK): 32ms [31.5 commands per second]

Please note that these tests do not include any extra "performance tuning" beyond that built into the stock PC SDK. If we took some time to do tuning, I bet we could ratchet those numbers up even further

Even with hops in a network (which average one to two, maybe three in a home), the speed of any Z-Wave command is still less than 150-200ms in typical cases, making it truly "instant."

Chris

Briefly, here's what the test code looks like:

        Dim controller As New ZWave.Controller()
        controller.Connect("COM25")

        Dim i As Integer
        For i = 1 To 50
            Dim device As ZWave.Devices.ZWaveDevice
            For Each device In controller.Devices
                If device.NodeID <> controller.NodeID Then
                        device.PowerOn()
                End If
            Next
            For Each device In controller.Devices
                If device.NodeID <> controller.NodeID Then
                        device.PowerOff()
                End If
            Next
        Next
        controller.Dispose()

 

[Dean Roddey]

That's not really a very useful test. The question is how snappy can you make outgoing commands while maintaining low latency on two way feedback with a representative number of modules in a representative distribution and various types of random interference that happens in a real system. It's easy to just make outgoing commands quick if you aren't using up any network or controller time getting feedback from the modules.

That's not to say that Z-Wave isn't faster or slower or whatever than any other protocol. I think it's pretty quick considering the low data rate of the network. But the real measure of performance is the big picture of low latency feedback plus quick response to outgoing commands.

 

[ChrisWalker]

Dean:

Thanks for the feedback. Regarding "representative" device count or distribution, we try to go way beyond that--we do real-world tests in small homes with a few scattered devices and scale to large 10,000+ sq. ft. homes with hundreds of devices.

The test I just posted _was_ getting network feedback and taking up processing time on the controller. So we do know that with Z-Wave we have low latency feedback plus quick response to outgoing commands.

Additionally, we have a test (which I just ran on the same 4 modules) which requests the current dim level (or on/off state) from each device. Those test results (technically with at least double-acknowledgement and lots of traffic) still show <= 85ms response times.

<= 85ms is the controller talking to the device, receiving a full acknowledgement (or two) of its request, and then receiving a response (and acknowledging the response). That's fast.

And we had a 0% failure rate for 4,000 sequential stressful requests, resulting in thousands of messages, and a response time of ~83ms per full request/response (with multiple acknowledgements). We don't always get 0% failure rate in our real-world tests, but we typically get sub-one-percent failure rates for sure.

Chris

 

[elcano]

But Chris, you mention that you made one test with ACK and other without it. Then, the source code does not seem to be receiving any acknowledgement. It just seem a plain one-way turn-on/turn-off algorithm. For the benefit of the group, can you post the source were you get the ACK? How do this work, the device.PowerOn() returns a value indicating the result of the command? How do the program gets and take action on the ACK or NoACK?

Thanks,
Eric

 

[Dean Roddey]

The test I just posted _was_ getting network feedback and taking up processing time on the controller. So we do know that with Z-Wave we have low latency feedback plus quick response to outgoing commands.

There is no querying of state in that code you posted above. Getting acks of the writes isn't what I meant. That's just part of sending outgoing messages. I meant how responsive are the outgoing messages while keeping all the current states of the modules withing, say, a 5 second maximum latency. So you have to have one thread costantly polling such that you get the status of each module within 5 seconds, an then measure average response for outgoing commands to complete and be acknowledged.

 

[ChrisWalker]

But Chris, you mention that you made one test with ACK and other without it. Then, the source code does not seem to be receiving any acknowledgement. It just seem a plain one-way turn-on/turn-off algorithm. For the benefit of the group, can you post the source were you get the ACK? How do this work, the device.PowerOn() returns a value indicating the result of the command? How do the program gets and take action on the ACK or NoACK?

Eric,

We tried to make things very straightforward in the PC SDK. Maybe we made them a little bit too simple

The sample code I posted does get a full acknowledgement. I believe that full 2-way acknowledgement is the only way to have a reliable system--so we built that into everything in the SDK.

With the PC SDK, if you turn a device on or off, and it does not respond, you will get a DeviceNotResponding exception. If you get an acknowledgement, things will just go along smoothly. We take care of all of that automatically.

If you request the dim level of a device, and it acknowledges the request, but does not reply with the actual level, you will get a CommandTimeoutException instead.

Does that clear things up? I guess part of the confusion here is that the general public doesn't really know how Z-Wave works "internally." I try to share as much as I can, but of course the really deep stuff is all under non-disclosure. Suffice it to say that there are levels of redundancy and fail-safe in the protocol to ensure that messages arrive at their destination--even when doing a simple "power on."

Chris

 

[ChrisWalker]

There is no querying of state in that code you posted above. Getting acks of the writes isn't what I meant. That's just part of sending outgoing messages. I meant how responsive are the outgoing messages while keeping all the current states of the modules withing, say, a 5 second maximum latency. So you have to have one thread costantly polling such that you get the status of each module within 5 seconds, an then measure average response for outgoing commands to complete and be acknowledged.

That's one of the nice things about the SDK--all that happens automatically in the background. We do indeed keep track of the current state of all the modules.

If you run that code with an ACT lamp module instead, you will actually get a LevelChanged event fired almost immediately (within 100ms, probably) with the new level of the device. That level should be the level it was set to before it was turned off, since the ACT lamp modules resume to the last dim level when turned on.

If you wanted to query the level manually, you could simply write a line of code like "level = device.Level;" and you'd have that in milliseconds as well.

We have run tests (which I just mentioned--with ~80ms response times) which query the state of each device in the network. Was there something else you were looking for? What have you found in your own real-world tests?

I can't give away tens of thousands of dollars of data gathered from testing and the such, but I can say that even the optional polling mechanism in our Z-Wave PC SDK has been painstakingly developed to deliver good results while ensuring that devices within the network (including future products) still have room to breathe, and that neighboring Z-Wave systems don't have to deal with tons of network collissions (like in apartment buildings).

Chris

 

[Dean Roddey]

We've been through this discussion before. Z-Wave modules do not seem to respond that fast. They take about a second at best from what I've seen to respond to a query for state. Therefore, if you are going to get the status of say 10 units, you will have to have at least two overlapped queries going at a time, more than that really because they can take longer to respond sometimes. You can only have so many overlapping queries going on without just clogging the whole system up. For the say 25 that you might have in a realistic home scenario, a 5 second maximum latency would require 5 outstanding queries at once, probably more like 8 in order to handle some lagards, and that doesn't deal with those modules that are struggling for whatever reason and no responding at all or sporadically.

Anyway, I'll just wait for someone else to use your SDK to create a system that third parties use and prove emperically that it works as automagically and at as high two way speed as you claim, rather than getting into another drawn out technical argument.

 

[ChrisWalker]

We've been through this discussion before. Z-Wave modules do not seem to respond that fast. They take about a second at best from what I've seen to respond to a query for state. Therefore, if you are going to get the status of say 10 units, you will have to have at least two overlapped queries going at a time, more than that really because they can take longer to respond sometimes. You can only have so many overlapping queries going on without just clogging the whole system up. For the say 25 that you might have in a realistic home scenario, a 5 second maximum latency would require 5 outstanding queries at once, probably more like 8 in order to handle some lagards, and that doesn't deal with those modules that are struggling for whatever reason and no responding at all or sporadically.

Anyway, I'll just wait for someone else to use your SDK to create a system that third parties use and prove emperically that it works as automagically and at as high two way speed as you claim, rather than getting into another drawn out technical argument.

I'm not looking to start any long technical arguments, for sure.

And I'm really surprised that you're seeing such long latency. You may have some bad routes in your routing tables, or missing nodes, or something to that effect. We fix all those issues (with the SDK) so we typically don't deal with those.

What Z-Wave modules are you using? I know that the modules can vary quite a bit, but none of the ones we recommend have ~1000ms response times at all. Have you tested those response times with Z-Wave Certified software?

I'll have to send a response-time testing tool out to a few people, and see what kind of real world results people are getting with various products and network scenarios.

Chris

 

[Tombo]

Sorry Chris,

I just ran a test I call rapid fire on several devices. I tried to turn on and off the device 60 times. Each device responded intermittently. I got about 10 on /offs and it took 10 seconds . There is no way to do a christmas light demo like shown in the video. Now granted I am using the ACT USB controller.

Tom

 

[Tombo]

Of course it would be nice to add the async on /off commands I asked for so I get to queue up the calls and not wait on them. Just add TurnOn_NoWait and TurnOff_NoWait.

Tom

 

[ChrisWalker]

Sorry Chris,

I just ran a test I call rapid fire on several devices. I tried to turn on and off the device 60 times. Each device responded intermittently. I got about 10 on /offs and it took 10 seconds . There is no way to do a christmas light demo like shown in the video. Now granted I am using the ACT USB controller.

Tom

Tom,

What kind of devices are you turning on/off (brand and model)?

Also, did you do a Controller.RediscoverDevice() on each of them first (or use ThinkEssentials to heal your network)?

I think the problem here is due to products from different manufacturers working differently. I also see now why you've wanted asynchronous methods for "instant" commands

Chris

 

[elcano]

I guess part of the confusion here is that the general public doesn't really know how Z-Wave works "internally." I try to share as much as I can, but of course the really deep stuff is all under non-disclosure. Suffice it to say that there are levels of redundancy and fail-safe in the protocol to ensure that messages arrive at their destination--even when doing a simple "power on."

Chris

Chris,

I think that you hitted the nail. The public doesn't know and Zensys has not made an effort to help the public know. Just the opposite, they keep secrecy on how the technology works - I'm sure your non-disclosure says it clear.

I have faith that your product is going to be a technical success, but this is not good enough for the public to trust about the redundancy levels and fail-safes. The public want to see, evaluate, discuss the complete technology/protocol and the theory behind it (not just part of the technology, like speed). If it is really good, it is going to win. Look at Insteon. They released a paper about the details, and while people are already discussing the weaknesess in security (etc.), people trust because they have the opportunity to know, and discuss the technology (message format, timing, responses; the paper is very complete). This transparency wins the favor of the people who influence other people, and the chain goes on. This paper might be a marketing effort, but it is a very good one.

I know you cannot do much..its Zensys fault. And even if the new Z-Wave Stick and the SDK become a technical success they could also become a market faliure. Just like the Betamax, OS/2, BeOS, etc.

Zensys need to leave it everything in this race if they want to win. Letting the public know (specially this public) is part of it.

 

[Dean Roddey]

And I'm really surprised that you're seeing such long latency. You may have some bad routes in your routing tables, or missing nodes, or something to that effect. We fix all those issues (with the SDK) so we typically don't deal with those.

I'm about to go through another round of Z-Wave driver improvements. I'll dig into this more at that time and post what I find. I'll probaby be getting into that by the weekend.

 

[Tombo]

Ok I just did some more tests. I cut back my rapid fire to 10 on/offs. It takes about two seconds on average to complete. Here is the interesting part. The multilevel dimmer switches are fairly fast and the light just starts to light and instantly dims. The relay switch devices only do about 3 on/offs even though I sent 10. It does have a led though that blinks on and off 10 times so I am confident that the protocol did send the 10 commands and the device received them. The hardware just cannot get the relay to respond . So in a nutshell the protocol and SDK is good and is faster than the devices I currently have.

Tom