Are you using the 'test' button? I believe there is a known issue which causes this kind of behavior when using that button. If you are using the button, try triggering the rule by having it monitor a certain output, then write a rule which will turn on the output (so you can use the test button).