Adding/changing rules doesn't always work immediately. You have to reset the firewall state (see Diagnostics | Reset state) or reboot.
I've never had to do this before? When I put in a block rule, it works as soon as I hit apply. If I turn it off, things are allowed back through as soon as I hit apply. I just did a test to verify this. I was only able to test a certain IP, do you mean this would apply to a range of IP instead?