f you are on 1.3b10 as your sig says, upgrade to 1.31 and in advanced tick the option for ' Disable Spoof Checking on bridge'
My .sig was out of date, corrected now. I had tried that option in m0n0wall 1.31 to no avail.
I've done further investigation. There are a lot of suggestions on the 'Net for solving this problem -- way too many, in fact, they are superstition, luck, what worked for one or two people.
I bought a cheap Netgear WAP and using this in place of m0n0wall as the access point for the iPhone to connect to machines on the wired LAN, the problem still occurred with my first test machine, which is a new minimal install of Win XP Pro (no AV, no firewall) on an Intel D510MO motherboard with its onboard Realtek 8111DL NIC. I purged and reinstalled iTunes, the NIC driver, etc. all to no avail.
I've tested five machines now on the same LAN/WLAN with the Netgear WAP: two Win 7 machines, three Win XP machines (all SP3, up to date as of today), and I cannot for the life of me connect the iPhone Remote application to two of the Win XP machines. The other three machines connect fine.
To prove that m0n0wall is not causing a problem, I should switch back to m0n0wall WAP and reproduce the same fail/success with each machine as with the Netgear WAP. I haven't done that; if I do I'll post again, but if m0n0wall is a problem, it's not the only one. It seems to be an Apple software issue of inconsistent installs/repairs of iTunes (and/or Bonjour, etc.). Here's what I've learned.
The five machines include one Win XP machine, so it's not an XP issue. That XP machine has a NIC that's not a Realtek 8111DL, which is also the NIC that's in an XP machine that fails, so it's not the XP NIC driver. On the original XP machine that is failing, I tried a USB-ethernet NIC to no avail. I put the XP machine that works at the end of the same ethernet cable as the XP machine that was failing and it worked, so it is not my switch or physical network.
One Win 7 laptop that succeeds can connect via WLAN or LAN, doesn't matter.
This problem affects only the ability of the Remote App to connect: iTunes can share libraries successfully between all computers. Once again, the "fail" behavior is for iTunes to prompt for the passcode to connect the remote device (iPhone 3GS in my case). It accepts the passcode, shows "verifying remote passcode" in the status line, hangs a few seconds, then reverts without any notice to the main iTunes screen (where instead on success one would get a confirmation screen that the remote is paired). Meanwhile the iPhone wipes the passcode off the screen and tries to connect to an unnamed library until it times out. Then you've got an unnamed (blank-named) library in your list, any attempt to connect to which times out.
I investigated network traffic on UDP port 5353 and TCP port 3689 using wireshark (capture filter "udp port 5353 or tcp port 3689") running simultaneously on both a machine that doesn't work and a machine that does, connecting to both machines. Both wiresharks show the same traffic, again confirming there is not a network connectivity issue. In a successful connect, the two devices do a kind of handshake on mDNS (UDP 5353) then switch to the TCP port and start jabbering. In a failed connect, the iTunes host never starts the TCP conversation and the iPhone is left hanging making a few more mDNS requests till it times out.
A library-sharing conversation is similar (switching UDP to TCP) and once again these work on all my machines.
I'm using iTunes 9.1 (latest as of today), except on one of the failing machines running XP and iTunes 9.0 (I will upgrade to 9.1 but think it won't help). I've fiddled with iTunes preferences settings a lot. I don't think they're relevant because I set two machines to the exact same preferences and one still failed while the other succeeded.
I'm at the end of my diagnostic abilities and time for this problem. I blame iTunes software: on some installations (of XP at least) it is failing to be a proper host to the Remote application. It doesn't depend on the NIC, and reinstallation/repair does not help. The definitive (?) proof of this theory will be to boot a failing machine from a hard drive taken from a working machine, and have it work. One failing machine is a pristine XP installation, the other failing machine and one of the working ones are old bloated hacked installations. So there's really nothing to hang on to here, and nothing to blame but Apple software. If only iTunes would log what seems like its silent failure after entering a correct passcode.