Either the WAN connection doesn't support the normal IP MTU of 1500 bytes for Ethernet networks, or something (firewall, router, ISP, ...) along the way between the two m0n0walls blocks fragmented ESP packets.
In the latter case, you should find out what drops ESP fragments; in the former case, you could try adjusting the MTU of m0n0wall's WAN interface through /exec.php:
/sbin/ifconfig xxx0 mtu 1400
(replacing xxx0 with the BSD name of the WAN interface). If that helps, you can make it stick by adding it in a <shellcmd> tag inside the <system> section of config.xml. Note that it could also make things worse for normal (i.e. non-IPsec) traffic if WAN MTU size is not really the reason.
The WAN MTU setting was removed because the way it worked in earlier versions (applying MSS clamping even for non-PPP connections) was based on an ugly kernel hack and can no longer be supported in FreeBSD 6.
|