Mac OS X only supports L2TP over IPsec out of the box, but m0n0wall does plain IPsec (ESP mode, no L2TP). You'll need an additional IPsec client; I recommend the free IPSecuritas (
http://www.lobotomo.com/products/IPSecuritas/), which already has a preset for m0n0wall in its setup wizard.