I am running 1.3b2 on a PIII 450 with 2x128 MB SD-RAM and a Kingston 1 GB CF-card on a ADSL2+ 25000/1500 connection. NIC's are two 3Com 3C905B. Performance and stability has been great. I did however experience quite high memory usage (90-95%) with only 1x64 MB RAM, and it's at 24% with 256 MB RAM. So it seems 64 MB is minimum, 128 MB is recommendable and anything above that is plenty.
This is a home network with 4-5 computers.
90% is high, did you tweak your m0n0wall installation to support more than 1000 concurrent connections? Cause that will definately increase memory usage from what I know. I'm currently at 40% memory usage as of now with my 64MB setup. Ram usage tends to increase all the way to 50%. I have not modified my m0n0wall install to use more ram. (enabling ram hogging features within the kernel or whatnot)