I solved a similar issue with building out a white box server with ESXi (
http://www.vmware.com/products/vsphere-hypervisor/index.html) This will let you run m0n0wall as a VM and run other VM's for web server, torrent, dlna, etc.
If you want to test a different OS, just download an ISO and build it. If two services don't play well together, give them a VM each and assign resources as you see fit.
You'll need a 64-bit board, preferably a multi-core CPU and the more RAM the better.
Bart...