If you are not trying to stream any media (and don't have a big media repository, i.e. thousands of cover art images), and the server is just serving a few clients, then it can be a quite light weight box. It needs to have sufficient I/O to get the data to the clients quickly. But it probably doesn't need much CPU. In those cases, the clients may be working harder than the server, just because the server is having to do so little.
But, if you are going to be setting up a good bit of scheduled and triggered events, or a sizeable media repository that multiple clients are accessing (not the media data itself but the meta data), running the web server, or running the new RIVA server to support some small devices, then you'll need more CPU and memory as you want to do more of those things.
You probably couldn't serve up much media on a really light weight box, and also provide snappy response to the clients.