I have played around with this stuff a lot and have a way that I am mostly happy with, where I'm not happy is mostly a limitation of MP3 tags not support multiple genre per file tags and a way to tag songs with explicit lyrics (iTunes does it but you can't control it).
The main constraint I have is that I ONLY have mp3 files so if you are using other formats than my structure probably won't work.
Where the work is (for me anyway) is getting the genre right and the cover art in. I use CDeX to rip and it hooks into CDDB to get the tag information. I am very careful to get the genre (Alternative vs Alt. Rock vs Alternative Rock) and artist (REM vs R.E.M.) right at rip time. Also of interest is what to do about multi CD releases. I tag the track number on the second CD starting from the end of the first. That way when the CD is played the order is maintained. Be careful on this because online databases are really bad at getting spellings right (disk1, disc1, (Disc1), disk 1 etc).
I have a single folder called MP3 that is made up of folders filled with Artists - ABBA to ZZTop. Each one of the artist folders has a folder for each CD. The songs are labeled as Track#-SongTitle.mp3. This structure is mostly for me to be able to find things. CDeX does this for me (as do most other rippers). The software, iTunes and SlimServer make their own databases that don't care about this structure too much, it is mostly for humans looking for files.
When I started I had a haphazard collection of file names. I use a program called MP3/Tag Studio to rename the files based on the MP3 tags. It does a very nice job of batch renaming. It is also easy to set all of the tags of files in a given folder. Changing R.E.M into REM, or 'Beatles, The' into 'The Beatles'. This program is very competent but other tools are surpassing it.
The final problem that I've had to deal with is album art. If you want album art to show up on your HA pages or on your iPod you need to get art at the right resolution. What looks good on the iPod doesn't look so hot on a 1600x1200 screen. I look for resolutions of album art that are 300x300 or bigger since I am targeting an automation screen as well as an iPod.
I have chosen to put the album art in each mp3 AND as a folder.jpg. This makes the files a little bigger but I'm willing to do this to keep things simple. YMMV
I do most of downloading art by hand by surfing the web. There are tools out there that will surf your collection and download art from various sources like amazon. I have not been too happy with these tools though if you only have newer, popular music they should work. Even under those constraints, quality can be hit and miss. You'd be shocked at how bad the same CD cover looks from different sources, some are poor quality scans and others look like they came right out of PhotoShop.
I use iTunes to apply album art because it lets me drop a bitmap into a all of the files on an album just by selecting the songs, right click to Get Info and paste the image into the image box in the Get Info dialog.
One final thing that I've played with and really love is
MusicIP Mixer. It is really cool because you can say that I like this song, say 'Rock and Roll Ain't Noise Pollution' and it will make a mix of songs that are 'like' it. You can control the mixes with many parameters. It comes up with mixes that are usually quite cool and you'll feel like you have a bigger collection because it pulls such cool stuff. This is where I wish explicit tags were standard because it will pull up something not so nice when the kids are listening. I'm surprised they haven't been bought by the big boys yet.
