So I understand the pros and cons to both methods and I'm still on the fence... looking for some alternate opinions to help me decide 
I have pre-wired my house and have a dedicated equipment room in the basement which will house 2 Middle Atlantic 43u racks. I'm planning a distributed a/v system, with some basic automation type stuff. Here's what I have in terms of wire:
CAT 6
4-5 cat drops to every zone in the house -- this means about 40-50 cables. 2-3 per zone are extras, with no immediate use.
Speakers
20 speakers (10 zones) for whole home audio etc + a 11 (7.4) speakers for my dedicated home theater.
Balanced XLR
11 (7.4) to the theater for possible active speakers in the future (won't be used initially).
RG6
1 cable to every TV location for a total of 6. These were installed for good measure and I won't be using them right away (if ever).
===
So my thinking is that it would be nice to terminate everything so that the racks can be kept as neat as possible. This way I don't have a ton of extra cable coiled up next to the racks.
It also allows me a bit of flexibility in terms of gear moving position in the racks. The system is likely to change and I feel like having the patch panels would allow me to make these tweaks easily as necessary in the future.
The concerns I have -- extra connections:
1. For the upstairs audio zones, would I notice a difference in quality with direct wiring vs the patch panel? Fidelity is not the first priority up there obviously. How about for the theater - If I go with the panel for the rest of the house, should I just terminate the theater speakers here too or should I run them directly? Fidelity is more important here.
2. I plan on using HDBaseT extenders for 3 of the zones upstairs. I will not be "cheaping out" on the extenders -- but since I've only wired with Cat6 UTP, how likely is it that I may run into issues (even if a lot of care is taken to punch down at the panel)? From what I gather, some people have success while others run into issues.
Any thing else that I'm missing? I'd love to hear some thoughts...

I have pre-wired my house and have a dedicated equipment room in the basement which will house 2 Middle Atlantic 43u racks. I'm planning a distributed a/v system, with some basic automation type stuff. Here's what I have in terms of wire:
CAT 6
4-5 cat drops to every zone in the house -- this means about 40-50 cables. 2-3 per zone are extras, with no immediate use.
Speakers
20 speakers (10 zones) for whole home audio etc + a 11 (7.4) speakers for my dedicated home theater.
Balanced XLR
11 (7.4) to the theater for possible active speakers in the future (won't be used initially).
RG6
1 cable to every TV location for a total of 6. These were installed for good measure and I won't be using them right away (if ever).
===
So my thinking is that it would be nice to terminate everything so that the racks can be kept as neat as possible. This way I don't have a ton of extra cable coiled up next to the racks.
It also allows me a bit of flexibility in terms of gear moving position in the racks. The system is likely to change and I feel like having the patch panels would allow me to make these tweaks easily as necessary in the future.
The concerns I have -- extra connections:
1. For the upstairs audio zones, would I notice a difference in quality with direct wiring vs the patch panel? Fidelity is not the first priority up there obviously. How about for the theater - If I go with the panel for the rest of the house, should I just terminate the theater speakers here too or should I run them directly? Fidelity is more important here.
2. I plan on using HDBaseT extenders for 3 of the zones upstairs. I will not be "cheaping out" on the extenders -- but since I've only wired with Cat6 UTP, how likely is it that I may run into issues (even if a lot of care is taken to punch down at the panel)? From what I gather, some people have success while others run into issues.
Any thing else that I'm missing? I'd love to hear some thoughts...