That's not an easy question to answer. In general, signal quality on a wire degrades the longer the wire is due to increased resistance of the wire as well as increased capacitance. The frequency of the signal also plays a role.
A larger gauge wire will lower the resistance, which tends to help. But it also changes the capacitance. The type of insulation and the spacing of the wires from each other also makes a difference in capacitance.
Finally, timing is another factor. In some cases, the time it takes for the signal to get from one end to the other and for a response to come back may matter. In those cases, changing the wire gauge won't help.
The easiest test would be to try a larger gauge wire and see what happens. You could also call HAI/Leviton and see if they can offer any guidance.
I was re-reading the Omni manual again about the maximum wire lengths. I think you said you had 4 home runs for expansion enclosures and another 3 for consoles (or it is one home run for all the consoles?). The way I interpret their wire length rule, I would say that the total of your home runs should be less than 142 feet if you have 7 home runs, or less than 200 feet if you have just 5 home runs. That sounds like a rather severe restriction, and would be another question to ask the folks at Leviton about.
