I hear ya, but when I went to Watson's I ended up picking a color and a band that I never woudl have over the phone. I also came close to buying a second hat, and I have a good idea of what I want from that hat. I can tell what clothes will work on me and I can order them without being in the store, but I can't do that with hats yet. I wish I could.Just be aware that not all custom hatters actually have a physical location. I don't know that Gannon has a physical shop location or not, he didn't when he was still in Portland. I know that Black Sheep does not, he runs it out of his home. The only thing having a physical location does is drive up the price of their products in order to cover their expenses.
I wouldn't necessarily make it a requirement that you need to visit the hat shop in order to get a great hat. The 5 customs that I have were never done in person, I took my own measurements and gave them to the hatter. I am also lucky to have two different conformateurs that I can use to check my measurements as well.