Personally I don't hate baseball caps per se, but I hate their overuse. Actually in the 1960s, because of their dominance in the culture previously, I used to hate fedoras.
I'll go even further. Because of the assertiveness (in movies I'm talking about, that's what I saw) of the wearers of fedoras and Open Road type hats, at that time I hated Open Roads also--think Cool Hand Luke. It was the association of highly assertive men trying to steamroll their worldview on everyone else (at least in the movies) that caused me to not like them. Same with fedoras. Now, forty-fifty years out, these hats are no longer associated with highly assertive people. Now, they're just hats, a style choice, and a minority one at that. Looking at it that way, now I like these hats. That's my attitude toward hats.
That's a healthy attitude. Of course, I think baseball caps are fine on baseball players when they are playing baseball, being photographed for baseball cards, etc. I guess I don't even have much of a problem with baseball caps that have baseball team logos on them. I think my problem arose from cheap, adjustable, baseball caps with advertising on the front. It absolutely drove me up the wall and still does. Turn it around so the adjustment band is on your forehead and it is doubly awful.
I actually own a baseball cap. It has a Chicao Bears logo on the front. It is dark blue with an orange bill. Quite tacky. I bought it to wear at a football game at Soldier Field so that I could identify myself with the good guys. I haven't worn it since.
I'm not against caps. I have a few Kangol caps and a newsboy or two. I wear them from time to time when I want to stress informality - jeans and T shirts only. Heck, I wear western hats most of the time so I stick out like a sore thumb in rural Indiana. Sometimes caps are de rigeur. I think someone driving a 1960's Austin Healy with the top down would look almost out of place without a tweed cap.