While perhaps not the "proper" way to do it, a small--and new, so still sharp--drill bit in a Dremel tool or similar will cut clean holes in hat felt. Easier to do than a leather punch because all you need for backing is a ball of wadded-up newspaper. It also allows you to place the holes exactly where you want them.
Figured that out while trying to duplicate the ventilation holes in a hundred-year-old military hat. Not something you use every day, but a handy trick to have once in a while.
This is more along the line of what I was thinking. A small sharp diamond tip drill bit.
I already have the dremel and I can always use another HQ drill bit. Win-Win!!!
Thank you!!!