A $20 Hat Jack is a good investment. Don't use it to stretch the hat but set it at your hat size and insert in the hat after use. It will help maintain the hat to your size. Or if you have wordworking skills you could cut yourself a band block in the correct size and insert that into the hat each night.
Hat Jack is on my shopping list, thank you.