It’s not entirely down to the females, but since both carry the genes, you’re exactly right in saying that targeting just males is pointless.
If you have an accurate pedigree, then you can figure out what genes a female has, and cull females appropriately - ie. although the female doesn’t have antlers, you’d be able to work out if she was a carrier of the dodgy gene.
Since we never have accurate pedigrees with deer, we have no way of knowing which females we need to shoot to improve antler characteristics.
As @Freeforester and others have said, age structure and food are just (if not more) important than genetics. And those are variables that you DO have some control over.
