I'm not sure it makes much difference to the buck itself. Really the only reason we leave them until they're out of velvet is because of our own appreciation of the aesthetic.
This is a bit like the thread around shooting does in March, with there being arguments (or at least opinions) on both sides.