Next time you do it burry the head but leave the antlers above the ground in the box, you can see some of the colouration got ripped out.
If your going for the long haul after doing that place It in a nice bucket of water with a nice heaping douse of dishsoap and leave it another month or two (again, with the antlers out the water) then give it a peroxide.
Bigger bonus if you can use a bucket heater or keep it somewhat indoors out of the cold, 46C Is the ideal temperature you want as that's when fats melt for deer tallow.

will help get rid of a lot of the staining with dirt.