I think we might need to tread carefully in classifying deer along with 'pests' or 'vermin'. In England & Wales the laws that allow landowners to overide sporting rights and control pests and vermin do not AFAIK include deer - Ground Game Acts of 1880 and 1906, Wildlife and Countryside Act 1981, General Licences - I'll stand to be corrected if I've missed something, (maybe a Notice under the Agriculture Act 1947? But I believe this may be more about out of season shooting as per S.7 Deer Act?). Do you have any further references?
EDIT: I suspect that S.7 Deer Act might actually give a way in as it specifies:
Damage by deer. [s.7 Deer Act]
On any cultivated land, pasture or enclosed
woodland, deer may be shot during the close season
and a shotgun can be used in certain circumstances
[see below] in order to prevent damage. This action
may be taken by the occupier of the land in person
and with his written permission by:
Any member of the occupier’s household
normally resident on the land;
Any person in ordinary service of the occupier
of the land;
Any person having the right to take or kill deer
on the land;
Any person acting with written authority of a
person having that right.
Subject to that person having:
Reasonable grounds for believing that deer of
the same species were causing or had caused
damage to crops, vegetables, fruit, growing
timber or any other form of property on the
land and;
further damage was likely to be caused and
was likely to be serious;
The action was necessary to prevent any such damage
Depends if the OP can satisfy the criteria of being in the service of the occupier. If not, then it again comes back to who has the 'right' to take the deer doesn't it?