Jingzy - I think you have nswered your question yourself - many rifles with particularly lightweight or sporter weight barrels will shoot slightly differently as the barel heats up. Probably a function of metal expanding and a barrel is never perfectly concentric.
If you really want to see how your rifle groups - shoot one round, open bolt, leave for ten minutes, shoot another, and do the same again. Barrel is thus cool for each shot. For each shot tough make sure the rifle is held and supported in the same manner.
Thats how I would sight any stalking rifle, and I would always take some final sighting shots using field positions as this will change (usually sightly) the point of impact.
In the field, what matters is your first shot, which will be taken from a cold barrel.
And you mentioned your rifle is shooting a 2 inch group at 160 yards. More than enough accuracy for any UK deer species taken at reasonable ranges.
yip what you are experiencing is a "cold zero" the one where the bullet travels down the barrel when it is still cold, for a stalker, this is the important one, as you only fire that one shot, usually.
do as was suggested, let barrel cool down between shots,and use your cold zero as your true one, not the poi through multiple shots, it should just be a difference in elevation and not deflection.
Make a note of it and adjust when on the range.
I had a new .308 Sako Mod. 75 Varminter that threw a nine at half-past seven on the target always when cold then in the bull thereafter.
Riflesmith Trevor Procter diagnosed a barrel fault and I took it back to the dealer from whom I bought it. After much hassle GMK sent it back to Finland and the barrel was replaced.
I had much time loss and aggravation but the rifle is a one-holer from cold now so all has ended well.