yip what you are experiencing is a "cold zero" the one where the bullet travels down the barrel when it is still cold, for a stalker, this is the important one, as you only fire that one shot, usually.
do as was suggested, let barrel cool down between shots,and use your cold zero as your true one, not the poi through multiple shots, it should just be a difference in elevation and not deflection.
Make a note of it and adjust when on the range.