Yes this is exactly what I do, I am right landed so the rifle is slung muzzle down over my left shoulder so the bolt is not inadvertently closed against my body, on my ground you most certainly don't get much time for the shot when it presents itself.
Sika for me may be less than 50yds away when one is found the rifle is moved slowly into position the bolt is then closed slowly and silently and if the shot is a safe one I take aim and squeeze the trigger, on more open ground where shots are say 100yds or more keeping the chamber empty until the shot is presented is good practice.
No matter whichever way we handle a rifle when stalking and as has been mentioned by many already it is good muzzle awareness at all times which is most important, when we handle any firearm "ESPECIALLY" when it is unloaded that muzzle awareness must play rule no "ONE" get into that habit at all times and it increases safety margins for all concerned.
D
I would be worried of the bolt catching and ejecting a live round without the bolt locked, I carry muzzle up safety on and part of the reason for this is that I with my right hand I can check the safety is on by checking the bolt is locked down, which I do very regularly. The other reason is from this position I can quickly and smoothly get on to the quads I carry in my left hand whilst having complete control of the muzzle, I don't just shoot deer and being able to quick, whilst still safe, is sometimes needed.