It shouldn't make much of a difference to accuracy at all.
No offense but you'll probably find it's down to flinching.
If you shoot with a mod or brake on all the time, you become less recoil tolerant, that's why many RFD's I know keep a boomer around just to get them suitably shell shocked and ready when a customers light sporting, heavy recoiling monster needs testing & rough zeroing.