For the type of shooting the OP describes, a solid mid-level, 2-8, 3-9, 3-12, 4-16 will do the job for many years to come. Agree with the OPs initial observations - under normal conditions, you're gonna struggle to pick a mid-level over a high-end scope if you didn't know in advance which one you were looking through. I would even argue that the last of the light performance of some of the current mid-level scopes is very very close to the NFs, S&Bs, etc.
That said, for short to medium range deer hunting, in the 90s I found buying close to the entry level model of a high quality brand served me very well, to the point that I was proud to hand them down to up-and-coming cousins, neices and nephews. And the rellies were stoked. But these days, as my fitness and eyesight slowly deteriorate, I can't see a significant difference between say a Zeiss Victory and a Vortex Viper PST in a woods / hill country hunting scenario, so the Vortex will win every time because of price and the no-questions-asked replacement warranty.
Where I have experienced significant performance differences between brands is in high magnification scopes for longer range varminting, e.g. 6-24x50. At the maximum magnication, on a bunny at 400m+, you want good quality. Scratch that, you want really good quality. Where this becomes problematic for the not-rich-or-famous is the hideous cost of these scopes; the combined capital cost of the rifle and scope makes for very expensive pest control... So as other priorities in life have come along, again, I've moved into the middle of the market. But I do notice the difference.... there was a S&B that was sold in Canada for slightly more than I paid for it, which seemed like a good idea at the time, but man I miss that scope.