Its not quite as black and white as that though is it? I do know, we have farmers in my family. I'd be happy to see high taxation on imported foods so that our farmers can be paid a proper competitive rate for the produce that they grow. For just one example, a third of the lamb consumed in the UK is imported. We should pay more more for decent food. Our consumer society is driven by how cheap the food is, rather than how good it is. That would help to wean our agricultural system off benefits and help to make farming into a proper industry. In turn the tax burden could lessen too, I think we all appreciate that grant aid of some sort will always be required to compensate farmers for countryside stewardship schemes that would otherwise not be undertaken for the reasons you mention and I don't think anyone would begrudge that. However whilst hard working taxpayers pay so much towards countryside upkeep then I'm afraid that there will be some sort of proprietary feelings towards it whether we like it or not. I certainly wouldn't want to see yours, mine or anyone else's efforts destroyed by stupid unthinking people but I'm not sure how we police that.
There is also the question of a number of extremely wealthy farmers and related companies that exploit the grant system to make enormous profits for very little effort, the 'slipper farmers' in Scotland for instance have been pillaging the system for decades with some individuals receiving millions of pounds per year, using portions of it to acquire vast tracts of poor land and further compound the income. I don't see them ploughing their fields with their Bentley Coupes. Meanwhile Lakeland, Welsh and hill farmers elsewhere work themselves to the bone for very little reward.