I think perhaps the question misses the point.
The territorial restrictions are part of a range of additional conditions which the FLDs are allowed to apply at their discretion.
A better question might be 'Should FLDs stop applying additional conditions to FACs where these neither improve public safety nor protect the peace?'
In my view, that would mean most (sporting, at least) FAC-holders would have no additional conditions at all - thereby getting rid at a single stroke of one of the commonest sources of disgreement and clerical error in the issuing of FACs.
The FLDs would (and, arguably should, as they've had that right since 1920) retain the right to apply conditions if these were indicated: for example, folk who undertake only target-shooting might be restricted to shooting at clubs/ranges without inconveniencing them at all - but on the whole, the less faffing about they have to do with inventing conditions about using this rifle here for this quarry, and that one at a range for targets, and so on, the more time they'll have for proper checks on the suitability of the actual applicant: which really is the main thing to keep the public safe.