It's up to individual FAC holders to question any police officer calling unannounced 'to inspect security' if this type of thing is not to become the norm. The HO Guidance is quite specific about the circumstances in which unannounced visits should be undertaken - "based on specific intelligence in light of a particular threat, or risk of harm" - and not just a spot check, which might be construed as an attempt to try and catch FAC holders out (aka a fishing expedition).
I'd personally want to establish the reason for any unannounced visit and who it was authorised by before deciding on a course of action as; "to mitigate any misunderstanding on the part of the certificate holder the police must provide a clear and reasoned explanation to the certificate holder at the time of the visit", and if it wasn't based on "specific intelligence" etc., or no "reasoned explanation" was forthcoming, then it would certainly result in a refusal and a letter or two going in to the appropriate people including my MP.
Yet another step down the slippery slope and I fail to see why, as the holder of an FAC, I should be subjected to unwarranted invasions of my privacy. Whatever next, Environmental Health officials demanding entry to see if there's anything in the fridge that's past it's best before date?
Refer to HO Guidance Chapter 19.11 & 12 for references.