Despite what many other people say about not cleaning a .22LR, it may have been for this very reason. I think it all depends on the particular rifle (and possibly how worn the barrel is). My Annie would send the odd round (about 1 in 100) well off target until I cleaned it properly, and then kept it clean. I then started smallbore target shooting and was told that lead accumulates in the barrel if you don't clean, then every so often one particular round will pick some of this up and be a "flier".
I was very sceptical about this, but gave my barrel a good clean and its accuracy improved dramatically using exactly the same batch of ammunition, i.e. much better grouping generally and no "fliers". I now clean all my .22LR rifles every 100 or so rounds with patches on a pull-through, and every 1000 rounds with a rod and brushes, in both cases using solvent to start (Bore Tech's Rimfire Blend) then dry patches. For my rifles it only takes between 3 and 10 fouling shots after cleaning to be spot on again, and after a year of doing this and about 3,500 rounds through my Annie alone I've only had one "flier", and I suspect that was a round with not enough powder as it sounded very quiet compared to normal and fell very low.
I'm not saying that this is definitely the case with your rifle, and I know many people that don't clean their barrels and seemingly have no problems, but it might possibly explain why one shot clipped the baffle whereas hundreds/thousands of others hadn't. Just be aware that it is possible to do more harm than good when cleaning if you don't do it properly and carefully, but there are plenty of other posts on this issue so I won't expand.
