There is quite a lot of common sense being talked on this thread.
Hearing damage generally happens due to a "dose" of noise - so loud noise for a short time can give you the same dose as much less loud noise over a much longer time period. Once you go over about 140dB(C) there is the potential for a noise to cause instant, and in some cases total, hearing loss though I've never seen a detailed treatment of this and never seen a documented instance of the total loss. Surprisingly there aren't many very good studies of the sound pressure levels produced by a rifle and a lot get it completely wrong by using meters that integrate or have a very slow rise time, or microphones unable to cope with the pressure wave. I've seen people adamant (this isn't the same as being a singer in the 1980s) that a rifle produces less than 100dB(A) for example. In the decent studies that have been done it is generally found that a centre fire will produce something just over 160dB(C), maybe 165dB(C).
Most moderators will give you a reduction in the sound pressure level of about 30dB and so they should be reducing the level of your rifle shot to below that at which instant damage can take place. The decibel scale is logarithmic and so small changes can be extremely significant: 3dB represents a doubling of power for example.
If your rifle is below the level at which instant damage can occur, as it should be if moderated, you are then into considering the "dose" that you will receive from a rifle shot and then you enter a world of considering the SPL and the duration and so on. I believe one of the forestry organisations looked into this and produced figures relating to the number of shots it was considered "safe" to fire in a day with a moderated rifle, if you hunt around the web you might find the research somewhere, this might be it but I don't have time to read it just now to check:
https://www.forestry.gov.uk/pdf/crsafety4.pdf/$FILE/crsafety4.pdf
In the end it would be unwise to tell you what to do and measuring and calculating your specific personal exposure with your setup is extremely complex indeed. As others have pointed out, however, if you are firing a lot of rounds or being exposed to others firing a lot of rounds in relatively short periods of time, say a single 24 hour period, then you need to limit your dose (dose is the key word here) by wearing additional hearing protection. Remember damage is caused not just by a maximum sound pressure level but also by time (i.e. number of shots) you are exposed to this maximum SPL and you also need to consider other noise that you will be exposed to in that time frame as that will also have an impact. With a rifle shot being of such short duration, and assuming you are not then going to spend the afternoon operating a pneumatic drill, it is likely that a single shot, maybe even a small number of shots, will be well below the level at which damage may occur however it is down to you to satisfy yourself of this as others can't take responsibility for your safety.