If you are measuring the group size by the "extreme spread" method - that is, the distance between the two furthest shots from each other in the group - then it matters a lot.But then how is shooting a 20 round group any different than shooting 7 3-round groups (20 doesn't go exactly into 3 but you get what I mean) assuming you're allowing the barrel to not overheat and keeping all other variables the same?
In three shot groups, only one shot in each group is not contributing to some information about the group size. But with so few shots, you do not know much about the group centre. The group might be a tight group but there is a significant chance it is three outlier shots in a much larger possible group.
In a 20 shot group, 18 shots contribute no information at all about the group size - but they do give you a much better idea where the group centre is. So it is a balance that has to the struck between knowing where the gun is actually shooting and how big the dispersion on the group is.
It turns out that the most efficient number of shots in a group to minimise the number of shots fired for a given confidence level of the group size is 7 shots in each group.