That's logic, isn't it. First of all you don't want the bullet to hit it but the alignment is important. Not only for left/right but also for up/down. Once the bullet starts traveling down the barrel it pushes an amount of air in front of it. Each irregularity will affect the airflow and will have its effect on the bullet trajectory and spread (as the tip of the bullet is in that 'air column'. The flow of the gases behind the bullet is important too, but the effect is less. It is for the same reason that the barrel crown needs to be precisely machined/finished. If there is more 'air resistance' on one side the bullet will be pushed the other way. Maybe ever so slightly, but it will be affected.
Sorry mate, got you all wrong there. It depends on the brake. Some are directional and others symetrical. For the symetrical ones you want the balanced (so holes up/down & left/right or all under 45 degrees) the directional ones are to be fitted as intended.