I go to international matches at Twickenham regularly. Never had, nor seen any trouble whatsoever. Once got a couple of tickets to watch them play South Africa in 2002, one of the autumn internationals before the World Cup that England won. South Africa were desperate to have a good showing and things got pretty intense on the pitch, a blatant and obvious late tackle on Jonny Wilkinson starting things off well.
I was there with my 10 year old son and the tickets we got were two that had been returned having been corporate tickets bought by a South African bank. We were surrounded by South Africans, me with my O2 England cap on a-la-Clive Woodward and my son in his England shirt. We had a whale of a time, with those surrounding us feeding us biltong and generally going out of their way to make us comfortable. They were a bit disheartened towards the end as England won comprehensively, but at no time did we feel anything but safe.
Other sports fans, football in particular, could learn a lot from rugby fans. Football fans obviously feel as strongly as rugby fans do about their team, but that tends to, a lot of the time be manifested by footie fans shouting and chanting insults at the opposition. Some of it is racist, as we see in the news regularly. There's no wonder things spill over into violence on occasions. Compare that to a rugby match, where out of respect for the kicker of an opposing team, silence is maintained wherever you are (except Scotland, I'm sorry to report).