In my experience with 22-250, 243, 25-06, 270 and 308 chamberings the smaller faster bullets kill deer every but as quickly and cleanly as the slower, heavier ones. No scientific reason offered, just based on how quickly I see them hit the ground.
There is absolutely no doubt in my mind that with a suitable bullet the 204 will also kill roe sized deer, why wouldn't it? We have been lucky in Scotland to have had the pleasure of being allowed to use centre fire .22 chamberings for many years. I have a 243 that is going to be rebarelled in 22-250 as I've got a good few roe to shoot nowadays and think its the perfect chambering for those deer with a soft point 55grn bullet.
I dont think think the OP has asked a daft question at all and I don't think a response of 'why would you' is much of a reply. He could say, 'well, why not', if its legal.
The damage a small, but extremely fast bullet causes inside the chest of a deer is incredible, far more traumatic than a bigger, slower and tougher bullet. It looks like a hand grenade has gone off inside them, but still with a very acceptable exit hole. I guess that's down to the speed...
I know this is relating to 22-250 so not quite what the OP was asking, but relevant none-the-less for the doubters of smaller calibres.