Not broken anything yet, but I do worry when using brute force in this way. I suspect you might be able to drop it out more easily from inside, but that would mean prising the body off which is another experience I try to avoid.
So much easier with the Bachmann coaches where the bogies unscrew.