Swagger can be incredibly annoying but can be solved by quite a few things. First, we have Prankster or outspeeding Taunt. Second, we have Magic Bounce. Third, we have a luck based ability known as Wonder Skin. With Wonder Skin, the accuracy of Swagger would be 50%. Fourth, for those who use special attackers, always make sure your attack stat is as minimized as possible to get the least possible amount of damage dealt to yourself by swagger. Fifth, throw a Lum Berry on your physical attackers from time to time if you're truly afraid of dealing massive damage to yourself from swagger. Swagger + Lum Berry = Profit.
However, swagger can be incredibly devastating when luck isn't on your side. When having such high stat boosts and a lack of defense, any hyper offensive physical attacker could easily ohko itself with confusion. Not much more I can say about its effects as that's all there is to it.
Baton pass, also, can easily be foiled by a few simple things. Once again we have a Prankster or outspeeding Taunt. Baton pass is incredibly situational. For example, smeargle using a shell smash could be difficult to track what pokemon could switch. But dragon dancers, swords dancers, belly drummers, or nasty plotters can make it much more easy to predict what might come in from the baton pass. For example, a belly drummed smeargle would baton pass to a salamence. If you have your greninja, let's say, you can easily ohko the brought in salamence, ruining the train. Prediction is the easiest way to destroy a train aside from taunt.
So all of these moves are easily preventable, so I doubt they truly need to be removed for any reason.