For me, it's pretty much all of the Safari Zones.
Generation 1's, while the best of the five IMO, still wasn't fun to navigate through, especially considering a full item run of the place would take at least 2-3 run through's of the zone, if you knew what you were looking for. Additionally, I can't stand searching and searching for all the rare Pokemon in this safari. As a kid, I was never able to catch Kangaskhan, Tauros, Chansey, Scyther, and Pinsir, making me very irritated towards this place for hoarding all the hard-to-get Pokemon.
'Generation 2's', aka the one in HGSS was a pretty fun idea going in, but the execution particularly drove me nuts. Although a concept of different areas that are swappable was good, it's the way that you have to get up the "points" to get the rare Pokemon that bothered me the most. Being a completionist my whole life and knowing that that was the only way to get certain Pokemon in HGSS really drove me a little nuts (enough that a combination of Gen 4's Swarms, this Safari, and the entirety of the Pokewalker put me off from completing the NatDex in this gen).
Generation 3's is not the worst, but not the best either. The items in this one were less plentiful than generation one's, and the Pokemon weren't too hard to find in this one. The two mechanics that really killed it for me were the bike-exclusive areas and the Pokeblocks. The former was annoying because I really hated the idea of having to fly back to one city just to exchange bikes just to go get a Rhyhorn (for example) and it just seemed like bad game design. Luckily, this was alleviated in ORAS by allowing the player to, at one point, have both bikes (and get TM Energy Ball in the Safari Zone!). The latter point was not necessarily a game breaker, it just seemed like a worthless feature to require the Pokeblock case for entry, especially when I would forget to pick it up, and then remove the rock/bait options from FRLG.
Generation 4's is the bane of my existence. The god-awful marsh walking speed, lack of any cool items to pick up, the frustrating and luck-based binoculars system to get the 'good stuff', and the stupid train system make this the worst safari yet. It is really that bad.
Of all the tropes that were not reintroduced in Gen 5, I'm glad that they left the safari zone out.
Generation 6 is a novel concept, and I can't really hate it that much because it is more of my own gripes that make me angry for this one. I didn't like having to resort to frequenting Serebii forums and f. adding randoms just to collect all the Pokemon in pre-bank XY, so this was a good idea that was just inconvenient to someone trying to catch them all.
Oh, and Safari balls suuuuuuuuuuck.
Maybe if they made it into a cool mechanic with a safari-bound Jeep that gets to roam around, I would enjoy the feature.