I'm sure one of the actual devs could give a much more accurate/comprehensive overview, but here's what I loosely know/remember at least.
- You should be able browse the whole AI code with either RMXP or something like Gemini to view the scripts file, but last I heard the E18 AI code was like, 50k lines long, so good luck. In general the AI uses a system where it will assign a score to the different current options it has (moves, item, switches) and then increase/decrease those scores based on tons of different factors (types, current hp, field, past turns etc), then picks the best scoring option.
- I'm not sure if there's a generic AI 'level' that varies with trainers, but I know a lot of the unique trainer classes have extra AI stuff attached that's intended to make them use their designed strategies better. The Ace thing might also vary from trainer to trainer because of this? Or more generally, I would guess that the Ace probably has reduced AI scoring when considering switches, but a significant enough difference might be enough to overcome that bias.
- I don't think there's a flat hard-coded pity chance exactly, but it does sometimes change moves to try and save pp, catch switches etc, which can backfire on it reasonably often. I might be misremembering, but I also do think there was something about it giving extra score to moves that have a chance to KO over guaranteed KOs as a little bit of a pity / tense moment creator?
- Yes the AI does have some memory of things it's seen in the past of the battle I believe, but I don't think it ever 'cheats' with knowledge that it shouldn't have access to or knowing the players action before deciding unlike some 'hard' romhacks/fangames...
- Yes Rejuv (and Deso) should be using a lot of the same AI base, but they'll have their own unique trainer extras layered on top at least.
Also, the AI will have tons of improvements/changes for E19 once that releases, so........