Okay, but like, if not for those two specific things, prejudice against uryuoms would be solved forever, right?
That sounds right.
(Spoilers: That's not right.)
Odds are good that at least some uryuoms really did try to take over that world. This is in large part because I'm writing this, and I think it'd be boring if they didn't.
What happened between "alien colony ship crashes" and "at least some of the aliens decide taking over sounds like a good idea" is more mysterious, however.
"Ancients", unlike immortals on this side of things, are all about telling the truth.
The technical truth.
They can definitely still lie.
This is something they agreed to instead of limiting what they can do to mortal beings. Was doing things to mortal beings less of a question mark in a world filled with magic and super-powered griffin knights? Possibly!