"Unity" "All-Under-Heaven"

I'm reading this book about The Great Wall by John Man, and I came upon that phrase. And I wondered: it seems that humans have always had this desire for unity, but it comes out in very different shapes, with different outcomes: empires, monasteries, clubs, families. And of course it is shaped by the very different individuals: dictators, principles, saints, matriarchs & patriarchs. We might not like the results, but the initial intention was for unity.
What changes a good thing into a bad thing? A desire for unity and harmony into a desire for all others to be like me, do things my way!