Those taken into exile would have been taken only from Jerusalem. This was a fairly common practice. The Assyrians took about 25,000 people from Samaria, capitol of the northern kingdom of Israel. Of the descendants of those taken by the Chaldeans, most, having been born in the vicinity of Babylon had no desire to leave. Since the exile began in 587 BCE, and the decree of Cyrus was in 538 BCE, a period of 49 years those who were 20 at the beginning were nearly 70, if they were even alive, when Cyrus made his decree.
There seem to have been about four "returns" under succeeding Persian kings. These would have been, in all probability, small groups of self-selected zealots. Their intolerance toward those they didn't consider sufficiently Jewish is highlighted in this passage from the Book of Nehemiah (Neh.13:23 - 25):
In those days also I saw the Jews who had married women of Ashdod, Ammon, and Moab; and half their children spoke the language of Ashdod, and they could not speak the language of Judah, the the language of each people. And I contended with them and cursed them and beat some of them and pulled out their hair; and I made them take an oath in the name of God, saying, "You shall not give your daughters to their sons, or take their daughters for your sons or for yourselves."
And let's not forget the famous curse in Psalm 137:
Remember, Yahweh, what the Edomites did
on the day Jerusalem fell.
“Tear it down,” they cried,
“tear it down to its foundations!”
Daughter Babylon, doomed to destruction,
happy is the one who repays you
according to what you have done to us.
Happy is the one who seizes your infants
and dashes them against the rocks.
Of course the Edomites / Idumeans were later converted at the point of the sword to Judaism. And eventually, they were installed as priests and High Priest in the Temple, at the command of Rome, even though they were not scripturally authorized to serve those roles.
At one point, a Galilean end-timer, a holy roller from Galilee, would curse the Edomite-led Temple and claim that God was soon coming to throw them into the fires of destruction like weeds, at which point his following, the true Israel, would enter into the Kingdom of God on Earth.
The next morning he was crucified by Pilate for rabble-rousing.