The way Ivo looks down at Earth when Gerald says "You have nothing down there. No one who cares about you" cuz Ivo was thinking about a certain someone who does care about him, the person he left on Earth all alone.
And he tells Gerald "But I have you now" not something like "You're right, I have no one except you". He can't even say that no one cares about him on that planet because he knows that it's not true. He was fully aware of Stone's feelings, he couldn't even deny it anymore. I think that this was the moment he started regretting how he treated Stone, it wasn't Gerald's betrayal that made him feel that way. But when Gerald mentioned how he wants to destroy the whole planet and how all humans deserve to die, Ivo was thinking about Stone who will get killed if the planet explodes. A person who cares about him, and the person he cares about. Not only that Gerald betrayed Ivo but he threatened to kill Stone too and Ivo couldn't let him get away with that.