It’s okay to outgrow people.
If they impede your progress, make you feel less than, or drain your energy, it is entirely your right to break ties with them for the betterment of yourself.
It isn’t selfish. Not if you’re hurting by keeping them close.
Would you hold your hand in a blazing fire just to keep a log propped up? If you let it go, it won’t stop being a log, and it’ll stay standing. Not only that, but you’ll stop getting burned.