Could you imagine all the inhabitants of the destiny universe in one place at one time? How much space does that require. If we are thinking logically, even if they were they would be fighting each other. You are aware that Goku can destroy the planet with a kamehameha that would charge much faster than a spirit bomb.