There aren't that many people, the great collapse happened. Also you have to remember that OP said entire universe. Another factor that comes into place is whether the enemy has prep time. Mobilizing an army of factions that were once enemies isn't a walk in the park. Goku is one man as well, you can only assume that the destiny universe will be on the defensive and be awaiting his attack. With how powerful he is I just don't see destiny universe succeeding in their efforts