What if, the giant walls built around the last city, and where you started after being revived, weren't built to keep the Fallen out, but were in fact built to keep the Speaker and "guardians" in. That perhaps the fallen and other enemies came to Earth in some massive battle to chase and stop the Traveller, but it ended up in a stalemate, the traveller exhausted, the Fallen just about stranded on Earth(or they leave a contingent to patrol the planet) and if any of the Travellers Guardians attempt to gain technology or interstellar flight, then they "jump" their ships to earth, to stop the Guardians from spreading. Of course, it could be that Guardians are neither good nor evil, it just depends on their actions, hence being able to wield the sword