If we ever dare to challenge the Traveler then I believe it could just zap our light away and we would be dead again...

Yeah but if she dies I don't think she'd be able to revive cause that's the ghosts duty right? If the traveler removes the ghosts then we're at a distinct disadvantage as we'd be like all the dead people from the gap and of lore. Many chose to abandon their ghosts and later died permanently, others were lost in time so we know a ghost's power limits them from bringing things back through time, and others had their ghost's light drained and then died because it was powerless. So ghosts can be turned off, left behind, too far out of reach, and drained of powers which almost always leads to a guardians death as well. Other than Eris morn and probably the speaker because that guy is a rather suspicious character.