I leave the Tower all the time. In fact, public record will show I went to Mercury in search of Osiris awhile ago... Whenever I leave, I do it under the Guise of a Guardian as to not cause any suspicion. I also make sure no one will come in search of me. Usually, I do it on weekends, as people are seeking Xûr, Lord Saladin, or Brother Vance instead of me...