It's my understanding that the Fallen were rather late to the party on Venus and that the collapse and the destruction of the academy and the rest of our colonies on Venus was mostly due to the Vex and the Fallen arrived later to pillage what was left. I may be wrong though