I remember one Prabhupada disciple telling me how he knew that in his previous life he had parents who were demi-gods. I had one where I had parents who were demi-gods but i was an asura. Prabiupada was from Goloka yet He came here in this World, He did'nt fall. SO i guess we dont fall from Goloka but from somewhere else. If you say we fell from Goloka then you would have to be a pure devotee, as only pure devotees exist in Goloka.