Well the earthquakes did plenty of damage, and their survival is due to frequent repairs as much as to the original architects. It is interesting to see how the buildings that still mattered to them in the fifth century, such as the Colosseum, the Circus Maximus, the Palatine, the baths and the churches, were rebuilt and it is mainly these buildings that are still standing. The other buildings were left to their fate; most pagan temples, parts of the Forum, some basilicas and porticoes that were no longer needed by the smaller population.
The floods may not have done much in the way of physical damage, but they would have greatly disrupted the daily lives of the citizens, perhaps rendering parts of the city uninhabitable.
For anyone interested in this part of the city's history, I would highly recommend visiting the Crypta Balbi (in Rome of course). It's an excellent, modern museum that focuses exclusively on the transition of the city from Late Antiquity to the Middle Ages.