There were MANY different factors, remember the downfall happened over a long period of time. It depends on whether you want to class the very end (When Constantinople was taken by the Turks) as the downfall, or the gradual reduction in size that happened over many years.
As well as looking in books specifically on the Roman Empire, I'd also reccomend looking in books that look at a number of the great empires of the world - these will then have good chapters focusing on the cause and effect of decline leading to the actual downfall.
Motyl (2001) "Imperial Ends: THe Decay, Collapse and Revival of Empires" is a good start.