I guess it's a matter of perspective. One of the things about ancient cities is that much more of the area was devoted to public space than is the case today. So if you were on the Palatine you'd have a pretty good view of the monumental areas of Rome, the forum and the Campus Martius.
The nine-storey brick tenements at the base of the Esquiline were precisely because something like sixty percent of the population of Rome occupied twenty percent of the city's area. The other eighty percent was more marbly.