There were far more wild areas like forests, swamps etc.
For example Romania was heavily forested and the wildlife comprised animals that became extinct by the time of the modern age like aurochs, bison, wild horses, saiga antelopes, beavers etc
In such a marginal area the ecological impact of the empire was minimal.