The engineers, surveyors, doctors, field hospital, ambulances, wagon train, artillery, etc were parts of the normal legionnaires, they had secondary duties.
The camp followers would live outside the camp, there are lots written about females who were attached but not married to men and that lived outside the camp.
I believe the cavalry had it's place inside the camp, I'm not sure thou. That's the impression Bellum Gallicum has given me so far.