Not at all. Take gladiators for instance. Professional lanistas weren't the only owners, military officers used them as bodyguards and trainers, even private citizens sometimes had a troupe among their possessions for rent. Cicero, in one of his letters, praises his friend Atticus for the splendid troupe he owns. But then I suppose the association with virile masculinity more than compensated for official infamy.
Prostitutes might be a little different. Wealthy men could of course have any slave they wanted on demand. Wives would suffer of course, that was not the correct behaviour of a Roman matron. I do note however that some of the urban villas in Pompeii have alcoves in the back wall which they could rent to prostitutes quietly. Let's not speak about that eh?