What could be resounding confirmation? Octavian bearing Caesar's child??
That's the point. It's a rumor. It may be true, it may not, there is little for us to use as confirmation or denial. It may be a result of the political machinations of Antony or it may be truthful reporting. What I mean is that Suetonius hardly provides any detail other than the term "unnatural relations" (he actually uses the word stupro which is indicative of sexual intercourse in case anyone is not completely sure what is implied by unnatural relations), but there is no confirmation other than Antony's accusations. Obviously we are quite familiar with the enmity between Antony and Octavian.
While the "Queen of Bithynia" story was widely known in contemporary circles, this particular story seems to have attracted much less attention. Surely Augustan propaganda could have helped nix any lasting record, but attempts to mute such things can often have a complete reverse effect. It seems probable, considering all the rumors, that the young Octavian may have been involved in some scandalous liaisons, but the who, what, why, etc is rather difficult to pinpoint.
The Calpurnia argument isn't really applicable. It would not be scandalous for a man to sleep with his wife, so why would Suetonius go out of his way to suggest it. Perhaps had he said that Caesar did not or would not sleep with his wife, I'd be inclined to agree with the comparison.