G-Ram...the thing is, the people in his circle knew about the infidelity at the time it was going on. His wife knew. But in order not to tarnish his public image, it was kept very quiet and not leaked in public other than the tacky Cointel-Pro stuff that leaked it to his wife.
I heard about his infidelities when I was about in the third grade. That was about 20 or so years after his death, so it's not allegations made in the present. It's just like Nelson Mandela was alleged to have cheated on his first wife with Winnie and was known to be a notorious flirt after his release from prison but because of all he did it was swept under the rug.