Have you read Homer though? Because for starters you say "several times" while we have only two female figures in the Odyssey and those are Circe and Calypso.
With Circe Hermes tells Odysseus that he has no choice but to bed her if he wants to save his men who were held hostage and with Calypso he clearly states that he was unwilling:
(I translate both passages in my answer there)
Now of course there is a passage about Calypso which I analyze here:
So no it is not "several times". If anything only Calypso is the person that seems to leave the element of doubt for many and even then the idea that he was willing is clearly debunked by Homer's definitive ουκ εθέλων εθελούση (unwilling [m.] by the side of the willing [f.]).
It is post-homeric scripts including Hesiod that mention the affair with Calypso starts as willing but Homer clearly states that Odysseus was unwilling by the side of Calypso (he even proceeds to write an amazing psychological passage when Odysseus doesn't let women touch him for quite a while for example refuses the help from Nausicaa's maids and wants to bathe himself instead) and Circe was a powerful goddess that was holding his men prisoner at first and then Odysseus claims how much he fears her. Not only does he pray to Gods and begs her to let them go but also passes the night by the ships after they return from the underworld till Circe herself summons him)
It is perfectly fine if you want to follow the idea that Odysseus was unfaithful if you read other sources but interestingly enough Homer mentions how unwilling he was. What is more both of the women that he associated with are powerful immortal goddesses. It is no different than female characters being conquered by Gods in mythology. I am sure you would not blame Leda for "seducing Zeus" right?
And again no matter what even if you assume the affair with Calypso started willingly he was still raped by her. Either it started and he tried to end it and Calypso didn't take no for an answer or he never wanted that and then it is even worse.
