Hey, that's not stupid at all, it's very important to understand when you're in a situation when you don't feel comfortable to do something you 100% have the right to say no, and you shouldn't let anyone bully you or push you into doing something wrong.
At the same time, if you are quiet and don't even mention that you don't want it to happen and just sit quietly and let him, he will take that as meaning you're ok with it (he's not a mind reader).
I guess whether the situation you're describing is assault depends on what you mean by "forced his way". When you say force, that sounds like you tried to stop him but he kept going. That would be illegal.
But if he tried his luck and you went along with it, then it is difficult to prove or say he is doing anything wrong.... Try to learn from this and not make the same mistake, otherwise it could really put you in a bad situation. trust me on this, one of my friends had a really bad incident happen to her while she was drunk and taken some stuff where a guy forced himself all the way