Only say "I Love You" if you feel it in your ❤ and mind. He wants the truth as you would want the truth.
I don't have a close relationship with my father. He is here for me now when i need it. I see him maybe 3 times a year for less than a few hours. Is it wrong for me to not him "I Love You" ? Because I don't feel like I do or not.
Children depend on their parents for just about everything.
When we don't get the things we need from our parents, we have to protect ourselves from being constantly hurt by them in the sense of omission rather than commission.
Sometimes, saying "I love you" is just too hard---and that's perfectly OK. You don't have to say it. I suspect in some way they consider your feelings justified.
Only say it if you're comfortable doing so. I'm sure you love your Dad in your own way and are probably kind of wary to saying it after your history with him. Just enjoy the times you have together and maybe you'll both find a way to be together more often. Saying the words isn't always as powerful as showing you love someone .. So don't worry so much about words.