I like him, but don’t want to complicate things

I [19F] have a sex buddy[20M] and sometimes we both will ignore each other during the day and only speak at night/ demon time. I sometimes look at his messages and don’t respond for a day or two because I don’t want to seem clingy or fuck anything up and he’ll send another message and I’ll finally respond and then he would leave me on open for a long time before finally answering like a day or two. Truth be told I wish I could talk to him more than just sex, I kinda despise myself because I have been thinking about him constantly. I knew I liked him when he didn’t respond to me and I actually got sad/mad about it. For all I know is that he could be around other girls since he is a frat guy and I’m away for the summer. I shouldn’t feel like he has to not be with other girls cause we’re young and not dating, but I kind of wish I was the only one.