He didn't didn't feel comfortable and the reason could be anything.

He might not feel confident to know that you're looking at his penis. He might not like seeing you going down on him for some reason.

What's for sure is that he was thinking about something that distracted him.