Chapter 173 of the manga is the first, and only, time where both of Yusuke's parents are seen. An anonymous guy is talking to Atsuko a few meters away from Yusuke's cart. It evident from their conversation that he is Yusuke's father. It can be seen that he is a smoker. He offers to give Atsuko money but she declines. It is implied that he sees Atsuko on occasion. He mentions that he used to spank Yusuke as a form of discipline. It shows that he and Atsuko broke up due to their inability to stop fighting.
It is stated in the manga that he and Atsuko split up. He still obviously has feelings for Atsuko and tries to get back together with her (his inability to actually keep a relationship hinting that he never got over her) but she refuses, though she offers to come back with him for the night. It is also implied that she had been sleeping with him on occasion since they broke up but never stayed the whole night. Despite being an absent figure in his son's life, he seems to speak fondly of Yusuke, surprised at Atsuko's claims that he's gotten strong since they last saw each other. This shows that despite being absent for most of his life, he still cares about Yusuke.