Dragon may be an absent father, but he does care for Luffy, and that shows in his behavior. Dragon is simply torn between duty and family.

Sabo wanted to become a pirate like his brothers but was shot down early on by a Celestial Dragon. If Sabo had set sail as a child, he might not have survived long in the dangerous East Blue.