Double Down is the father of Ricca and Tempo.
He was forced to work for Tesoro to pay off his debt. He was regularly humiliated there.
He was freed thanks to the actions of Luffy and his crew.