Duke spent his early childhood at Duke Farms, her father's 3000 acre estate in Hillsborough Township, New Jersey.
She was presented to society as a debutante in 1930, 18 years, at a ball at Rough Point, the family's residence in Newport, Rhode Island. Duke 's interest in horticulture led to a friendship with Pulitzer Prize winning author and renowned scientific farmer Louis Bromfield operating Malabar Farm, his country home in Lucas, Ohio in Richland County.
Duke had learned to play piano at a young age and developed a lifelong understanding of jazz and jazz musicians friends. She also loved gospel music and sang in a gospel choir.
