William was born at Melbury House in Dorchester at his mother’s childhood home, the home of his grandfather the 2nd Earl of Ilchester. Although his father William Davenport Talbot owned Lacock Abbey, the family did not live there and it was leased to the Countess of Shrewsbury.
William’s father died only six months after his only son and heir to the estate was born, and left his mother with debts. He spent much of his childhood at the home of his aunt and uncle, Thomas Mansel Talbot and his mother’s sister Mary Lucy (nee Fox-Strangways), at Penrice. He attended Harrow School and studied at Trinity College Cambridge.