John Ivory (later John Ivory Talbot) was the eldest son of Anne Talbot and Sir John Ivory of Wexford. Because his grandfather Sir John Talbot did not have any surviving sons from either of his marriages, John inherited his estates through his mother. He assumed the name Talbot as a condition of his inheritance.
He married Mary Mansel, daughter of Sir Thomas Mansel of Margam, in 1716. They had four children: John Talbot the younger, who inherited Lacock; Thomas, who inherited Margam and Penrice through the will of his uncle Christopher Mansel; Martha, who married William Davenport, her cousin (the son of her aunt Barbara Ivory, who married Henry Davenport); and Ann, who died aged 29.