The grandson of Katherine (de Wingfield), Countess of Suffolk, William de la Pole, 1st Duke of Suffolk, who in the early 15th century married Alice Chaucer, granddaughter of the great English poet, Goeffrey Chaucer. During this time they lived at Wingfield Castle, Charles, Duke of Orleans, prince and poet, and father of Louis XII of France, was captured by the English at the Battle of Agincourt in France in 1415, and was held prisoner for a time at Wingfield Castle — while his family raised the enormous ransom demanded. He was held prisoner at Wingfield Castle from March 1433 to June 1435 and October 1435 to May 1436, during which time he supposedly had a love affair with the mysterious Anne Molins.

In 1504 the castle came to Charles Brandon, Duke of Suffolk, whose grandmother was Elizabeth Wingfield, daughter of Sir John Wingfield of Letheringham and Elizabeth FitzLewis.

The Duke of Suffolk and his wife, Mary Tudor, Duchess of Suffolk, would have spent some time at Wingfield Castle when they were not at their main residence at Westhorpe Hall in Suffolk. The Duchess of Suffolk wrote at least one letter that we know of from Wingfield Castle.

Mary Tudor, when French Queen Dowager (former wife of Louis XII of France), was Godmother to Thomas Maria Wingfield. He was later Member of Parliament for Huntingdon and father of President (the Council at Jamestown in 1607) Captain Edward Maria Wingfield.