Thomas Howard's extravagant additions to the house led to his downfall and left his successors seriously indebted. The 3rd Earl's position was eased in 1667 when Charles II bought the house and park at Audley End for £50,000. Of this sum £20,000 remained on mortgage, and the Suffolk family were given accommodation in the northern pavilion of the outer court, as keepers of the new palace.
Audley End's particular attraction to Charles II, a horse-racing enthusiast, lay in its proximity to Newmarket races. After about 1670, however, neither Charles nor his successors made much use of it. It was by then both old-fashioned and fast deteriorating, despite repairs by the Office of Works. The house was returned to trustees on behalf of the heirs of the 3rd Earl in 1701, in settlement of the mortgage.
