Sir John, a distinguished professional soldier, inherited Audley End in 1762 from his Aunt Lady Portsmouth, having fulfilled her condition that he change his surname to Griffin. Griffin employed the architect Robert Adam to add a stack of galleries behind the great hall to connect the two sides of the house, and to build new kitchen offices to the north. Adam’s principal creative contribution, however, was a new set of ground-floor reception rooms on the south front.
At the same time ‘Capability’ Brown began remodelling the grounds, sweeping away the remains of the formal landscape, with Adam providing designs for the garden buildings, including the Temple of Victory and Palladian (Tea House) bridge. After a quarrel with Sir John, Brown was replaced by Joseph Hicks. Sir John took an active role in devising and directing the works, employing some of the best designers and craftsmen of the day.
In 1784 George III (r.1760–1820) recognised Sir John’s claim to the Barony of Howard de Walden. This event prompted Sir John to create a new state apartment at first-floor level on the south front, opening off the saloon. The east ends of the north and south wings, reduced to a single storey by Lady Portsmouth, were built back up to their original height. The remodelling continued, as well as improvements and extensions to the park until Lord Howard died in 1797. In the 1780s the Elysian Garden was created on the banks of the Cam, and the Temple of Concord was designed in 1790.