The Castle was built by or for Bishop Odo of Bayeux, half brother of William the Conqueror, as the headquarters of his extensive estates in Oxfordshire, and appears to have been a place of considerable strength in the twelfth century. William de Chesney, Lord of Deddington, held the Castle in the mid twelfth century when England was ravaged by civil war, but during the struggle between King Richard and his brother Prince John in the late twelfth century it was seized by the Crown. Descendants of the Chesneys, the Dive family, regained possession in the thirteenth century and styled themselves 'Lord of Deddington Castle', but by the end of that century it was partially demolished and by 1310 there seems to have been little left apart from "a chamber and a dovecote". By 1377 stonework from the walls was being sold off. To the south east of the Castle site is a field known as 'The Fishers' which was presumably originally the fishpond for the castle.