These buildings are the product of over a century of adapting to the needs of the Swift and Stewart families who lived here from 1815 until 1981.
The original portion of the Main House was built by Samuel Swift (1782-1875) in 1814 when he was at the beginning of a long career of public service. Shortly after being admitted to the bar in 1808, he started a bookstore, printing business, 
and eventually a newspaper, which was established in 1812. The focus of the paper, in part, was to communicate war news, since Middlebury was so close to the enemy's country and to the expected seat of war.
After the war, Samuel Swift settled down to practice law. He first served as Judge of Probate, then of the County Court, and eventually served in the state legislature. He returned to literary pursuits later in life when, at the request of the Middlebury Historical Society, he wrote histories of Addison County and the Town of Middlebury - still major sources for the study of area history.
Shortly after Judge Swift's death in 1875, the house was bought by Governor John W. Stewart (1825-1915). Stewart's political career included eight years in Congress and one term as Governor of Vermont. Stewart was proudest of his role in the 1860 Republican National Convention when he was influential in turning the Vermont delegation from William H. Seward to Abraham Lincoln.
The house served as home base for the Stewart family, which consisted of Governor Stewart, his wife Emma Battell Stewart, and their children: Philip Battell, Elga, and twins, Jessica and Robert Forsyth. They lived in the house for only part of the year, using it as a base from which they traveled, attended school, or sat in Congress. Summers were spent in Middlebury, with side trips to Lake Champlain and to Ripton where Mrs. Stewart's brother, Joseph Battell, ran the Bread Loaf Inn. This is now the summer campus of the Middlebury College School of English and home of the Bread Loaf Writers' Conference.
Mrs. Stewart, who spent winters in Boston or New York, had one of the first telephones in Middlebury installed to connect her home with the Congregational Church so that she could hear the services she was unable to attend. She also kept the church supplied with fresh flowers from her gardens and greenhouse. In 1885, the Stewarts hired Clinton Smith, Middlebury's most prolific Victorian architect, to design and build the horse barn and a carriage house located on the northeast corner of the property. Smith's other work is evident throughout Middlebury in the County Courthouse, the Methodist Church, the downtown Beckwith Block, and numerous other commercial, public, and private buildings. In 1906, Governor Stewart removed the original summer kitchen wing and replaced it with a two-story addition. Stewart's architect, Harding & Seaver of Pittsfield, MA, designed the new wing so that it would appear to have been built at the time the house was originally constructed. At the time of this remodeling, doors and porches were added to both the east and west facades of the original house, as well as a large arched window on the east side.
In 1915, on the death of Governor Stewart, the house passed to his daughter, Jessica (1871- 1981). After a romantic and brief first marriage to J. Walter Sylvester, a minister who died of tuberculosis within eight months of the wedding, she married an older man, Charles M. Swift, grandson of the original owner of the house. Still another lawyer, he had made his legal reputation in his younger years in a complicated property case that he spent 40 hours summing up for the jury. He later gave up practicing law and went into the business world where his major accomplishment was building a streetcar system for the Philippines. Mr. and Mrs. Swift traveled around the world, cruised on his yacht in Florida, maintained a New York apartment, and built a summer home complete with a golf course on Lake Champlain in Ferrisburgh. They used the Middlebury house occasionally and made it available to friends and relatives.
In 1939, ten years after the death of her husband, Mrs. Swift gave up her New York apartment and made her Middlebury house her primary residence. She continued her family's tradition of service with her donations to local institutions, which included building a library wing for the Sheldon Museum that contains her family papers. She also donated her early childhood home, built by her great-grandfather and now known as the Community House, to the people of Middlebury. In 1943, she also donated her home to be used as a restaurant for charitable purposes by Sir Wilfred and Lady Anne Grenfell. After Mrs. Swift's passing in 1981, her home in Middlebury and its contents were eventually auctioned off and the property was turned into a New England inn. Inn Keepers are: Dan and Michele Brown.