George Washington Richardson acquired the 510-acre “Byrd tract and mill.” Some time between 1808 and 1815 the mill ceased production and in 1820 the first part of The Oaks, what is now the kitchen, was constructed. George later built the two-over-two front addition onto his the building in 1830.

George continued the family tradition of public service. He was a county justice for several years beginning in 1805, and then again from 1819 until 1845. He was also the first postmaster in the village of Wilmington in 1813. He had two daughters (Betsy and Mary, who each married into the Bowles family, prominent neighbors on adjoining farms) and a son, Samuel. George died in 1849 and the Byrd tract was willed to his three children.