The house takes its name from the Field family, who were long resident in Paganhill. Thomas Field, who died in 1510, probably lived here. From him, it descended through Richard Field (d.1693), Edward Field, JP (d.1786), to John de la Field Phelps of Dursley. He either sold or leased it to James Tyers, but it was later reacquired by the Phelps family, who finally sold it in 1840, to Charles Stanton, a successful clothier.
The present house dates largely from c.1800, when James Tyers built the E-shaped south front with projecting wings in a rich and very pretty Gothic style. The use of quatrefoil windows above the central porch and the extensive use of crenellation suggest its relatively early date. The plan of the house probably derives from its early 17th century predecessor, of which some fragments survive at the rear. This was a large house, taxed on nine hearths in 1672.
A service wing was added to the house on the south-west in the 19th century, and Field Place was divided into three houses in 1957.