In 1651, John Stephens of Lypiatt Park bought a house and 51 acres at Rodborough from the Sheppard family of Minchinhampton. This formed the basis of the later Rodborough Manor estate.
At first the house was called Hill Living, then Hill House, but from the late 19th century the present name was used. During the 18th century, the estate changed hands a number of times before being bought in 1757, by Onesiphorous Paul, a successful Woodchester clothier. He built a villa there shortly after acquiring the property. He was knighted in 1760 and made a baronet two years' later. When he died in 1774, both the title and the property passed to his son, Sir George Onesiphorous Paul.
Sir George Paul was High Sheriff in 1780 and went on to pursue a distinguished career as Chairman of Quarter Sessions and as a prison reformer of national significance. In 1788, King George III was entertained at the Manor, during his visit to Cheltenham. Sir George Paul showed him his clothing mill at Woodchester and the Stroudwater Canal. In 1784-92, Paul made major alterations to the house and grounds to the designs of local architect, Anthony Keck: these works cost over £7,000.
After the death of Sir John Dean Paul, the house was sold in 1854 with 141 acres. Between 1855 and 1870, it was the Stroud home of Lord John Russell, who had been MP for Stroud in 1835-41 and later Prime Minister. In 1861, he was created 1st Earl Russell, and took as one of his subsidiary titles, Viscount Amberley.
Rodborough Manor changed hands several times before being bought by Sir William Marling in 1889, and an Italianate lodge was built on the Bath Road drive to the house in 1885, to the designs of William Clissold. In 1906, however, the house was gutted by fire. It stood roofless and decaying until about 1923, when E. Lee Godfrey, who bought it in 1922, demolished much of the shell and rebuilt in on a much smaller scale. The rebuilt house was divided into two dwellings.