Although the land in the Fairfield and Yeerongpilly areas was low-lying and flood prone, the soil proved suitable for agriculture. By the late nineteenth century the Grimes brothers ran three farms in the region and were reported to own most of Fairfield. Early crops included sugar, cotton and vegetables, but the climate proved unsuitable for sugar and the end of the American Civil War reduced the demand for cotton. The Grimes brothers instead shifted their focus to arrowroot, purchasing a former sugar mill in what is now Tennyson and establishing the ‘Coongoon’ plantation nearby. The brothers experienced some difficulties, including the destruction of the uninsured ‘Coongoon’ by fire in 1876, and were embroiled in the debates over the use of kanaka labour. In 1885 'Coongoon' was relocated to a site on the Coomera River, although the produce was transported to Tennyson and packaged for shipping at the original plantation and mill.
The Grimes brothers also played active roles in the political and public life of Fairfield and Queensland. Samuel Grimes served as Oxley’s Member of the Legislative Assembly (MLA) for 24 years, while George represented Bulimba for one session in Parliament from 18 November 1876 to November 1878 and contested Albert in 1888 and Logan in 1896. George sat on a variety of local committees and was a member of the Stephens Shire Council. He was a founder of the East Moreton Farmers’ Association, which became the National Association, and a member of the Queensland committee of the 1888 International Exhibition in Melbourne. Additionally, the Grimes family contributed significantly to the growth of the Baptist Church in Queensland. They donated land in 1862 for the creation of the first Baptist church in the colony, and George served as President of the Baptist Association of Queensland as well as a deacon of the Vulture Street Baptist Church. ‘Thorough and practical in everything he took in hand,’ a later commenter remarked, Grimes was a man of ‘sterling character’, whose counsel and advice were much sought after.
Surprisingly, when George Grimes had a new residence constructed, it was not built on the extensive Grimes landholdings. Instead, ‘Ashby House’ was planned for a twelve-acre site adjacent to one of the Grimes’ farms, purchased by the brothers in 1883 and transferred to George alone in 1888. The site had been part of an 80-acre holding, known as portions 3 and 4 of Yeerongpilly, which was granted to John Williams in 1854. Williams had quickly subdivided the site into small farming allotments and sold them. Farmer John Gaynor purchased a number of subdivisions in 1865, and his wife Matilda obtained additional land in 1875, culminating in the twelve-acre site. The Gaynors had adapted their Fairfield land for farming in the 1860s but did not live there, instead residing at Yeronga. This land use reflected the nineteenth century development of Fairfield. Despite the opening of the railway line, which tended to bring new residents to suburban regions, Fairfield remained a largely agricultural, lightly populated district. However, for those who were untroubled by Fairfield’s rural nature, the Gaynor’s elevated site was ideal for a residence.
The Grimes families, being among the earliest residents of Fairfield and the Yeerongpilly and Stephens parishes, were not averse to Fairfield’s isolation. A number of residences had been built for the Grimes brothers and their families in the mid to late nineteenth century, including 'Brighton Cottage' at Coongoon, 'Fairfield' at Fairfield, 'Kadumba' at Yeronga and 'Tarragindi' at Sandy Creek, renamed Tarragindi for the house. From the mid-1870s George was listed as residing in 'Ashby', Fairfield, but this appears to be an earlier residence on a different site. The Grimes’ later buildings, including the firm’s large drapery store on Queen and Albert Streets built in 1881 and 'Kadumba' constructed in 1886, were designed by eminent architect Richard Gailey. Of these nineteenth century buildings associated with the Grimes family, only George Grimes’ ‘Ashby House’ is extant.
An exact date of construction and designer have not been identified for 'Ashby House'. Given the Grimes family's long association with Richard Gailey, it is likely that he was engaged to design Ashby House. The engagement of a prolific and highly-regarded architect such as Gailey would also reflect the financial success and social standing of Grimes. The residence was probably constructed by 1890, and was certainly standing by 1892, when George Grimes was listed as living in the so-named house on Brougham Street, Fairfield. In 1892 Grimes took out a mortgage of £1,300 with Richard Gailey and William Henry Ewing, further supporting the possibility that the house was a Gailey design.
George and wife Mary lived in the house until their respective deaths in 1910 and 1919, after which title to the property passed to their unmarried daughters. Grimes was well-remembered after his death, with an obituary in the Brisbane Courier noting that Grimes and his wife ‘were highly respected and esteemed in the Fairfield district, where the deceased spent the 40 years of his wedded life and many acts of kindness performed by them will be long remembered.’
The Grimes daughters owned Ashby House until 1924, by which time Fairfield was beginning to experience limited population growth. Fairfield was inundated in the floods of the 1890s, which had curbed development and forced the relocation of the Fairfield railway station. Much of the land remained tied up in estates belonging to the old farming families like the Grimes. This combination of factors had channelled settlement away from Fairfield, but increasing demand for housing in the interwar period and the breaking up of the old estates sparked a shift towards urbanisation. Ashby House held particular appeal due to its elevated position and views over the Brisbane River. Sales advertisements drew attention to the road access on three sides of the ‘charming residential property’, its five bedrooms, wide verandahs and ‘every convenience’, as well as its proximity to public transport.
The house was purchased by John and Ellen Richards, manager of the Finney Isles furniture department and his wife, who resided in the house with their large family for the next two decades. Ashby House was again sold after John's 1942 death, and the house was rumoured to have been the residence of one of General McArthur's staff during WWII. Further sales over the ensuing years saw the additional land subdivided and the residence site whittled down to one rood sixteen perches and one tenth of a perch.
In the postwar period industrial development, including the opening of the Tennyson Powerstation on 26 March 1955, brought major changes to the suburbs of Fairfield, Yeerongpilly and Tennyson. These developments led to an increased demand for worker accommodation within the local area. Accordingly, Ashby House was converted into flats in the 1960s, a reconfiguration often undertaken in substantial residences no longer needed to house large families. After a number of owners the house was substantially renovated in 2005. This included an extension to the rear to improve the street frontage of the house whose front elevation had been obscured by subdivision and infill development from the 1950s.