Following many decades of various ownership, the building was threatened from urban renewal activity in the early 1960s. Urban renewal Project R-51 resulted in the disturbance and demolition of buildings in historically Black neighborhoods just north of downtown Athens. The Hoyt House was abandoned at this time, and the structure endured years of neglect resulting in vandalism and a damaging fire. Although negatively impacted by urban renewal work, the house was not demolished.

By 1973, a group of individuals from Athens, Atlanta, and Rome, Georgia, had finalized plans to develop the Athens History Village on land located just west of the present-day intersection of E Dougherty Street, North Avenue, and N Thomas Street. The project scope also included the relocation and rehabilitation of the Hoyt House at the new site. The group relocated the first story only of the house on another parcel of land prior to its move to the center of the Athens History Village in 1974.

Most recently, the Hoyt House has been associated with the Hotel Abacus complex that owned the site until early 2026 when it was announced that a student housing development will replace it. As of March 2026, the Hoyt House has been systematically deconstructed and is being stored in safe and secure location. Verity Works has completed a thorough Historic Structure Report and Relocation Plan for the Hoyt House building. We are looking forward to its future reconstruction at 794 W. Hancock Avenue in Athens.