The mansion was originally built between the years 1824 and 1828 from the plans of famed architect Robert Mills, whose other works include the U.S. Treasury building, the Washington Monument and parts of the White House. Distinguished visitors include U.S. Presidents Andrew Jackson, Ulysses S. Grant and Andrew Johnson. Local lore has it that some of the original garden was planted by Varina Davis while her husband Jefferson was attending the affairs of government in the mansion library. The mansion served as Grant's headquarters and as a hospitality lodge and hospital for the troops during the planning and the siege at Vicksburg. It also served as the first Freedman's Bureau to educate the newly freed slaves.