The house at 1 Brown St., near the entrance to Oak Hill Cemetery, has a deep and unique history in Newbury and Newburyport.