The Locust Valley Library was founded by Frank N. Doubleday, Frederick A. Horsey, Edward Latting, James T. Mitchell, Jr., Dr. Edward Seaman, Charles F. Williams, and Edward M. Ward on October 17, 1909, with a $300 contribution from the year-old Matinecock Neighborhood Association. Its first home was a two-room cottage at One Bayville Road, near the corner of Bayville and Buckram Roads. The first Chairman of the Locust Valley Library Association was Neltje de Graff Doubleday (Mrs. Frank N.), who wrote nature and gardening books under the pen name "Neltje Blanchan." The first librarian was Miss Bertha Whiting.
The building that now houses the library was built in 1914 as the Matinecock Neighborhood House through gifts of money from local residents and donated services of carpenters, plumbers, masons, and others. There was a kitchen and bowling alley in the basement. The basement also housed the Locust Valley Fire Department. With the help of Frank N. Doubleday a library wing was added 1923, and in 1936 the Matinecock Neighborhood Association turned over the building and grounds to the Locust Valley Library. Throughout the years the Locust Valley Library has continued to serve as a community center.
In the early 1980's the building was renovated to improve library services. In the late 1990's the building was again renovated to add an adult computer center and to move the Children’s Library to a larger area on the ground floor.