The North Salem Free Library, now called Ruth Keeler Memorial Library, was started in 1932 in two rooms on the main floor of the Town House, with 1600 books contributed by the Universalist Church. In 1939, it moved to the North Salem Grade School (since burned down and replaced by the firehouse). In 1952, the library was granted an absolute charter by the State of New York. The adult book section was moved back to the Town House in 1957, there to remain until the current library building was built in 1980. A major expansion and renovation that just about doubled the library's size was completed in 2003. The project was funded with private donations, both by larger gifts and selling a "square foot" to people in the community.

The end of 2005 the library was renamed "Ruth Keeler Memorial Library" in recognition of the major contribution that the Ruth Keeler Charitable Trust, established by Miss Keeler before her death in 2003, has pledged to the library, by paying off the Library Building Fund's remaining indebtedness over the next five years.