In 1903, the Rowayton Library was organized by a group of public-spirited citizens and housed in a room in Craw's Hall at 101 Rowayton Avenue. It opened with capital of $154, a loan of 100 books from the State for three months and 25 books from the Bodley Book Club to be exchanged monthly.
In 1926, having outgrown the first home of the library, the Board held a special meeting and approved the purchase of 145 Rowayton Avenue, former firehouse of the Reliance Hook and Ladder Company (and now the Rowayton Arts Center).
The Library paid owner, Frank Machette, $5700 for the property and the building. Upon moving in, the first floor was used to house the collection and the second floor was used as a community meeting place, known as the Library Hall.
In 1966, the present building and surrounding six acres at 33 Highland Avenue were purchased by the Sixth Taxing District to become the Rowayton Library and Community Center. Originally this Tudor-style building, constructed in 1912, was a part of the extensive estate of James A. Farrell.
It was later purchased by the Sperry Rand Corporation which developed the first commercial computer on this site. In 1966, the Library entered into an agreement with the Sixth Taxing District whereby the Library gave the Sixth Taxing District the property and building at 145 Rowayton Avenue and would continue to perform the services of a "public library" for the community of the Sixth Taxing District in exchange for their new home at 33 Highland Avenue for the next 99 years and a partial funding grant every year.