The Library of the Academy is tightly connected with the history of the Academy in general. It was established by the Constitution of the Society of Serbian Letters, November 7, 1841. It started working in 1842 when Dimitrije Tirol, the tutor of the Prince Obrenovic's sons and the benefactor of the Serbian people, bestowed 89 of his books to the Society Library. The initial library holdings were enlarged by the legacies and gifts and by the Society publications, later by the exchange with home and foreign academies and scientific societies, and from 1851 by the purchase of new books.

The name of the library has during the Academy's history changed several times: in 1864 it changed its name into the Library of the Serbian Learned Society, in 1886 it became the Library of the Royal Serbian Academy, in 1945 the Library of the Serbian Academy of Sciences, in 1951 Central Library of the Serbian Academy of Sciences, in 1960 Library of the Serbian Academy of Sciences and Arts.

The Library was partially opened to the public in 1952.

The Library directors were: Konstantin Bogdanović, Stojan Novaković, Smilja Mišić, Danica Nikolić, Đuza Radović, Mile Žegarac.