Bikur Holim first opened in a residential building in the Old City in 1826. In 1843, the hospital had only three rooms for patients. In 1854, a building was purchased which soon grew overcrowded. In 1864, another complex of buildings was acquired incorporating treatment rooms, a pharmacy, a hospice for the terminally ill and administrative offices. The Ashkenazi Perushim Hospital, as it was known, became the favorite charity of the British Jewish philanthropist Moses Montefiore, who described the facility in his diary in 1875. The general ward consisted of two rooms, each with eight beds. One room was reserved for men, and the other for women. In 1893, the hospital cared for 781 patients and treated 12,347 people in its out-patient clinics.
By 1907, hospitalizations exceeded 1,000 per annum. A decision was reached to build a new hospital outside the walls of the Old City. The cornerstone of the new building was laid in 1912, but construction work was delayed by the outbreak of World War I. The building on Chancellor Avenue (now Strauss Street), just off Jaffa Road, was completed in 1925 and opened its doors to all residents of Jerusalem, Jews and non-Jews. The hospital in the Old City continued to treat the chronically ill until 1947. 
Many of the wounded from the Arab riots of 1929 and 1936 were brought to Bikur Holim. Jewish underground fighters were hospitalized under fictitious names to keep the British mandatory police from finding them. During the War of Independence in 1948, the hospital came under artillery fire from Jordanian guns. Hadassah Hospital on Mount Scopus was evacuated, and many patients were transferred to Bikur Holim. 
Situated near the religious neighborhoods of Geula and Mea Shearim, Bikur Holim admits a very high percentage of Haredi Jews, and tries to cater to their needs. Shabbat is strictly observed. Non-Jewish employees record medical information and answer telephones on the Sabbath. Food is warmed in ovens operated by a timer, in keeping with Orthodox religious rulings.