The original colonial building on the site of Istana Bogor was a mansion named Buitenzorg, meaning: Without a care in Dutch (also Sans Souci, meaning: Without a care in French), which dates back to 1745 as a country retreat for the Dutch Governors to escape the heat and diseases of Batavia. The location for the new palace was discovered by Baron van Imhoff on 10 August 1744, in a village named "Kampong Baroe". On the site he ordered a mansion to be built, however the construction wasn't completed by the end of his career in 1750 and thus it was continued by his preceder Jacob Mossel.

In 1811 Stamford Raffles led a successful Invasion to Java and Britain took possession of the colony until 1816. Raffles had Buitenzorg Palace as his residence due to the temperate climate of the hills of Bogor, while conducting most of the council meeting in the palace of Rijswijk in Batavia. During his stay he had the garden of the Buitenzorg palace transformed into an English garden, which became an inspiration for the later botanical garden.