The origins of the palace date back to the sixteenth century. Brick castle in Gothic-Renaissance style was built by the royal architect Job Bretfusa, known for the construction of the Lower Castle in Vilnius, the royal court in Knyszyn, and the modernization and expansion of the Old Castle in Kamieniec Podolski or the castle in Tykocin.

Stefan Mikołaj Branicki, the father of Crown Hetman Jan Klemens Branicki, commissioned to transform the castle into a Baroque mansion to Tylman von Gameren. That engineer before he became famous for, among others, construction of Krasiński Palace in Warsaw and Primate Michael Radziejowski residence in Nieborów. The ongoing reconstruction in the years 1691-1697 completely changed the building. Among other things, one of the towers was used as a staircase. In the second tower located chamber. During another reconstruction increased lateral outbuildings, erected Ionic colonnade, numerous sculptures. There is also a legend about an underground passage between the castle and the tower standing near the parish church, which in the seventeenth century was also adapted for defense.

The palace was destroyed in 1944 and rebuilt in the years 1946-1960. Since the end of World War II, it houses the Academy of Medicine, now Medical University of Białystok.