The first records of Castello di Poppi date back to 1191, when it was owned by the largest feudal family in Casentino, the Conti Guidi, however, it is thought that the foundation may be several hundred years older.

It is thought that the architects Arnoldo di Cambio was responsible for the design and build of the castle. He would later design the Palazzo Vecchio in Florence, and the Castle of Poppi can be seen as its prototype. Restoration work took place in the 19th-century when the original square tower was converted into a bell tower following a lightning strike. A bronze bust of the poet Dante Alighieri stands outside the castle as a historic reminder of the time he was hosted at the castle by the Conti Guidi during his exile from Florence. It is thought that Dante composed the XXXIII canto of his Inferno during the stay.

With the outbreak of the Second World War in 1940, the art collections of Italy were transferred out of the cities for safekeeping. The sculpture collection of the Bargello Museum in Florence was evacuated to the Castle of Poppi on December 29, 1942, alongside the painting collections of several of the most important galleries in Florence, including the Uffizi. By mid-1944 the Allies were advancing north through Italy and would soon reach Florence. On the night of August 18, 1944, a German officer arrived at the castle with the apparent task of checking for concealed weapons and ammunition. Four days later a German captain, second lieutenant, and a non-commissioned officer arrived and insisted on inspecting every room again with the excuse that “the village was a nest of spies and rebels.” If a key was not produced quickly enough, they broke down doors.

Only a few hours later, at midnight, more soldiers returned, informing the local townspeople to stay in the cellars of their homes as they would be soon detonating mines placed under the town's medieval gate. With no witnesses to their looting, the German 305th Infantry Division set to taking crates of art with them. Shots were fired into the air to keep the locals away, and by six in the morning, the trucks were gone. An inspection of the Castle of Poppi after the raid by the Commissario Prefettizio found broken walls, doors, and debris all around. At 2 pm that afternoon, the Germans stayed true to their word, detonating the mines around the town. The medieval gate, some local homes, and the only road into Poppi were all destroyed. Monuments Man Frederick Hartt reached the castle on September 25, after three weeks of trying to reach the area. Two days later, a detailed inspection revealed some masterpieces remaining and some that were conspicuously absent. One hundred and ninety-eight paintings had been taken, including by artists Rembrandt, Ingres, Lippi, Raphael, and Rubens.