The palace incorporates part of the ancient Albornoz fortress (1358) in turn built above the seat of the Oratory of the Raccomondati of Santa Maria della Misericordia. The Baglioni family lived there for few years, and from 1583 the palace was occupied by the apostolic Governor until 1860 when the city returned under the strict control of the Papal State.
Severely damaged by the earthquake of 1832 it was restored in the 1970s to house the State Middle School. Today almost nothing is left of the Baglioni’s ancient building: from via del Seminario Vecchio it’s possible to see stratified construction elements that show the different construction stages.
The interior has been heavily transformed for use as school building; nevertheless on the first floor, in the Governor’s hall, there is still  ceiling decoration made of painted tiles and a large border running along the walls with female figures and views over the Umbrian valley (16th century).