King Christian IV bought Skinnerup, an estate by the innermost shore of Kertinge Bay, in 1616. In 1632, the king decided to build impressive new buildings on the site, which would become the residence of his son-out-of-wedlock, Ulrik Christian Gyldeløve, who was 2 years old at the time. The name of the estate was changed to Ulriksholm.
After the death of Christian IV, his mistress, Vibeke Kruse, fell out of favor. When she died, she was buried in a pauper's grave outside the city walls of Copenhagen. However, her son, Ulrik Christian Gyldenløve, saw to it that her remains were moved to Kølstrup Church. He died himself at the early age of 28 after fighting bravely in the Swedish War (1658-60).
The castle is today privately owned and undergoing remodeling. There is no public access to the castle or garden