My husband and I braved the frigid cold wind and went to the top. It was super cool and we are glad we did not skip it due to the weather. **Apple Maps did not take us to the correct region of it, so listen to your gut feeling when you see the stairs, park there.
The gardens are more like forest, there aren't too many flowers actually (maybe there's more in spring?) but the place is beautiful nonetheless. The are is vast, covering a big land behind the fortress.