Azkaban was a fine structure. That was what Harry thought the first time he entered the building. It was huge and could keep perhaps a thousand of prisoners. The prison was dark, gloomy and cold but, as Harry learned fast enough, the prisoners were warm and friendly ( well... you could say that... ) and when Harry was cold, all he had to do was ask(?) and big arms would envelop him into a warm and tight embrace.
Harry liked hugging people and being hugged in return. It was a fact that the prisoners learned almost immediately. Apparently it also was one of the reasons that sent Harry in prison: the small teen killed Snape when the old bat refused to be hugged. Oh... well, too bad for him. They wouldn't complain, they never really liked him anyway.
Harry always had a thing for dark things so it wasn't surprising to seet that he wasn't scared of Azkaban. It also shouldn't come as a surprise to see him hugging former Death Eaters or worse (really?): Dementors.
Before that, Barty wasn't aware of the fact that, yes, Dementors could blush.
So yeah, Harry liked Azkaban... and the guards... and the prisoners... and also - !
So, no, he won't follow the headmaster nor the minister out of his new home. No way in Hell.
See ? He even learned how to properly curse while being stuck here.