I love to say I don't give a fuck, two shits, or three rats' assess, but that's usually about not caring what others think of me. But yeah, it applies as you write here. What you describe is what Frankl wrote about the ultimate freedom.

“Everything can be taken from a man but one thing: the last of human freedoms — to choose one’s attitude in any given set of circumstances, to choose one’s own way…between stimulus and response there is a space. In that space is our power to choose our response. In our response lies our growth and our freedom.”