That's precisely the reason why one should aim to retire as early in life as possible. Being financially independent is a great stress reliever. You should try it if you have not already done so.
Once financial worries are removed from the equation, you can then pursue activities that you enjoy rather than being forced into doing tasks just to earn a living.