Moral considerations regarding FIRE

An issue I always have with early retirement is whether it is morally acceptable. When retiring early from a skilled profession you are depriving society of a big contribution you could have given, that was also expected and invested in by society. Utilising a power dynamic by having more money and knowledge to capitalise on other people exacerbates this issue.

How are you dealing with this? Are you of the mindset that you do not owe anything to society? That it is completely fair, as you earned that money and there is a perfect market that trades all aspects in a meritocratic fashion (e.g., delayed consumption should be gratified this hard)? Or that you were not just lucky to have the talents to earn so much money?