You exactly stated it: you are trying to justify what is wrong. Masturbation may seem like an acceptable or even necessary outlet for your sex drive, but it is not.
The Church calls it a disordered drive because God wants us to use our sexuality not simply for procreation but as a covenantal bind between spouses. It is a unitive matter.
As you age the desire may decrease, as it has in your wife, but you still need to direct it to your spouse. You need to get her back in the game if you want to have sexual activity. You may have to change some of the dynamics in your relationship to make that happen, for women do not warm up the same way men do. Maybe that is the challenge God wants to you work on because your marriage needs it.
Go to Confession and stop rationalizing. You know the right road, just walk it.