Thanks for the question.
It can be a sin without sufficient cause to withhold it. A spouse has a responsibility to nurture the sacramental nature of the relationship, which is expressed in the conjugal act. This is the only act that is proper to marriage alone. In the absence of this intimacy, and openness to life ultimately, the relationship suffers, and unremedied, will eventually wither on the vine (unless advanced years have obviated the act)
That being said there are circumstances that put obstacles in the way: time, emotional hurts, distance, and in the worst case, abuses or other problems. That is for a different discussion. But assuming there is no abuse, one has a obligation to eliminate the obstacles to making this integral union of the marriage happen. It is always about the "other" and being disposed to meet the others needs.