Sri Krishna says in the Bhagavat gita that those people who do not sacrifice their actions to Him enjoy sense and body enjoyments(probably also mental)like the animals do and their life is not worth the human birth.
The abandoning sense enjoyment and the suggestions in the mind of indulgence in sense enjoyment knowing fully that there is a higher delight of our self existence within us and experiencing this delight which is far superior in quality to sense enjoyment is the secret that a wise man can really experience in life.
Does it mean that one also abandon relationships with other human beings? Relations do not necessarily be abandoned but care needs to be taken that one does not fall to the temptations of sense enjoyments and remain calm during the tumult of desires to indulge. It needs great faith in the divine to reject lower happiness for higher happiness of the self. One may temporarily abandon the physical circumstance that cause the lower happiness of the senses but life does not provide such opportunities all the time. Most of us are householders and have to take care of our families. A temporary vacation is fine but complete jettison of responsibilities is questionable. Of course all depends on the inner call. But eventually the yogin has to deal with everybody and he needs to learn through experience not to succumb to pressure and suggestions of lower happiness of the body.
The higher happiness of the self is waiting for you. Do not fall into the lure of the lower happiness of the body. Be wise. God will take care of you if you do your part.