St. Francis de Sales has some courageous, insightful wisdom and advice for you!
He says “We Must Disregard the Criticism’s of this world’s children!”
See if you are encountering some of the arguments St Francis mentions and let him help you stay firm in your resolve.
” DIRECTLY that your worldly friends perceive that you aim at leading a devout life, they will let loose endless shafts of mockery and misrepresentation upon you; the more malicious will attribute your change to hypocrisy, designing, or bigotry; they will affirm that the world having looked coldly upon you, failing its favour you turn to God; while your friends will make a series of what, from their point of view, are prudent and charitable remonstrances. They will tell you that you are growing morbid; that you will lose your worldly credit, and will make yourself unacceptable to the world; they will prognosticate your premature old age, the ruin of your material prosperity; they will tell you that in the world you must live as the world does; that you can be saved without all this fuss; and much more of the like nature.
My daughter, all this is vain and foolish talk: these people have no real regard either for your bodily health or your material prosperity.
“If ye were of the world,” the Saviour has said, “the world would love his own; but because ye are not of the world, but I have chosen you out of the world, therefore the world hateth you.”
We have all seen men, and women too, pass the whole night, even several in succession, playing at chess or cards; and what can be a more dismal, unwholesome thing than that? But the world has not a word to say against it, and their friends are nowise troubled. But give up an hour to meditation, or get up rather earlier than usual to 
prepare for Holy Communion, and they will send for the doctor to cure you ofhypochondria or jaundice!
People spend every night for a month dancing,and no one will complain of being the worse; but if they keep the one watch of Christmas Eve, we shall hear of endless colds and maladies the next day!
Is it not as plain as possible that the world is an unjust judge; indulgent and kindly to its own children, harsh and uncharitable to the children of God? We cannot stand well with the world save by renouncing His approval. It is not possible to satisfy the world’s unreasonable demands: “John the Baptist came neither eating bread nor drinking wine; and ye say he hath a devil. The Son of Man is come eating and drinking, and ye say, Behold a gluttonous man, and a winebibber, the friend of publicans and sinners.”
Even so, my child, if we give in to the world, and laugh, dance, and play as it does, it will affect to be scandalized; if we refuse to do so, it will accuse us of being hypocritical or morbid. If we adorn ourselves after its fashion, it will put some evil construction on what we do; if we go in plain attire, it will accuse us of meanness; our cheerfulness will be called dissipation; our mortification dulness; and ever casting its evil eye upon us, nothing we can do will please it. It exaggerates our failings, and publishes them abroad as sins; it represents our venial sins as mortal, and our sins of infirmity as malicious. S. Paul says that charity is kind, but the world is unkind; charity thinks no evil, but the world thinks evil of every one, and if it cannot find fault with our actions, it is sure at least to impute bad motives to them,–whether the sheep be black or white, horned or no, the wolf will devour them if he can.
Do what we will, the world must wage war upon us. If we spend any length of time in confession, it will speculate on what we have so much to say about! if we are brief, it will suggest that we are keeping back something! It spies out our every act, and at the most trifling angry word, sets us down as intolerable. Attention to business is avarice, meekness mere silliness; whereas the wrath of worldly people is to be reckoned as generosity, their avarice, economy, their mean deeds, honourable. There are 
always spiders at hand to spoil the honey-bee’s comb.
Let us leave the blind world to make as much noise as it may,–like a bat molesting the songbirds of day; let us be firm in our ways, unchangeable in our resolutions, and perseverance will be the test of our self-surrender to God, and our deliberate choice of the devout life.
The planets and a wandering comet shine with much the same brightness, but the comet’s is a passing blaze, which does not linger long, while the planets cease not to display their brightness. Even so hypocrisy and real goodness have much outward resemblance; but one is easily known from the other, inasmuch as hypocrisy is short-lived, and disperses like a mist, while real goodness is firm and abiding.
There is no surer groundwork for the beginnings of a devout life than the endurance of misrepresentation and calumny, since thereby we escape the danger of vainglory and pride, which are like the midwives of Egypt, who were bidden by Pharaoh 
to kill the male children born to Israel directly after their birth. We are crucified to the world, and the world must be as crucified to us. It esteems us as fools, let us esteem it as mad.
Source: Introduction to the Devout Life, Part 4, Chapter 1