I feel for you as over 2 years ago I walked out of a very good job I held for 25 years.  I was tired (mentally), saw future problems, and simply just wanted to spend my energy on the farm, not divided between the two.  That was a true leap of faith.  With a family that appreciates what I was going through at work and God's help it is working. 

Since your family is in agreement you will do just fine whatever you choose to pursue.  Keep the faith and hug the family.