Havent read all the responses, I'm sure they are wide and varying but truthful so I'll give you my perspective.

You are not alone,  

My wife and I have been doing somthing similar.  Her and I both were children of multiple generation farms that saw its operations downsize over the years due to $$ from the 80s, 90s. etc.  

We left in 1996 started our family out of state and our careers.  I knew the goal would be to come home eventually which we did in 2010 with 3 kids.  I've ben foruntate to have a good job/career  - but we never did good at a "savings" account.  I took the 200 acre farm back with a 4020 in 2010.  Bought 10 heifers and since then, have expanded.  99% of the capital was was extra off farm $$ reinvested into the farm.

Tons of elbow grease and definitely aged me.  Rented some **** pasture, and crappy ground that no one else wanted.   Maintained sales job while she took care of kids.   Paid dump trucks of interest to the bank but grew our balance sheet.....not profit.   We feed 200 head, row crop 2500 acres, 2 full time employees with a fleet of equipment (mostly 30,40, 50, 60 series JD) that's paid for except a skid steer.   The farm is now paying for itself with little to no debt.

Ag is all about equity and assets.   Good luck - human putty is the common denominator that keeps it going.   1 of my FT employees is 21 year old son so the last 14 years has been worth it as I can see him taking one reins one day and then I'll know its worth it.

Not saying what we did or doing is right just sharing.  Made lot of mistakes that no one could have told nor would I have listened but man I've learned alot about a lot!