I have never intentionally downsized my farming, but I did own a business once that I stepped back from.  I owned a tree service that mainly did utility line clearance.  I had 60 employees at the peak.  I sold my half of the business to my partner in 2004.  I started getting requests from locals to do residential tree work around home, and shortly I was back in the business with just two employees.  I found I had very little stress, and I made almost as much money with the new business with three of us doing only high margin jobs as I had with 60 of us doing low margin work.  

Because of family needs, I later sold out of that and concentrated on farming.  The lesson I learned was to concentrate on the thing that was the most profitable.  Sometimes growth in any sort of business only means that the lawyers, bankers, insurance companies, and suppliers are making more money off of you, and all you are "gaining" is more stress, and a higher "cool" factor.

One other thing, I have learned that the American male farmer and/or business owner has way too much sense of self worth tied up in what they do for a living.  Be very careful with how you deal with major changes in your farming or business if you downsize it.  I struggled with that some, and it was tough to slow down and let some of the stress go.