If we both agree to build half of the fence because we both use or possibly will use it we did just that we "agreed".  

Nobody was forced into being responsible for putting up a fence they didn't want or need.

Even in that scenario I would keep responsibility for my livestock, if they got out I would pay for damages and fix the fence without asking him to do it.

I'm not opposed to neighbors/adjacent landowners building a fence together if they both agree and both want or need a fence.  

I'm opposed to the law that says if one wants a fence the other "has to" put up his half and the livestock owner is not responsible for damage his livestock does to others property.