I have not been in your shoes.  However, two questions come to mind as I read your post:

1) Does your brother have the flexibility to deal with getting fertilizer, trucking, etc if that becomes a problem for you?  Can you restructure your working relationship to deal with these issues?  For example, you might spend more time doing actual field work than before, to offset the hours your brother may spend dealing with obtaining inputs and moving grain. Is this an angle to consider?

2) Since you say that farming full-time is probably not in the cards, it may come down to what your priorities are. Which is more important to you: having the flexibility in your day job to do your farming?   or having a day job you enjoy?  I think the choice between these is something only you can answer.