Drones are almost like light planes re. wind speed. Many times in high wind conditions, the enroute flying phase is not the problem, it's  the take-off & landing phases.

But, to answer your question Tom, I'd say for the Phantom 4 that 20 mph is enough. As others have said, keep in mind that if you fly too far down wind you might not be able to make it back before battery goes down, though the drone should land somewhere if battery is dying & drone isn't home yet. Hopefully you'll know where it landed. 

In less than 20 mph winds I usually take off & land from the pull-over bed cover on my pickup. But, if it's 20 mph or more, then for landing I use the "catch in hover" technique. I'm not recommending this for anyone, I'm just saying that's what I do. Also, when you take off in high winds, make sure you have a clear area & give it full power until you get about 50' in the air. It takes the drone a couple seconds to figure out hover position when you first take off. If it's windy, hover might be in the side of your pickup.

Have fun!