Dronedude...

Time to clear up some misconceptions here...

Read and understand this..especially part C

 Sec. 91.119

Minimum safe altitudes: General.

Except when necessary for takeoff or landing, no person may operate an aircraft below the following altitudes:
(a) Anywhere. An altitude allowing, if a power unit fails, an emergency landing without undue hazard to persons or property on the surface.
(b) Over congested areas. Over any congested area of a city, town, or settlement, or over any open air assembly of persons, an altitude of 1,000 feet above the highest obstacle within a horizontal radius of 2,000 feet of the aircraft.
(c) Over other than congested areas. An altitude of 500 feet above the surface, except over open water or sparsely populated areas. In those cases, the aircraft may not be operated closer than 500 feet to any person, vessel, vehicle, or structure.
[ (d) Helicopters, powered parachutes, and weight-shift-control aircraft. If the operation is conducted without hazard to persons or property on the surface--
(1) A helicopter may be operated at less than the minimums prescribed in paragraph (b) or (c) of this section, provided each person operating the helicopter complies with any routes or altitudes specifically prescribed for helicopters by the FAA; and
(2) A powered parachute or weight-shift-control aircraft may be operated at less than the minimums prescribed in paragraph (c) of this section

I fly airplanes and drones....in sparsely populated areas...I can fly airplanes with my wheels barely off the ground...as long as I stay 500 feet HORIZONTALLY from  any person, vessel, vehicle, or structure.

For part 137 operations ( rules for spray planes) there is even more leeway..

G.    Minimum Safe Altitudes—Other Than Congested Areas. Part 137, § 137.49 permits operations at altitudes contrary to part 91, § 91.119, Minimum safe altitudes: General, provided such operations are conducted without creating a hazard to persons or property on the surface and are in conjunction with aerial application activities. However, flights between dispensing operations must comply with the provisions of § 91.119. For example, the pilot of an agricultural aircraft dispenses an economic poison on a field adjacent to a farmhouse. The pilot may operate less than 500 feet above the surface and closer than 500 feet to the house provided the house and its occupants are not exposed to undue hazards from the aircraft or the chemicals. However, when flying to the dispensing area or returning after application, the aircraft must maintain § 91.119 requirements.

P.S.

Don't know where you figure a guy flying an airplane without a radio could somehow be irresponsible...I have 3 planes and 2 of them don't even have a radio.

I could legally fly them coast to coast without needing a radio, a flight plan, or ever talking to anyone.