Pretty sure it is just the leaves - as 40DODGE said.

Here is a link http://www.canr.msu.edu/horseadults/publications/Toxic%20Plant%20Bulletins/Trees.pdf that discusses the toxic affects.