As the original humans who crashed in the forest morphed into what are now known as the elves, they also became strong in magic. The process was so gradual that it was not considered as an anomaly to the elves but would have been seen as an anomaly to the original humans.
This magic was basically arcane in nature and has been developed to a very high level.
All elves have a strong magical component in their makeup. With the High Elves this manifests itself as arcane magic with an increase in intelligence.
It is the difference in the magical practices between the Forest and High Elves that is the main reason for their distrust. Neither can fully understands the inherent magical power of the other but just know that it can be quite deadly.