I think the question of which is more difficult to a certain extent depends on where you come from, as well as some other important factors. If you've grown up speaking Modern Greek, or one of the Slavonic languages which use an alphabet which is derived from the Ancient Greek one, then I think then you're more likely to find Greek easier. I do realise, though, that the original poster said explicitly that he only knew English, so Latin would be easier in this case.
The alphabet is the same as English, and something like 30% of English words derive from Latin, in comparison to something like 5% from Greek. I think Latin grammar is altogether much simpler; bear in mind that traditional Greek textbooks, at least older ones, assumed that you had Latin before you even started Greek. Maybe this gives some kind of helpful indication as to the difficulty of the two when compared to each other, I don't know. I think a lot has to be said for your motivation to learn it, as well. In my case, I am interested in Roman history as well as the influence of Latin on modern English, so I chose Latin.
I hope that helped to some small degree.