With a few exceptions, as swtwentyman notes, learning the grammatical concepts is probably one of the easier parts. This is especially true if you’ve studied other languages before. Moving from a conceptual understanding of the grammar to actually reading continuous text is where the trouble lies. Historically, people would hit the wall when the finished their first year of Latin (which was mostly focused on learning/drilling grammar, such as conjugations and declensions) and then embarked on Caesar in their second year course. Of course, Caesar is considered one of the easier authors–the difficulty lies in having to deal with novel syntax, constructions, and vocabulary.
Otherwise, a big challenge is vocabulary. It’s easy to feel pretty sharp when you are working with the same handful of nouns/verbs/adjectives in every example. But reading can be a real drag when you have to look up every other word (and then forget them and look them up again later). Most of my memories of my first years of Latin involve me endlessly looking up words.
Getting the hang of poetry vs. prose is also a hurdle. I was starting to get used to simple prose when I started a class on Vergil. At the beginning I remember being totally confused on the syntax; even after having looked up all of the words/declensions/conjugations, I couldn’t figure out how everything went together.