As long as you remember that adjectives agree in GENDER not in FORM, you will have little difficulty with this sort of thing. Once you get into the third (and other) declensions, adjectives and nouns will start to look less and less alike. In fact, Latin authors, especially poets, but also prose writers, prefer to avoid the jingle (tintinnatio?) of sounds like uirorum bonorum.
Of course, there's the whole problem of memorizing the gender of nouns! But there are some good clues you will learn for the 3rd declension, and the other declensions are pretty easy. Plus, with practice will come facility.