It’s a nice figure, making a point of the mutuality: each is absent from the other. Lit. … the remembrance of whom (when you were) absent [from me] I (when I was) absent [from you] used to enjoy just the same as I (when I was) present had been accustomed to the consuetudo of (you when you were) present. This should make sense of the tenses for you too. Doesn’t work in English of course.
Anthony Appleyard posted a pretty poem at http://discourse.textkit.com/t/a-latin-poem-by-t-s-evans/15458/1
that ended huc deflecte pedem, memoris memor hospes amici. Each is memor of the other.
Hope this helps.