I think if you understand nec more as “and…not” rather than “nor” in this context, you’ll get your translation where you want it to be.
“I do not know why I told you all this, and indeed not without the greatest pain do I remember that time…”
Also, another point, keep in mind that reminīscor will frequently take the genitive case for the thing that is being remembered/recalled. In this case, that thing is eius temporis.
I see your point, but now I think I’d give ‘enim’ a more proactive role and keep ‘nec’ just as negation to get, what seems to me, a more natural translation:
“I do not know why I told you all this, because not without the greatest pain do I remember that time…”