causa is ablative, it's true. I'm not sure why that's the case (pun!), but I do know this construction works basically like a preposition. Since, however, causa comes after the word it governs, you might call it a postposition, though I don't know if that's actually a grammar category.
Also, as you've already determined, causa takes a word in the genitive case. This is obviously unusual for a preposition. Maybe it's better just to think of this construction as an idiom and leave it at that?
One way to translate it is "For the sake of..." + (whatever genitive word comes before it).