eorum as the antecedent for quo?

[akisame]

uxores habent deni duodenique inter se communes et maxime fratres cum fratribus parentesque cum liberis; sed qui sunt ex iis nati eorum habetur liberi, quo primum virgo quaeque deducta est (de bello Gallico 5, 14).

How is the last phrase connected to the rest of the sentence? I thought it is a relative clause modifying the previous eorum but they don't agree in the number. Is agreement sometimes not so rigorously enforced?

[adrianus]

eorum habetur liberi, quo primum virgo quaeque deducta est
= eorum habentur liberi, quo primum virgo quaeque deducta est

"the children are considered to belong to those [/of those, masculini generis pluraliter], by whom [masculini generis singulariter] whatever woman was taken first [in marriage] as a virgin."

The sense can override agreement about number [and gender]. He's talking about the responsibilities of each (singularly) in a group (of several).
Sensus concordantiam numeri [et generis] superare potest. Caesar curas singulis in grege (plurum) proprias tractat.

See Allen & Greenough, §280a, on construing by sense.
De constructione ad sensum, vide A&G §280a.