I suppose it would. However, "ad" usually carries the conotation of "to the limits of" or "to the outside of". A typical explanation is that "Ad Romam" means "to the walls/gates [limits] of Rome" whereas "In Romam" means "into the city". Also to note that in+acc can also mean "against" with a strong opposition or an attack, eg "in Catilinam", "in hostes".
"ad castra" would thus have that conotation, not that bringing the booty inside, but rather carrying carrying it to the limits. So you could, but the solution given seems to me more appropriate.
In any case, if someone more knowledgeable would shed more light on the subject, I think it would be helpful.