I agree with the analysis you both have arrived at but want to add a few thoughts about three things: (1) what is the actual construction we see, (2) how might someone fluent in this type of Latin put together the meaning of this text as the words are encountered, and (3) what might justify the actual word order we encounter.
(1) Looking in Perseus at the entry for diffīdō, we see that it is normally construed with the dative or a dependent clause with an accusative and infinitive. Based on this, the complement of diffīdēns is such a clause, with esse omitted. I.e.,
diffīdēns (dē numerō diērum Caesarem fidem servātūrum (esse)).
What would trigger us to expect this construction as we read?
(2) In English, the preposition “about” can be used with any verb implying mental processing, emotion, or communication. In Latin, the use of dē normally complements an expression of motion or of communication, rather than an expression of emotion as in English. After reading diffīdēns and encountering dē, we should anticipate that the structure introduced by dē relates to something not yet encountered. Since there is no independent dative, we should anticipate an accusative and infinitive.
When we reach numerō diērum, a time reference, rather than a place, we are lead to expect some following expression of communication, rather than a verb of motion. In fact, we should recall that Caesar had left Cicero in charge of the garrison and had made such a communication, saying:
Discēdēns post diem septimum sēsē reversūrum cōnfirmat
Servō is not a verb of communication, but fidēs in this sense means “promise” and therefore forms an expression of communication, allowing dē to introduce what the promise concerns. In other words, Caesar had communicated a promise about the seven days.
(3) With a participle like diffīdēns, its complement could syntactically precede or follow. Since the intention is not just to say that Cicero was in a certain mood, but to shift the conversation to the trigger for the mood, the complement phrase follows.
The phrase itself has at least three elements that easily could be emphasized: the date, Caesar, or the promise keeping. Cicero’s doubts are not about who made the promise or about the promise to return as a whole, but only about the promised date. As a result, the date is fronted. If the text read, diffīdēns Caesarem servātūrum fidem de numerō diērum, you might conclude that Cicero was in doubt because it was Caesar that made the promise and Caesar was not generally trustworthy. Some of these nuances could be expressed in English by choosing to raise the intonation to a high pitch on the stressed syllable of the element to be emphasized and dropping it to a low pitch immediately after.