This is wrong. Except in extremely rare circumstances, native speakers speak their language perfectly. You might be thinking of what careful language would call 'imperfections' (saying, for instance, 'I didn't do no wrong'), but those are cases when what's being passed on is a modified version of the language that's accepted within the speaker's circle, and so (linguistically) both counts and is correct.
Using Latin imperfectly has nothing to do with this, and falls rather within the range of mistakes that a toddler himself would commit, only to be later corrected by the other speakers. ('Cuna est bona!' 'Ah, puerulle, cunae sunt bonae? En ita sunt!') If an adult can't go beyong toddler-level in teaching, well.
I agree with having fun with it. I also agree with teaching some small elements (numbers, names, and so on), and so I'm not trying to bring you down, but this shouldn't by any means be equated with a revival of the language from infancy. For that, much more than 'decent level' would be required. But insofar as you'll both learn, and that you take utmost care not to mislead the child and pass wrong information, I don't see what harm could come out of it.