Yes, LLMs are stochastic word (token) prediction engines. They do not have any sense of true or false, they just figure out the next most likely “word” or “character” based on its training data and context. Hallucination is a pejorative term to define undesirable model outputs. But there are strategies to mitigate hallucinations. When it hallucinates, it means that its training data and its context cannot appropriately weight the model to accurately predict the expected outcomes. That’s why I said the model, prompt, and context matter.
Models are trained on different sets, have different context windows (the ability to retain more or less information in its mind when answering), and different post-training refinements. So model selection matters a lot.
Prompts set the initial pattern and conditions of what the model is doing. And you can derail an LLM easily through a bad prompt.
Context speaks to all the things the model has at its disposal to actually answer the question of the user. If you just open ChatGPT or Claude and throw a succinct question without providing it context, the chances of the model being able to accurately answer your question significantly increases. But if you provide it context around your question (a pdf, a website, a document with additional information, etc.) the model’s ability to answer your question accurately goes up significantly and in most cases crosses the threshold of reliability.
Hallucinations just reveal the models lack of training data in the particular field it is being consulted.
All that said, I completely respect and understand your perspective. I really appreciate your push back! I think we are aligned on the concerns, I am just a little more optimistic in their use than you? Model outputs should not be trusted blindly or uncritically. Having a knowledgable human in the loop makes a big difference.