Logično sklepanje v naravnem jeziku za slovenščino

In recent years, large language models have been the most successful approach to natural language processing. An important problem in this field is natural language inference, which requires models to contain relatively broad general knowledge. Moreover, the requirement for models to explain their reasoning can offer additional insights into their functioning. We tested several approaches for natural language inference in Slovene. We used two Slovene large language models, SloBERTa and SloT5, as well as a much larger English model GPT-3.5-turbo. Training data consisted of the Slovene dataset SI-NLI and an additional 50,000 machine-translated samples from the English dataset ESNLI. The SloBERTa model was fine-tuned on both datasets. Fine-tuning it on the SI-NLI dataset achieved a classification accuracy of 73.2% on the SI-NLI test set. Pretraining it on the ESNLI dataset improved its accuracy to 75.3%. We observe that models make different types of errors compared to humans and that they generalize poorly across different datasets. The SloT5 model was also fine-tuned on ESNLI to generate explanations for natural language inference samples. Less than a third of explanations were appropriate, with the model learning common sentence patterns from the domain and producing semantically meaningless explanations. We assume that the tested Slovene large language models with up to several hundred million parameters are capable of identifying and using language patterns, but their language understanding is not necessarily sufficient to understand reality. When the considerably larger GPT-3.5-turbo was used both for classification and explanation generation, it achieved an accuracy of 56.5% on the SI-NLI test set using zero-shot learning, but with 81% of the explanations being appropriate for the correctly classified samples. In comparison with smaller Slovene models, this model shows a reasonable understanding of reality but is limited by its lower Slovene proficiency.