SemEval-2018 Task 11: Machine Comprehension Using Commonsense Knowledge
Published in Proceedings of the 12th International Workshop on Semantic Evaluation, 2018
This report summarizes the results of the SemEval 2018 task on machine comprehension using commonsense knowledge. For this machine comprehension task, we created a new corpus, MCScript. It contains a high number of questions that require commonsense knowledge for finding the correct answer. 11 teams from 4 different countries participated in this shared task, most of them used neural approaches. The best performing system achieves an accuracy of 83.95\%, outperforming the baselines by a large margin, but still far from the human upper bound, which was found to be at 98\%.
Recommended citation: Simon Ostermann, Michael Roth, Ashutosh Modi, Stefan Thater, and Manfred Pinkal. (2018). "SemEval-2018 Task 11: Machine Comprehension Using Commonsense Knowledge." Proceedings of the 12th International Workshop on Semantic Evaluation.
Download Paper