Automated Evaluation for Terminology Translation Related to the EEA Agreement

AbstractThis paper presents a submission to the WMT25 test suite subtask, focusing on terminology in official documents related to the EEA Agreement. The test suite evaluates the accuracy of MT systems in translating terminology for the English-Icelandic translation direction when applied to the EEA documents. We focus on the use of terminology in four domains of the agreement; science, technology, economics, and society. We find that the manual evaluation confirms that our test suite can be helpful in selecting the best MT system for working with these domains. Surprisingly, an online system that does not achieve very high scores on the general translation task, according to preliminary results, is most adept at translating the terminology our test suite evaluates.