Monthly Archives: January 2018

Proper nouns: handling some false positives

Now handling some kind of false positives related to proper nouns translation. As this type of error is somewhat widespread, it could result in a 0.2% increase in overall accuracy.

Of interest in the present case:

  • recall that ‘détroit’ is French name for strittonu (straight, i.e. the straight of Gibraltar)
  • ‘Tours’ (the French city of) is also left untranslated, also being ambiguous with torri (towers) or ghjiri (turns)
  • 12th Street riot, Michigan are left untranslated
  • self-evaluation finds erroneously 2 vocabulary errors : riot and ‘th’ in 12th

Testing self-evaluation accuracy

Testing self-evaluation accuracy: in the present case, it yields a 100% performance. However, there is one error ‘par des explosifs’ should read da splusivi or even da i splusivi (by explosives): a problem of partitive article. Arguably, there is a second grammatical error to which self-evaluation is blind: ‘sont ensuite détruits’ should read sò distrutti dopu (are then destroyed): the problem lies in the fact that prepostion dopu should be placed more adequately before the verb. In short : human evalution yields 98,14% performance in the present case. (by the way, it seems average performance on MT open test is currently nearing 94%.)

Adjective disambiguation: a hard case

A hard case of disambiguation is successfully handled here (amid some vocabulary errors) : ‘les premiers parachutistes des 82e et 101e divisions…’ is properly translated into … di i 82a è 101a divisioni… It is a hard case, since ultima facie ’82e’ and ‘101e’ are both feminine singular whereas ‘divisions’ is feminine plural (since there are two divisions). But prima facie, ’82e’ and ‘101e’ in French (in full ’82ème’ and ‘101ème’, i.e. 82th and 101th) are both ambiguous between masculine singular and feminine singular.

Also of interest:

  • self-evaluation in this case is rather reliable, as it provides roughly the same result than human evaluation
  • there is a false positive: ‘CG-4A’ being translated erroneously into CG-4Hà: this raises the issue of the correct translation of proper nouns that are composed of letters and numbers

Proper nouns: false positives again

Now we face false positives again: French proper noun ‘Détroit’ is translated erroneously into Strittonu when it shouls have been left untradslated, being a proper noun.  The ambiguity of ‘Détroit’ lies in the fact that it can be translated either into:

  • Détroit, the city
  • Strittonu, the Corsican word strittonu/strittone being the corresponding word for French noun ‘détroit’ (strait, i.e. the strait of Messina).

This raises the general issue of the proper disambiguation of proper nouns.

Evaluation of machine translation: why not self-evaluation?

Evaluation of machine translation is usually done via external tools (to cite some instances: ARPA, BLEU, METEOR, LEPOR, …). But let us investigate the idea of self-evaluation. For it seems that the software itself is capable of having an accurate idea of its possible errors.

In the above example, human evaluation yields a score of 1 – 5/88 = 94.31%. Contrast with self-evaluation which sums its possible errors: unknown words and disambiguation errors, thus entailing a self-evaluation of 92,05%, due to 7 hypothesized errors. In this case, self-evaluation computes the maximum error rate. But even here, there are some false positives: ‘apellation’ is left untranslated, being unrecognized. In effect, the correct spelling is ‘appellation’. To sum up: the software identifies an unknown word (and lefts it untranslated) and counts it as a possible error.

Let us sketch what could be the pros and cons of MT self-evaluation. To begin with, the pros:

  • it could provide a detailed taxonomy of possible errors: unknown words, unresolved grammatical disambiguation, unresolved semantical disambiguation, …
  • it could identify precisely the suspected errors
  • evaluation would be very fast and uncostly
  • self-evaluation would work with whatever text or corpus
  • self-evaluation could pave the way to further self-improvement and self-correction of errors
  • its reliability could be good

And the cons:

  • MT may be unaware of some types of errors, i.e. errors related to expressions and locutions
  • MT may be unaware of some types of errors, i.e. errors related to expressions and locutions
  • MT self-evaluation could be especially blind to grammatical errors
  • it would sometimes count as unknown words some foreign words that should remain untranslated
  • MT would be unaware of erroneous disambiguations