Resources

MTQE Evaluation

Analyzing MTQE Results

Once MTQE has been enabled and employed as part of your MT process, you may want to measure how accurate the MTQE scores are for your content and engine. While direct comparison between segment-level MTQE scores and post-editing analysis is not available, the following options provide ways for you to evaluate MTQE scores by both quantitative and qualitative measures.

Evaluating with Post-editing Analysis

The post-editing analysis shows the "editing effort"—that is, how much the Linguist or the Proofreader had to edit the text. For the post-editing analysis in projects with Machine Translation and MTQE, the results are calculated as the difference between the machine translation suggestions and the final text after the post-editing has been done.

Note: In order to be able to evaluate the results of the post-editing analysis, you should run a Default Analysis before the first step of the workflow to see how the MT matches are categorized into MTQE bands. To run a default analysis, select the job(s) from the Project page and click the Analyze button. Make sure to select Include machine translation matches (Beta) and then click Analyze.

To run the Post-editing Analysis, open the project, select one or more jobs, and click on the Analyze button. Change the Type to Post-editing, and click the Analyze MT post-editing checkbox along with other options as desired.

If the MT or NT suggestion was accepted without further editing (that is, the Linguist did not need to change it at all), it will come up as a 100% match in the analysis.

If, on the other hand, the Linguist changes the MT, the match rate will be lower. The more the Linguist changes the segment, the lower the score will be. This is the same score-counting algorithm as the one we use to calculate the score of translation memory fuzzy matches.

If your default analysis shows a high number of quality MT matches (75% or above), the post-editing analysis should reflect a correspondingly minimal to moderate amount of edits to the MT suggestions.

Evaluating the Segment Changes

If you want to dive deeper to evaluate the substance of the changes made during post-editing, you can create a workflow that will generate a report showing the changes on a segment-level. To do this, follow the steps below:

  1. Create a project with two Workflow steps (e.g. pre-translation and post-editing).
  2. In the first Workflow step, pre-translate the job with only MT. This will give you a snapshot of the matches to be used.
  3. In the second Workflow step, let the translator post-edit normally.
  4. Once the workflow is completed, you can run the post-editing analysis to see the edit distance between the two steps (the number of changes).
  5. To see the different versions of the segments, you can select the relevant jobs, then go to  Tools and select Export Workflow Changes.