Analyzing MTQE Results
Once MTQE has been enabled and employed as part of your MT process, you may want to measure how accurate the MTQE scores are for your content and engine. While direct comparison between segment-level MTQE scores and post-editing analysis is not available, the following options provide ways for you to evaluate MTQE scores by both quantitative and qualitative measures.
Evaluating with Post-editing Analysis
The post-editing analysis shows the "editing effort"—how much the Linguist or the Proofreader had to edit the text. For the post-editing analysis in projects with Machine Translation and MTQE, the results are calculated as the difference between the machine translation suggestion and the post-editors final text.
Note: In order to be able to evaluate the results of the post-editing analysis, you should run a Default Analysis before the first step of the workflow to see how the MT matches are categorized into MTQE bands. To run a default analysis, select the job(s) from the Project page and click the Analyze button. Make sure to select Include machine translation matches (Beta) and then click Analyze.
To run the Post-editing Analysis, open the project, select one or multiple jobs, and click on the Analyze button. Change the Type to Post-editing, and click the Analyze MT post-editing checkbox along with other options as desired.
If the MT or NT suggestion was accepted without further editing (the Linguist did not need to change it at all), it will come up as a 100% match in the analysis.
If, on the other hand, the Linguist changes the MT, the match rate will be lower. The score-counting algorithm is identical to the one that we use to calculate the score of translation memory fuzzy matches.
If your default analysis showed a high number of quality MT matches (75% or above), the post-editing analysis should reflect a correspondingly minimal to moderate amount of edits to the MT suggestions.
Evaluating the Segment Changes
If you want to dive deeper to evaluate the substance of the changes made during post-editing, you can create a workflow that will generate a report showing the changes on a segment-level. To do this, follow the steps below:
- Create a project with 2 Workflow steps (e.g. pre-translation and post-editing).
- In the first Workflow step, pre-translate the job with only MT to get a snapshot of the matches to be used.
- In the second Workflow step, let the translator post-edit normally.
- Once the workflow is completed, you can run the Post-editing analysis to see the edit distance between the two steps (quantity of changes).
- To see the different versions of the segments, you can select the relevant jobs, then Tools > Export Workflow Changes