Evaluation Result Board

 

Under ‘Testing and Benchmarking’ project funded by MeitY, C-DAC has evaluated some of consortia system which were also funded by MeitY. The system name and its accuracy level results has been published here under Accuracy Result Board. Evaluation of the MT systems were  carried out by evaluators per language on sentences test data for tourism, health domain & legal domain. The performance of the system was evaluated on two parameters Comprehensibility (with respect to the meaning in the source text) & Fluency (naturalness in the target language text). 

Hindi - English Machine Aided Translation For Judicial Domain has been developed under consortium mode, led by IIT Patna. This evaluation was done in year 2020.

Accuracy Result Board

Language Pair Comprehensibility % Fluency %
Blind Open Blind Open
English - Hindi 91.12 96.53 71.99 81.41
Hindi - English 96.53 94.6 84.13 76.83

 

Under ‘Testing and Benchmarking’ project funded by MeitY, C-DAC has evaluated some of consortia system which were also funded by MeitY. The system name and its accuracy level results has been published here under Accuracy Result Board. Evaluation of the MT systems were  carried out by evaluators per language on sentences test data for tourism, health domain & legal domain. The performance of the system was evaluated on two parameters Comprehensibility (with respect to the meaning in the source text) & Fluency (naturalness in the target language text). 

Project Anuvaad has been conceptualised to provide translation capabilities for Indic languages. Project Anuvaad is open sourced under the MIT license. This evaluation was done in year 2020.

Accuracy Result Board

Language Pair Comprehensibility % Fluency %
Blind Open Blind Open
English - Hindi 81.93 96.33 71.10 78.93
Hindi - English 92.47 93.13 77.10 68.55

 

Under ‘Testing and Benchmarking’ project funded by MeitY, C-DAC has evaluated some of consortia system which were also funded by MeitY. The system name and its accuracy level results has been published here under Accuracy Result Board. Evaluation of the MT systems were  carried out by evaluators per language on sentences test data for tourism, health domain & legal domain. The performance of the system was evaluated on two parameters Comprehensibility (with respect to the meaning in the source text) & Fluency (naturalness in the target language text). 

Anuvadaksh MT System is an English to Indian Language Machine Translation System which allows translating the text from English to eight other Indian languages.This system has been developed by EILMT consortium, led by C-DAC. This evaluation was done in year 2019.

Accuracy Result Board

Health Domain Tourism Domain
Language Pair Comprehensibility % Fluency % Comprehensibility % Fluency %
Blind Open Blind Open Blind Open Blind Open
English - Hindi 73.53 52.32 68.33 50.93 48.39 39.73 77.11 52.28
English - Marathi 54.46 37.22 59.09 39.52 37.15 32.18 49.00 36.66
English - Bangla 32.39 33.08 43.66 36.28 30.66 33.38 52.68 40.76
English - Urdu 50.37 40.56 64.73 45.24 59.43 45.19 82.67 54.85
English - Odia 42.52 36.72 44.00 37.84 39.14 36.81 42.62 37.78
English - Gujarati 68.00 45.47 78.69 50.47 9.82 17.79 31.33 29.86
English - Tamil 49.67 38.53 51.33 38.13 63.64 45.32 58.58 43.28
English - Bodo 1.54 11.54 3.27 15.45 0.53 8.33 3.81 13.28

 

Under ‘Testing and Benchmarking’ project funded by MeitY, C-DAC has evaluated some of consortia system which were also funded by MeitY. The system name and its accuracy level results has been published here under Accuracy Result Board. Evaluation of the MT systems were  carried out by evaluators per language on sentences test data for tourism, health domain & legal domain. The performance of the system was evaluated on two parameters Comprehensibility (with respect to the meaning in the source text) & Fluency (naturalness in the target language text). 

AnglaBharati MT system is a machine-aided translation methodology specifically designed for translating English to Indian languages. AnglaMT is a pattern directed rule based system with context free grammar like structure for English. This evaluation was done in year 2014.

Accuracy Result Board

Health Domain Tourism Domain
Language Pair Comprehensibility % Fluency % Comprehensibility % Fluency %
Blind Open Blind Open Blind Open Blind Open
English - Hindi 73.81 77.21 49.66 49.32 91.33 90.67 54.58 59.25
English - Bangla 8.59 20.62 14.78 22.16 7.33 17.67 13.42 21.25
English - Urdu 23.16 31.93 31.32 34.04 38.72 45.79 36.62 38.47
English - Punjabi 39.19 39.01 30.68 29.96 44.44 57.35 34.86 43.37
English -
Malayalam		 15.41 11.36 22.22 14.47 28.96 33.33 27.86 27.36
English - Telugu 37.11 39.52 31.79 29.38 44.78 48.48 35.35 36.53

 

Under ‘Testing and Benchmarking’ project funded by MeitY, C-DAC has evaluated some of consortia system which were also funded by MeitY. The system name and its accuracy level results has been published here under Accuracy Result Board. Evaluation of the MT systems were  carried out by evaluators per language on sentences test data for tourism, health domain & legal domain. The performance of the system was evaluated on two parameters Comprehensibility (with respect to the meaning in the source text) & Fluency (naturalness in the target language text). 

Sampark: Indian Language to Indian Language MT System has been developed by ILMT consortia led by IIIT Hyderabad. This evaluation was done in year 2014.

Accuracy Result Board

Health Domain Tourism Domain
Language Pair Comprehensibility % Fluency % Comprehensibility % Fluency %
Blind Open Blind Open Blind Open Blind Open
Hindi - Punjabi 44.00 29.00 34.50 26.75 51.00 49.00 42.75 42.75
Punjabi - Hindi 75.76 89.90 51.01 57.32 77.00 75.00 54.25 53.25
Hindi - Urdu 38.00 29.00 36.00 33.25 41.00 54.00 40.25 45.50
Urdu - Hindi 47.00 75.00 37.75 46.50 51.00 70.00 41.75 49.50
HIndi - Telugu 27.00 52.00 27.25 38.00 77.00 90.00 53.25 58.75
Telugu - Hindi 8.60 41.94 18.28 32.53 60.82 74.23 41.24 47.94
Hindi - Marathi 17.00 24.00 25.50 27.50 32.00 24.49 28.50 24.74
Marathi - Hindi 20.00 16.00 28.75 28.25 41.00 27.00 33.25 26.50
Hindi - Bangla 31.00 34.00 33.75 32.75 56.00 53.00 39.50 37.00
Bangla - Hindi 8.00 10.00 19.25 24.25 34.34 37.37 36.87 37.88
Hindi - Tamil 56.00 66.00 42.00 47.75 33.67 63.27 33.67 43.37
Tamil - Hindi 8.51 38.30 17.55 36.70 26.92 55.77 29.81 40.87
Tamil - Malayalam 0.00 0.00 0.53 0.27 0.00 0.00 0.00 0.26
Malayalam- Tamil 10.00 10.00 16.75 20.00 28.57 32.65 36.99 37.76
Hindi - Kannada 54.00 69.00 38.75 43.00 43.00 48.00 42.50 41.00
Kannada - Hindi 0.00 4.00 10.00 16.75 21.43 32.65 23.21 30.61
Tamil - Telugu 51.55 55.67 38.14 41.24 72.45 76.53 47.96 51.53
Telugu - Tamil 84.04 82.98 51.33 53.99 88.54 85.42 63.28 62.24

 

Testing and Evaluation Strategies

Under CoE-Bahubhashak pilot project, testing and evaluation strategies for MT System, OCR, and TTS have been formulated for Leadership board. The initial draft of test methodology includes subjective and automatic metrics. This document can be referred by the testing agency for determining the overall quality of such system.

Links to Testing & Evaluation Specifications document below: