KEOPS (Keen Evaluation Of Parallel Sentences) provides a complete tool for manual evaluation of parallel sentences.
- Your role in keops
- Getting started
- Managing tasks and projects
- Evaluation modes
- Managing corpora
- Managing users
- Managing languages
- Feedback
The parallel sentences are organized in tasks. A task defines a group of sentences which should be reviewed by an evaluator in a certain way.
As a project manager, your mission is to create tasks, assign them to evaluators and track their progress.
Even if you are a project manager, you can be assigned tasks. For more information about how to evaluate tasks, please refer to the evaluators guide.
In order to use KEOPS, you need an invitation. Only another project manager can provide you with one. In addition, the project manager who invited you has to grant project manager privileges to your account.
If you already have your project manager account, sign in to use KEOPS. You can do this by clicking on Sign in in the homepage.
This will take you to the Sign in screen, where you just have to fill the fields with your email and password.
When you are invited to KEOPS, you will get an invitation link. If you follow that link, you will be able to sign up.
Fill the required fields and click on Sign up. This will create your evaluator account. From now on, you can sign in using your email and password.
Instead of an invitation link, you could get an invitation token. This token will look like this:
5d4404fb9f66a
On the Sign up screen, fill the Access Token field with this token.
Projects group tasks which are somehow related. A task must always belong to a project, so projects may contain only one task. When creating a project, you will be asked for a name and a description. Use this information to correctly identify your projects.
Tasks group sentences which should be evaluated in the same way. You can create tasks once you create the project they belong to. When creating a task, you will be asked for:
- Mode: the way this task should be evaluated. Refer to the modes section for more information.
- Source language: the language of the source texts. For certain kinds of evaluation, this will not be required.
- Target language: the language of the target texts.
- Evaluator: the evaluator who should evaluate this set of sentences. Only evaluators who have knowledge of both source and target languages can be chosen.
- Corpus: the set of sentences to evaluate. Refer to managing corpora for more information.
If you click on Recap all on the Projects screen, you get an overview of all your current projects and their tasks.
A task has always an evaluation mode. This will change the way evaluators review the sentences.
This mode is used to classify pairs of sentences using the European Language Resource Coordination (ELRC) validation guidelines. These guidelines define a set of labels to classify sentences:
- Wrong language identification
- Incorrect alignment
- Wrong tokenization
- MT translation
- Translation error
- Free translation
- Valid translation
You may refer to the ELRC Validation Guidelines document for more information about these labels.
If a label is not selected, the sentence will be pending.
When selecting some of the labels, additional information may be asked:
- Wrong language identification: whether the source, the target or both texts are wrong identified.
- MT Translation: whether the source, the target or both texts are MT-translated.
- Free translation: whether the translation should be kept, even though it is a free one.
Please note that this additional information is completely optional to provide.
Moreover, KEOPS can be also used to flag sentences which contain personal data or inappropriate language. This is done by using the checkboxes on the bottom right of the screen.
- A pair of sentences contains personal data when the content includes proper names or other personal data that could be anonymised for data protection purposes.
- A pair of sentences contains inappropriate language when it includes profane expressions.
When evaluating adequacy, evaluators must assess that the candidate translation expresses the meaning of the source text in an adequate way. This is, that the candidate translation accurately captures the meaning of the source text.
The source text is presented to evaluators on the left of the screen and, the candidate translation, on the right.
Adequacy mode adds quality control sentences to the corpus. These sentences can be previewed at any time, and their creation is based on the research of Cambridge Core.
Once the evaluator is done with the task, a score is computed assessing their performance on the control sentences. This score may be mediocre even for good evaluators, since the used metrics are strict. We do not recommend to ban an evaluator because of this score as the only reason. The score may be used as an illustrative measure of the performance of the user on that task, but project managers should always corroborate the score.
When evaluating fluency, evaluators must assess that the presented text is fluent in its language. The sentence to evaluate is shown on the left of the screen.
When performing this kind of evaluation, a source text and a reference translation is presented to the evaluator. Then, evaluators must rank a set of candidate translations. The first position belongs to the best translation and, the last position, to the worst one.
When evaluating paraphrases, a source text and a candidate paraphrase is presented to the evaluator. The evaluator must assess that the candidate is a valid paraphrase of the source text. A bunch of guidelines are available on the evaluating screen, in order to help the evaluator assess the paraphrases.
A set of sentences is called a corpus. Corpora may have a different structure depending on the evaluation mode they are meant to be used in. KEOPS lets you upload, preview and assign corpora for evaluation.
Corpora is always uploaded in TSV format. This format uses one line per record and a tab character to separate fields.
The specific format of the TSV file is explained below for each of the evaluation modes. This information is alsa available on KEOPS clicking on First time uploading corpora. You can also download a template and use it to upload your data:
Please note that all formats contain headers which are necessary in order to upload corpora to KEOPS. If you have a corpus without headers, you can add them using:
echo -e "Header\tAnother header" | cat - corpus-without-headers >
corpus-with-headers
| Source text | Tab | Target text |
|---|---|---|
| You can contact our customer service department using the form below | Tab | Puedes ponerte en contacto con nuestro departamento de servicio al cliente mediante el siguiente formulario |
You should only include one target text for each source text.
| Source text | Tab | Candidate translation |
|---|---|---|
| You can contact our customer service department using the form below | Tab | Puedes ponerte en contacto con nuestro departamento de servicio al cliente mediante el siguiente formulario |
You should only include one candidate translation for each source text.
| Candidate translation |
|---|
| Puedes ponerte en contacto con nuestro departamento de servicio al cliente mediante el siguiente formulario |
Corpora for fluency evaluation consist only on one column because they are monolingual tasks.
| Source text | Tab | Reference text | Tab | Name of system 1 | Tab | Name of system 2 | Tab | ... |
|---|---|---|---|---|---|---|---|---|
| You can contact our customer service department using the form below | Tab | Puedes ponerte en contacto con nuestro departamento de servicio al cliente mediante el siguiente formulario | Tab | Manual de empleo y manutención | Tab | Manual de empleo y manutención | Tab | ... |
Include as many systems as you want.
| Source text | Tab | Candidate paraphrase |
|---|---|---|
| Have the right to choose | Tab | Have the right to elect |
You should only include one candidate paraphrase for each source text.
As a project manager, you have de capability of inviting new users to KEOPS. The new users may have evaluator or project manager privileges.
Use the Invitations tab on the Management page to invite a user. You only need to provide their email address. Make sure to provide a valid one, since they will be notified via mail about the invitation.
Once the user accepts the invitation and signs up, you can modify their information:
- Name: the name of the user.
- Email: the email address of the user.
- Languages: the languages the user has knowledge of.
- Role: whether this user is an evaluator or a project manager.
- Active: whether the user can sign in.
A comprehensive list of languages is already provided when KEOPS is installed. However, you may add additional languages using the Add language action on the Languages tab of the Managment screen.
When adding a new language, you are asked to type a language code and a language name.
- The language code must have the form of an ISO 639-1 code.
- The language name can be any string which describes the language (for example, "Spanish")
When an evaluator finishes a task, they can provide feedback about their experience. As a project manager, you can read this feedback on the Managment screen.
A list of the provided comments is displayed, along with the user who made them. If you click on Contact, you can get in touch with that user.