Annotation Instructions Tutorial
General Instructions:
Given a group of words, the purpose of this work is to determine whether:
-
a large language model was able to label the group correctly?
-
was the label bu the LLM better than human label?
For example, the following group of words (numbers) form a meaningful group. Please check below for more detailed annotation instructions.
The annotation interface will show word groups like the one shown above and then asks the following questions:
Q1: Does the label {LLM LABEL} meaningfully describe the above group of words?
Options:
-
Accurate if the label is perfect for the group
-
Acceptable if the label could be better, or if it needs to be processed
-
Unacceptable if the label is irrelevant
Q2: Given the LLM label {LLM LABEL} and the Human label {HUMAN LABEL}, which is better?
Options:
-
LLM is Better if the LLM label is better
-
Equivalent if the LLM label is equivalent
-
Human is Better if the LLM label is worse