Annotation Guideline
Purpose
Rate the audio clip using three 1-5 scores:
- Naturalness
- Audio Quality
- Intelligibility
Rate only what you hear. Do not judge based on any outside information.
Listening Instructions
Use headphones if possible, listen in a quiet place, and keep the volume at a comfortable level.
- You may replay the clip if needed.
- If the audio is broken or cannot be played, mark it as invalid.
Rating Scale
| Score | Meaning |
|---|---|
| 5 | Excellent |
| 4 | Good |
| 3 | Fair / acceptable |
| 2 | Poor |
| 1 | Bad |
1. Naturalness
Question
How natural and human-like does the speech sound?
Focus on:
- Human-likeness
- Rhythm and flow
- Pauses
- Intonation
- Robotic or synthetic feeling
| Score | Guideline |
|---|---|
| 5 | Very natural; sounds like a real human speaker. |
| 4 | Mostly natural, with only small unnatural moments. |
| 3 | Acceptable, but clearly synthetic or slightly awkward. |
| 2 | Unnatural, robotic, or distracting. |
| 1 | Very unnatural, broken, or hard to listen to. |
2. Audio Quality
Question
How clean and high-quality is the audio?
Focus on:
- Noise
- Distortion
- Clipping
- Muffled sound
- Glitches or audio cuts
- Volume problems
Do not judge whether the voice sounds human here. Only judge the technical sound quality.
| Score | Guideline |
|---|---|
| 5 | Clean, clear audio with no noticeable problems. |
| 4 | Good audio with minor issues that are not distracting. |
| 3 | Noticeable noise or artifacts, but still usable. |
| 2 | Poor audio quality; problems are distracting. |
| 1 | Very poor or unusable audio. |
3. Intelligibility
Question
How easy is it to understand the words?
Focus on:
- Clear pronunciation
- Correct words
- Missing words
- Repeated words
- Slurred or unclear speech
| Score | Guideline |
|---|---|
| 5 | All words are clear and easy to understand. |
| 4 | Mostly clear, with one or two minor issues. |
| 3 | Some unclear or incorrect words, but the meaning is understandable. |
| 2 | Hard to understand; several words are unclear or wrong. |
| 1 | Mostly unintelligible or incorrect. |