Data Annotation Chapter 04: Video Tracking & Audio Transcription
Introduction: The Dimension of Time
In Chapter 03, we learned how to label a single moment in time (an image). In Chapter 04, we introduce the Temporal Dimension. Video annotation isn't just about labeling an object; it is about tracking that object as it moves, changes shape, and disappears behind other things.
Additionally, we will cover Audio Transcription, which is the backbone of ASR (Automatic Speech Recognition) systems like Siri, Alexa, and Google Assistant.
Part 1: Video Annotation & Object Tracking
In video annotation, you don't want to draw a new box for every single frame (a standard video has 30 frames per second!). Instead, we use Interpolation.
1. Linear Interpolation (The "Keyframe" Method)
How it works: You draw a box around a car in Frame 1 (Keyframe A). You then skip to Frame 30 (Keyframe B) and move the box to the car's new position.
The Magic: The annotation software automatically calculates the movement for the 28 frames in between.
Pro Tip: If the car turns a corner, you must add a keyframe at the "peak" of the turn to ensure the box doesn't drift off the car.
2. Temporal Consistency
This is the most important rule in video. An object must keep the same Unique ID throughout the video.
If a man is labeled "Pedestrian_01" in the first second, he cannot become "Pedestrian_02" halfway through. If he does, the AI will think the first person disappeared and a new one was born. This is called a "Track ID Switch"and is a major error.
Part 3: Dealing with Occlusion in Video
What happens when a car drives behind a tree and comes out the other side?
Standard Rule: You keep the ID. You might use a "Hidden" or "Occluded" attribute while the car is behind the tree.
The Re-appearance: When the car emerges, you must link it back to the original ID. This teaches the AI Object Permanence (the understanding that things still exist even when you can't see them).
Part 4: Audio Transcription (ASR)
Audio annotation is about converting human speech into "Perfect Text" that a machine can study.
1. Verbatim vs. Clean Read
Verbatim: You transcribe exactly what is said, including "Ums," "Ahs," stutters, and false starts. (e.g., "I... I think, um, it's raining.")
Clean Read: You remove the filler words for better readability. (e.g., "I think it's raining.")
The Industry Standard: Most AI training requires Full Verbatim.
2. Speaker Diarization
This is the process of labeling who is speaking.
[Speaker 1]: "Hello, how are you?"
[Speaker 2]: "I am fine, thank you." If you fail to separate the speakers, the AI will get confused and think one person is having a conversation with themselves.
3. Noise Tagging
You must also label non-speech sounds that might confuse the AI:
[Background Noise]: Traffic, wind, or fans.
[Crosstalk]: When two people speak at the same time.
[Inaudible]: When the audio is too low to understand.
Part 5: Quality Metrics for Module 04
In video and audio, "Good enough" is not enough. Here is how your work is audited:
WER (Word Error Rate): For audio, this measures how many words you missed or misspelled compared to the "Ground Truth."
Jitter: In video, if your boxes "shake" or "jump" between frames, it creates "Jitter," which makes the AI's "vision" blurry.
Sync Accuracy: Does the text appear at the exact millisecond the person starts talking?
Part 6: Professional Tools for Chapter 04
VAT (Video Annotation Tool): Often built into CVAT.
Audacity: Used for cleaning audio before transcribing.
VTT/SRT Files: These are the file formats you will most likely export.
Chapter 04 Knowledge Check (Advanced)
Step 4 of 11 Modules Completed
Comments
Post a Comment