STJ Format Enhancement Proposal: Frame-Accurate Timing Support

[yaniv-golan]

STJ Format Enhancement Proposal: Frame-Accurate Timing Support

Proposal Version: 1.0
Target STJ Version: 0.7.0
Author: Yaniv Golan
Date: 2024-10-28
Status: Draft

Table of Contents

1. Executive Summary
2. Motivation
3. Proposed Changes
4. Specification Details
5. Validation Requirements
6. Timing Conflict Resolution
7. Migration Guide
8. Implementation Requirements
9. Testing Requirements
10. Security Considerations
11. Timeline
12. References

Executive Summary

This proposal introduces frame-accurate timing support to the Standard Transcription JSON (STJ) format version 0.7.0. The enhancement adds optional but first-class support for frame-based timing, SMPTE timecodes, and professional video workflows while maintaining format simplicity for basic transcription use cases.

Motivation

Current Limitations

1. Time-in-Seconds Limitations

   • Lacks frame-accurate timing precision
   • Cannot represent SMPTE timecodes natively
   • Difficult video editing integration
   • No drop-frame timecode support
   • Rounding issues at frame boundaries

2. Professional Workflow Gaps

   • No direct frame number references
   • Missing industry-standard timing formats
   • Limited broadcast compatibility
   • Insufficient for frame-accurate subtitle placement

Use Cases Requiring Frame Support

1. Professional Video Editing

   • Frame-accurate subtitle placement
   • Edit Decision List (EDL) integration
   • Cut point alignment
   • Frame-based trimming

2. Broadcast Applications

   • Caption synchronization
   • Commercial break timing
   • Live production timing
   • Drop-frame compensation

3. Quality Control

   • Frame-accurate validation
   • Standards compliance checking
   • Timing verification
   • Cross-format validation

4. Format Conversion

   • Accurate conversion between frame rates
   • SMPTE timecode preservation
   • Drop-frame handling
   • Legacy format compatibility

Proposed Changes

1. Schema Additions

{
  "stj": {
    "version": "0.7.0",
    "metadata": {
      "source": {
        "uri": "example.mp4",
        "duration": 3600.5,
        "frame_rate": {                    // New optional object
          "numerator": 30000,
          "denominator": 1001,             // 29.97 fps
          "drop_frame": true
        }
      }
    },
    "transcript": {
      "segments": [
        {
          "start": 3825.5,                 // Existing (mandatory)
          "end": 3826.0,                   // Existing (mandatory)
          "frames": {                      // New optional object
            "start": 114765,
            "end": 114780
          },
          "smpte": {                       // New optional object
            "start": "01:23:45:12",
            "end": "01:23:45:24"
          }
        }
      ]
    }
  }
}

2. New Field Definitions

Frame Rate Object

• numerator (integer, required): Frame rate numerator
• denominator (integer, required): Frame rate denominator
• drop_frame (boolean, optional): Drop-frame timecode flag

Frames Object

• start (integer, required): Starting frame number
• end (integer, required): Ending frame number

SMPTE Object

• start (string, required): Starting SMPTE timecode
• end (string, required): Ending SMPTE timecode

Specification Details

Frame Rate Support

1. Standard Frame Rates

   // 23.976 fps
   {
     "numerator": 24000,
     "denominator": 1001,
     "drop_frame": false
   }
   
   // 29.97 fps drop-frame
   {
     "numerator": 30000,
     "denominator": 1001,
     "drop_frame": true
   }
   
   // 25 fps (PAL)
   {
     "numerator": 25,
     "denominator": 1,
     "drop_frame": false
   }

2. Frame Number Format

   • Zero-based counting
   • Integer values
   • Monotonically increasing

3. SMPTE Format

   • Pattern: HH:MM:SS:FF
   • Hours: 00-23
   • Minutes: 00-59
   • Seconds: 00-59
   • Frames: 00-29 (for 30fps)

Validation Requirements

1. Frame Rate Validation

Rules

1. Basic Validation

   def validate_frame_rate(frame_rate):
       if frame_rate.numerator <= 0:
           raise ERROR("Frame rate numerator must be positive")
       if frame_rate.denominator <= 0:
           raise ERROR("Frame rate denominator must be positive")
           
       fps = frame_rate.numerator / frame_rate.denominator
       if fps < 1 or fps > 300:
           raise ERROR("Frame rate must be between 1 and 300 fps")

2. Drop Frame Validation

   def validate_drop_frame(frame_rate):
       is_ntsc = (
           frame_rate.numerator == 30000 and 
           frame_rate.denominator == 1001
       )
       if frame_rate.drop_frame and not is_ntsc:
           raise ERROR("Drop frame only allowed for NTSC rates")

2. Frame Number Validation

Rules

1. Basic Constraints

   def validate_frames(frames, duration, frame_rate):
       if frames.start < 0:
           raise ERROR("Start frame must be non-negative")
       if frames.end <= frames.start:
           raise ERROR("End frame must be greater than start frame")
           
       if duration:
           max_frames = ceil(duration * frame_rate)
           if frames.end >= max_frames:
               raise ERROR("Frame number exceeds duration")

2. Time Alignment

   def validate_frame_alignment(segment, frame_rate):
       fps = frame_rate.numerator / frame_rate.denominator
       start_time = segment.frames.start / fps
       end_time = segment.frames.end / fps
       
       tolerance = 0.5 * (1.0 / fps)
       
       if abs(start_time - segment.start) > tolerance:
           raise ERROR("Frame start time misaligned")
       if abs(end_time - segment.end) > tolerance:
           raise ERROR("Frame end time misaligned")

3. SMPTE Validation

Rules

1. Format Validation

   def validate_smpte(smpte, frame_rate):
       pattern = r'^([0-9][0-9]):([0-5][0-9]):([0-5][0-9]):([0-2][0-9])$'
       if not re.match(pattern, smpte):
           raise ERROR("Invalid SMPTE format")
           
       hours, minutes, seconds, frames = parse_smpte(smpte)
       max_frames = ceil(frame_rate) - 1
       
       if frames > max_frames:
           raise ERROR(f"Invalid frame number for frame rate")

2. Drop Frame Validation

   def validate_smpte_drop_frame(smpte, frame_rate):
       if frame_rate.drop_frame:
           if not is_valid_drop_frame_timecode(smpte):
               raise ERROR("Invalid drop frame timecode")

Timing Conflict Resolution

1. Precedence Rules

1. Time in Seconds (Authoritative)

   • Base timing source
   • All other representations must align within tolerance

2. Frame Numbers

   • Must align with seconds within ±0.5 frame duration
   • Used for frame-accurate positioning

3. SMPTE Timecode

   • Must align with seconds and frames
   • Used for display and interchange

2. Tolerance Calculation

def calculate_tolerance(frame_rate):
    fps = frame_rate.numerator / frame_rate.denominator
    return 0.5 * (1.0 / fps)

3. Conflict Detection

def detect_conflicts(segment, frame_rate):
    tolerance = calculate_tolerance(frame_rate)
    fps = frame_rate.numerator / frame_rate.denominator
    
    # Check frame alignment
    if segment.frames:
        frame_start = segment.frames.start / fps
        if abs(frame_start - segment.start) > tolerance:
            raise ERROR("Frame/time start conflict")
    
    # Check SMPTE alignment
    if segment.smpte:
        smpte_start = smpte_to_seconds(segment.smpte.start)
        if abs(smpte_start - segment.start) > tolerance:
            raise ERROR("SMPTE/time start conflict")

4. Resolution Strategy

1. On Conflict Detection

   • Use seconds as source of truth
   • Issue ERROR level validation message
   • Provide expected values
   • Detail correction requirements

2. Error Message Format

   ERROR: Timing conflict in segment
   - Base time: 10.500s
   - Frame time: 10.542s (deviation: +0.042s)
   - SMPTE time: 10.333s (deviation: -0.167s)
   Expected values:
   - Frames: 252 (actual: 253)
   - SMPTE: 00:00:10:12 (actual: 00:00:10:08)
   

Migration Guide

1. For STJ 0.6.0 Users

No Changes Required If:

• Not using frame-based timing
• Only using second-based timing
• No video workflow requirements

Updates Required If:

• Implementing frame support
• Using SMPTE timecodes
• Requiring frame accuracy

2. Implementation Steps

1. Basic Implementation

   # Minimal frame support
   def add_frame_support(segment, frame_rate):
       fps = frame_rate.numerator / frame_rate.denominator
       start_frame = round(segment.start * fps)
       end_frame = round(segment.end * fps)
       
       return {
           **segment,
           "frames": {
               "start": start_frame,
               "end": end_frame
           }
       }

2. Full Implementation

   # Complete frame support
   def add_full_frame_support(segment, frame_rate):
       fps = frame_rate.numerator / frame_rate.denominator
       start_frame = round(segment.start * fps)
       end_frame = round(segment.end * fps)
       
       smpte_start = frames_to_smpte(start_frame, frame_rate)
       smpte_end = frames_to_smpte(end_frame, frame_rate)
       
       return {
           **segment,
           "frames": {
               "start": start_frame,
               "end": end_frame
           },
           "smpte": {
               "start": smpte_start,
               "end": smpte_end
           }
       }

Implementation Requirements

1. Required Functions

# Frame rate conversions
def frames_to_seconds(frame_num, frame_rate):
    return frame_num * frame_rate.denominator / frame_rate.numerator

def seconds_to_frames(seconds, frame_rate):
    return round(seconds * frame_rate.numerator / frame_rate.denominator)

# SMPTE conversions
def smpte_to_frames(smpte, frame_rate):
    hours, minutes, seconds, frames = parse_smpte(smpte)
    total_seconds = hours * 3600 + minutes * 60 + seconds
    return seconds_to_frames(total_seconds, frame_rate) + frames

def frames_to_smpte(frame_num, frame_rate):
    seconds = frames_to_seconds(frame_num, frame_rate)
    hours = int(seconds // 3600)
    minutes = int((seconds % 3600) // 60)
    seconds = int(seconds % 60)
    frames = int(frame_num % ceil(frame_rate))
    return f"{hours:02d}:{minutes:02d}:{seconds:02d}:{frames:02d}"

2. Validation Implementation

class FrameValidator:
    def __init__(self, frame_rate):
        self.frame_rate = frame_rate
        self.tolerance = calculate_tolerance(frame_rate)
    
    def validate_segment(self, segment):
        self.validate_base_timing(segment)
        if segment.frames:
            self.validate_frames(segment)
        if segment.smpte:
            self.validate_smpte(segment)
        self.validate_alignment(segment)

Testing Requirements

1. Test Cases

def test_frame_rate_validation():
    # Valid cases
    assert validate_frame_rate({"numerator": 30000, "denominator": 1001})
    assert validate_frame_rate({"numerator": 24, "denominator": 1})
    
    # Invalid cases
    with pytest.raises(ValidationError):
        validate_frame_rate({"numerator": 0, "denominator": 1})
        validate_frame_rate({"numerator": 1, "denominator": 0})

def test_timing_alignment():
    # Within tolerance
    assert validate_alignment({
        "start": 1.0,
        "frames": {"start": 24},
        "frame_rate": {"numerator": 24, "denominator": 1}
    })
    
    # Outside tolerance
    with pytest.raises(ValidationError):
        validate_alignment({
            "start": 1.0,
            "frames": {"start": 25},
            "frame_rate": {"numerator": 24, "denominator": 1}
        })

Security Considerations

1. Input Validation

   • Frame number range checks
   • SMPTE format validation
   • Frame rate sanity checks

2. Resource Usage

   • Frame calculation limits
   • Validation complexity
   • Memory usage

3. Error Handling

   • Secure error messages
   • Resource cleanup
   • State recovery

References

1. SMPTE ST 12-1:2014 - Time and Control Code
2. SMPTE ST 12-2:2014 - Transmission of Time Code
3. EBU-TT Part 1 Tech 3350
4. W3C TTML Specification