-
Notifications
You must be signed in to change notification settings - Fork 0
Quality Metrics
Our framework currently employs a subjective quality assessment methodology, focusing on human evaluation of generated outputs. While we recognize the limitations of subjective analysis, this approach allows us to capture nuanced aspects of image quality that automated metrics often miss. Key evaluation areas include fine detail preservation, color accuracy, compositional coherence, and artifact presence. Evaluators examine multiple samples per prompt to assess generation consistency and stability across different conditions.
We are actively developing a comprehensive automated metrics system to complement our subjective assessments. This system will incorporate perceptual metrics, CLIP-based semantic evaluation, and specialized artifact detection. Our goal is to establish quantitative benchmarks that align with human quality assessment while providing reproducible measurements for tracking improvements. Future metrics will focus particularly on areas where current automated systems struggle, such as aesthetic quality and semantic nuance.
Training runs are currently evaluated through regular quality review sessions, with results documented and tracked to guide development. As automated metrics are developed and validated, they will be integrated into the training pipeline for real-time quality monitoring. This hybrid approach will maintain the benefits of human assessment while adding the consistency and scalability of automated evaluation.
See Training Pipeline for details on how quality assessment integrates with the training process.