The three scores for the same prompt in the same video are very unstable.

[Jingwei-Bao]

Video Path: /videos/badcase_14_1.mp4
Visual Quality: 4.0
Text-to-Video Alignment: 4.0
Physical Consistency: 4.0

Video Path: /videos/badcase_14_1.mp4
Visual Quality: 3.0
Text-to-Video Alignment: 4.0
Physical Consistency: 1.0

Video Path:/videos/badcase_14_1.mp4
Visual Quality: 3.0
Text-to-Video Alignment: 4.0
Physical Consistency: 2.9771

I used the same prompt for the same video and evaluated it using the instructions from GitHub. Each time, I got completely different results.

[totalhuang]

I also observed this phenomenon, and I further found that the inference scores are usually integers, and the model typically has high confidence in the logits for those scores.

[totalhuang]

The reason for the unstable scores might be that the temperature was set to $0.7$ instead of $0$. Of course, if greedy sampling were used, I don't know what kind of impact that would have on the scores.

[Jingwei-Bao]

Due to the algorithm principle, you cannot set the temperature coefficient to 0. However, I have tried setting it to 0.1 and still encountered this unstable phenomenon.