About Llama-2 7B training

[sunnywyang]

Hello, contributors. When I was training the draft model of Llama-2 7B, I used 10,000 rounds of dialogue and did not use deepspeed for training. The loss at the beginning was more than 1400, and after four rounds, the loss was 1100. Is this normal?

[sunnywyang]

But the accuracy rate is still high, at 65 percent