Adding a chat template only makes sense when you SFT the model (instruction tuning). Doing it before DPO, CPO etc, doesn't make sense, so I'd remove them all.

If you use Excel 40 hours a week (and those are the weeks you are on vacation), welcome to the MrExcel channel. Home to 2,400 free Excel tutorials. Bill "MrExcel" Jelen is the author of 67 books about ...