𝗥𝗲𝘀𝗼𝗹𝘃𝗶𝗻𝗴 𝗘𝗻𝗰𝗼𝗱𝗶𝗻𝗴 𝗘𝗿𝗿𝗼𝗿𝘀 𝗶𝗻 𝗟𝗼𝗰𝗮𝗹 𝗟𝗟𝗠 𝗕𝗲𝗻𝗰𝗵𝗺𝗮𝗿𝗸𝗶𝗻𝗴
I wanted to build a tool to switch and benchmark local LLM models. I also added many Korean benchmark questions to test them better.
Everything worked fine until I ran the benchmarks. I kept seeing this error: UnicodeEncodeError: 'cp949' codec can't encode characters.
I thought the problem was in my Python code. I tried changing my encoding settings to utf-8. I tried decoding strings manually. Nothing worked for hours.
The real problem was not my script. The local LLM worker was trying to save model responses using the system default encoding. On Windows, this is often CP949. When the worker tried to save Korean characters using CP949, it failed.
The fix was simple. I changed the worker to use utf-8 explicitly when saving files.
I also built an automated system to manage this process. It follows these steps:
- Downloads candidate models.
- Runs benchmarks on the current best model.
- Tests all new candidate models.
- Selects and recommends the best model based on scores.
Through this automation, I found that Gemma2:2b performs much better than the EXAONE model for Korean language tasks. It provides more natural answers and better creative results at a similar speed.
Lessons learned:
- Never rely on default system encoding. Always use utf-8 for file I/O.
- When using json.dump with Korean text, use ensure_ascii=False and specify utf-8.
- If you see encoding errors, check the entire system and file saving logic, not just your immediate code.
- Use automation to find the best models for your specific needs.
Optional learning community: https://t.me/GyaanSetuAi