Skip to content

fix: Windows 系统下，与 accelerate 一起使用时，local 模式因默认 gkb 编码导致报错 #1195

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Jul 18, 2025
Merged

fix: Windows 系统下，与 accelerate 一起使用时，local 模式因默认 gkb 编码导致报错 #1195

merged 1 commit into from
Jul 18, 2025

Conversation

Notte118
Copy link
Contributor

问题背景

在使用最新版的 accelerate==1.9.0 与 swanlab[dashboard]==0.6.7 进行本地实验记录时（mode="local"），我在 Windows 平台 下遇到了编码相关的崩溃。具体来说：

  • 项目运行到 swanlab.config["FRAMEWORK"] = "🤗Accelerate" 时触发异常
  • 根因是 swanlab 内部在读取 YAML/JSON 文件时未指定编码，Windows 默认使用 gbk，无法解析其中包含的 emoji 字符（如 🤗）

复现方式

测试环境：

  • OS: Windows 11
  • Python 版本: 3.13.5
  • accelerate==1.9.0
  • swanlab==0.6.7（含 dashboard）

复现代码（仅在官方示例基础上添加 mode="local"）：

from accelerate import Accelerator

accelerator = Accelerator(log_with="swanlab")
accelerator.init_trackers(
    project_name="accelerator",
    config={"dropout": 0.1, "learning_rate": 1e-2},
    init_kwargs={"swanlab": {"experiment_name": "hello_world", "mode": "local"}}
)

for i in range(100):
    accelerator.log({"train_loss": 1.12, "valid_loss": 0.8}, step=i + 1)

accelerator.end_training()

报错信息

C:\Users\wangm\anaconda3\envs\swanlab\python.exe N:\swanlab\swanlab_fix\main.py 
swanlab: Tracking run with swanlab version 0.6.7
swanlab: Run data will be saved locally in 
N:\swanlab\swanlab_fix\swanlog\run-20250718_174443-be5ml5naxc2h1z8xeawyg
swanlab: 🌟 Run `swanlab watch N:\swanlab\swanlab_fix\swanlog` to view SwanLab 
Experiment Dashboard locally
swanlab: Error happened while training
Error in sys.excepthook:
Traceback (most recent call last):
  File "C:\Users\wangm\anaconda3\envs\swanlab\Lib\site-packages\swanlab\data\callbacker\callback.py", line 81, in _except_handler
    get_run().finish(SwanLabRunState.CRASHED, error=error)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\wangm\anaconda3\envs\swanlab\Lib\site-packages\swanlab\data\run\main.py", line 237, in finish
    getattr(run, "_SwanLabRun__cleanup")(error)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
  File "C:\Users\wangm\anaconda3\envs\swanlab\Lib\site-packages\swanlab\data\run\main.py", line 139, in __cleanup
    self.__operator.on_stop(error)
    ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
  File "C:\Users\wangm\anaconda3\envs\swanlab\Lib\site-packages\swanlab\data\run\helper.py", line 121, in on_stop
    r = self.__run_all("on_stop", error=error, epoch=epoch, *args, **kwargs)
  File "C:\Users\wangm\anaconda3\envs\swanlab\Lib\site-packages\swanlab\data\run\helper.py", line 57, in __run_all
    return {name: getattr(callback, method)(*args, **kwargs) for name, callback in self.callbacks.items()}
                  ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "C:\Users\wangm\anaconda3\envs\swanlab\Lib\site-packages\swanlab\data\callbacker\local.py", line 147, in on_stop
    print(error, file=fError)
    ~~~~~^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'gbk' codec can't encode character '\U0001f917' in position 1237: illegal multibyte sequence

Original exception was:
Traceback (most recent call last):
  File "N:\swanlab\swanlab_fix\main.py", line 7, in <module>
    accelerator.init_trackers(
    ~~~~~~~~~~~~~~~~~~~~~~~~~^
        project_name="accelerator",
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
        config={"dropout": 0.1, "learning_rate": 1e-2},
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        init_kwargs={"swanlab": {"experiment_name": "hello_world", "mode":"local"}}
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        )
        ^
  File "C:\Users\wangm\anaconda3\envs\swanlab\Lib\site-packages\accelerate\accelerator.py", line 807, in _inner
    return PartialState().on_main_process(function)(*args, **kwargs)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "C:\Users\wangm\anaconda3\envs\swanlab\Lib\site-packages\accelerate\accelerator.py", line 3043, in init_trackers
    tracker.start()
    ~~~~~~~~~~~~~^^
  File "C:\Users\wangm\anaconda3\envs\swanlab\Lib\site-packages\accelerate\tracking.py", line 91, in execute_on_main_process
    return function(self, *args, **kwargs)
  File "C:\Users\wangm\anaconda3\envs\swanlab\Lib\site-packages\accelerate\tracking.py", line 1183, in start
    swanlab.config["FRAMEWORK"] = "🤗Accelerate"  # add accelerate logo in config
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "C:\Users\wangm\anaconda3\envs\swanlab\Lib\site-packages\swanlab\data\run\config.py", line 288, in __setitem__
    self.__save()
    ~~~~~~~~~~~^^
  File "C:\Users\wangm\anaconda3\envs\swanlab\Lib\site-packages\swanlab\data\run\config.py", line 204, in __save
    self.__on_setter(r)
    ~~~~~~~~~~~~~~~~^^^
  File "C:\Users\wangm\anaconda3\envs\swanlab\Lib\site-packages\swanlab\data\run\helper.py", line 109, in on_runtime_info_update
    return self.__run_all("on_runtime_info_update", r, *args, **kwargs)
           ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\wangm\anaconda3\envs\swanlab\Lib\site-packages\swanlab\data\run\helper.py", line 57, in __run_all
    return {name: getattr(callback, method)(*args, **kwargs) for name, callback in self.callbacks.items()}
                  ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "C:\Users\wangm\anaconda3\envs\swanlab\Lib\site-packages\swanlab\data\callbacker\local.py", line 108, in on_runtime_info_update
    self.porter.trace_runtime_info(r)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^
  File "C:\Users\wangm\anaconda3\envs\swanlab\Lib\site-packages\swanlab\data\porter\__init__.py", line 89, in wrapper
    return wrapped(*args, **kwargs)
  File "C:\Users\wangm\anaconda3\envs\swanlab\Lib\site-packages\swanlab\data\porter\__init__.py", line 62, in wrapper
    result: Union[List[BaseModel], BaseModel] = wrapped(*args, **kwargs)
                                                ~~~~~~~^^^^^^^^^^^^^^^^^
  File "C:\Users\wangm\anaconda3\envs\swanlab\Lib\site-packages\swanlab\data\porter\__init__.py", line 36, in wrapper
    return wrapped(*args, **kwargs)
  File "C:\Users\wangm\anaconda3\envs\swanlab\Lib\site-packages\swanlab\data\porter\__init__.py", line 289, in trace_runtime_info
    self._publish((UploadType.FILE, [runtime.to_file_model(file_dir)]))
                                     ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "C:\Users\wangm\anaconda3\envs\swanlab\Lib\site-packages\swanlab\proto\v0.py", line 154, in to_file_model
    config = yaml.safe_load(open(config_path, "r").read())
                            ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
UnicodeDecodeError: 'gbk' codec can't decode byte 0xa4 in position 46: illegal multibyte sequence
swanlab: Experiment hello_world has completed

进程已结束，退出代码为 1

修复方案

在文件 swanlab/proto/v0.py 中的 to_file_model() 方法内，所有涉及读取文件（如 JSON、YAML）的位置（139-154 行），添加：

open(path, "r", encoding="utf-8")

以显式指定 UTF-8 编码，避免依赖平台默认设置。

补充说明

这个修改理论上侵入性极小，应该不会引入其他bug，望采纳

@Zeyi-Lin Zeyi-Lin requested a review from SAKURA-CAT July 18, 2025 13:05
@Zeyi-Lin Zeyi-Lin added the 🐛 bug Something isn't working label Jul 18, 2025
@SAKURA-CAT
Copy link
Member

LGTM.

由于我们一般都用 mac/linux ，所以对 Windows 的测试忽略的比较多，后续我们也会逐渐加强 windows 相关单元测试。BTW，未来会增加 protobuf 支持，能一定程度避免类似编码问题再现

@SAKURA-CAT SAKURA-CAT merged commit ea588e4 into SwanHubX:main Jul 18, 2025
5 checks passed
@Zeyi-Lin
Copy link
Member

Thank you very much for your contribution to the SwanLab community ❤️ @Notte118

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@SAKURA-CAT SAKURA-CAT SAKURA-CAT approved these changes

Assignees

@Notte118 Notte118

Labels
🐛 bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@Notte118 @SAKURA-CAT @Zeyi-Lin