Refactor/output stream #988
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
There was a problem hiding this comment.
Pull Request Overview
This PR refactors and modernizes the logging system by introducing a new structured LogModel, improving log proxying with start_proxy/reset APIs, and adding a thread‑safe AtomicCounter for epoch management. Key changes include:
- Replacing the deprecated install/uninstall methods with start_proxy and reset in swanlog.
- Integrating structured log handling via LogModel across log functions and API upload.
- Removing outdated classes and cleaning up tests to align with the new logging API.
Reviewed Changes
Copilot reviewed 19 out of 19 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| test/unit/test_swanlab_settings.py | Added tests to verify the new log_proxy_type setting in Settings. |
| test/unit/test_package.py | Inserted a sleep call to allow log proxy operations to stabilize. |
| test/unit/log/test_log.py | Updated tests to use start_proxy/stop_proxy as part of the new logging proxy mechanism. |
| swanlab/swanlab_settings.py | Incorporated Literal typing for the new log_proxy_type setting. |
| swanlab/log/log.py | Refactored log proxy installation and deinstallation logic to use start_proxy and reset. |
| swanlab/log/counter.py | Added a thread‑safe AtomicCounter for managing the log epoch. |
| swanlab/log/init.py | Updated public API to expose start_proxy and reset instead of install and uninstall. |
| swanlab/api/upload/model.py | Introduced LogModel for structured upload of log data. |
| swanlab/api/upload/init.py | Updated upload_logs to process logs using LogModel and flatten its contents. |
| Other files | Adjusted usage of the logging API (e.g. in cloud and local callbacks) accordingly. |
Zeyi-Lin
approved these changes
May 22, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request introduces significant updates to the logging system in SwanLab, including a new
LogModelfor structured log handling, improved log proxying mechanisms, and the removal of outdated code. Additionally, a thread-safeAtomicCounterclass has been added for managing epochs. Below is a summary of the most important changes grouped by theme.Logging System Enhancements:
LogModelinswanlab/api/upload/model.pyfor structured log data, including log levels (INFO,WARN,ERROR) and associated contents. This model is now used across the logging system ([swanlab/api/upload/model.pyR233-R247](https://github.com/SwanHubX/SwanLab/pull/988/files#diff-2c4391382afbb16619bb2524a64d44751f6eea5c9ddc344038b95759a331f8f9R233-R247)).upload_logsinswanlab/api/upload/__init__.pyto useLogModelinstead of raw dictionaries, simplifying log handling and enabling structured uploads ([swanlab/api/upload/__init__.pyL29-R39](https://github.com/SwanHubX/SwanLab/pull/988/files#diff-7eef39be6eb534f3b784e5d8b8a0ea5882cc017cdc756d4c3ccfb01dde50ec9fL29-R39)).install/uninstallmethods inswanlab/log/__init__.pywithstart_proxyandreset, aligning with the new log proxying mechanism ([swanlab/log/__init__.pyL11-R20](https://github.com/SwanHubX/SwanLab/pull/988/files#diff-8fbab3d8532f931b9a25498b7964119f52ba18575f082522a36aee6fb32045c0L11-R20)).swanlab/data/callbacker/cloud.pyby introducing_write_handlerfor log proxying and updatingon_runto use the newstart_proxymethod ([[1]](https://github.com/SwanHubX/SwanLab/pull/988/files#diff-733d6742a0ed52a329f3872a2f17de48637ce80a1af7f2a7a41a975d1d858972R103-R112),[[2]](https://github.com/SwanHubX/SwanLab/pull/988/files#diff-733d6742a0ed52a329f3872a2f17de48637ce80a1af7f2a7a41a975d1d858972L126-R144)).swanlab/data/callbacker/local.pyby adding_write_handlerfor file-based log writing and integrating it with the new proxying system ([[1]](https://github.com/SwanHubX/SwanLab/pull/988/files#diff-7044b26107bc0f6a35bff2f331d0f3a685dfc7959626efb71136147af28c0798R135-R157),[[2]](https://github.com/SwanHubX/SwanLab/pull/988/files#diff-7044b26107bc0f6a35bff2f331d0f3a685dfc7959626efb71136147af28c0798L148-R173)).Code Simplification and Cleanup:
SwanWriterProxyandSwanConsolerclasses fromswanlab/log/console.py, as their functionality has been replaced by the new proxying system ([swanlab/log/console.pyL1-L147](https://github.com/SwanHubX/SwanLab/pull/988/files#diff-470e329fc59b8300dd468331d87fbb3ec54ce8e0657b1b826d72c6b6132cd652L1-L147))._valid_filesinswanlab/api/upload/__init__.py, as it is no longer relevant to the current implementation ([swanlab/api/upload/__init__.pyL67-L76](https://github.com/SwanHubX/SwanLab/pull/988/files#diff-7eef39be6eb534f3b784e5d8b8a0ea5882cc017cdc756d4c3ccfb01dde50ec9fL67-L76)).New Utility:
AtomicCounterclass inswanlab/log/counter.pyto manage epoch counters safely within a multi-threaded environment ([swanlab/log/counter.pyR1-R50](https://github.com/SwanHubX/SwanLab/pull/988/files#diff-965e96240a8380e1dbbc95f94adcf2401309ee849ff4c583d68bdb748a08ed1eR1-R50)).Minor Improvements:
LogModelandLiteralfor type annotations, ensuring consistency and clarity ([[1]](https://github.com/SwanHubX/SwanLab/pull/988/files#diff-7eef39be6eb534f3b784e5d8b8a0ea5882cc017cdc756d4c3ccfb01dde50ec9fL13-R13),[[2]](https://github.com/SwanHubX/SwanLab/pull/988/files#diff-2c4391382afbb16619bb2524a64d44751f6eea5c9ddc344038b95759a331f8f9L12-R12),[[3]](https://github.com/SwanHubX/SwanLab/pull/988/files#diff-733d6742a0ed52a329f3872a2f17de48637ce80a1af7f2a7a41a975d1d858972R11),[[4]](https://github.com/SwanHubX/SwanLab/pull/988/files#diff-733d6742a0ed52a329f3872a2f17de48637ce80a1af7f2a7a41a975d1d858972L19-R20)).swanlog.uninstallwithswanlog.resetinswanlab/data/run/main.pyto align with the new logging API ([swanlab/data/run/main.pyL305-R305](https://github.com/SwanHubX/SwanLab/pull/988/files#diff-716884b2803891f14f0e31a96ce3c380a52b38283304a509d29b4caa9d9abceaL305-R305)).These changes collectively modernize the logging system, improve maintainability, and introduce safer concurrency practices.
closes #937