Problem

Because, currently, logs are stored as base64-encoded bytes, it's extremely hard to:

1. Show logs along with their timestamps.
2. [MAJOR] Implement descending pagination
3. Allow to search for logs in a given time range
4. [MAJOR] Query logs, esp. when they are stored in an external system such as AWS CloudWatch or GCP Logging
5. Support other external log storage systems (via a OpenTelemetry reporting interface)

Underlying problem

1. Currently, containers run in the TTY mode. This allows to stream interactive logs via a websocket. Interactive logs are important to show interactive output from pip, uv and other similar tools.
2. At the sametime, to store also these logs on the server. Because TTY logs are in bytes (this is the only way they can be shown in the terminal once received via a websocket), we encode them via base64.
3. As the result, a) logs are encoded in base64 and not searchable; b) logs can be displayed in UI /CLI only if appended from the very beginning (due to ANSI codes). This means we cannot load these via pagination.

Solution

In order to solve this, we need to automatically strip ANSI codes from the logs and convert bytes to plain text. Then, this text can be stored in any storage and then a) it can be searchable; b) it can be loaded via pagination starting from the most recent.

Note, stripping ANSI codes is a non-trivial task as it requires emulating support for all ANSI codes and buffering the logs.

Here's the initial implementation of this algorithm that needs to be tested: http://github.com/dstackai/ansistrip

TODO:

• Update the runner to use ansistrip and write the logs without encoding.
  • Initial algorythm http://github.com/dstackai/ansistrip
  • Tune timeout settings for optimal experience and performance
  • Test extensively on various workloads
• Update the File/GCP/CloudWatch logging services to properly support pagination and descending
  • Initial support with File/GCP/CloudWatch logging services
  • Support descending
• Update the dstack logs CLI and UI to support pagination and descending
  • Initial update of CLI/UI
  • CLI: Support displaying timestamps, plus --start
  • UI: Support displaying timestamps, and descending