This PR introduces a major change to the LogStorage

• adds a new file logging storage that appends log entries to csv file(s)
  • file logging supports both a normalized and denormalized mode which results in a single or multiple files
• reworks log storage to make implementing new log storages (hopefully) a blast
  • 2 base classes can be inherited from to reuse shared log storage code
    • BufferingLogStorage (used by all 3 available log storages)
    • CSVLogStorage (used by file and stdout log storages)
• adds a mechanism for passing/parsing configuration parameters to log storages in an extensible way
• adds switched the enable_logging, disable_logging and truncate_duckdb_logs to be table functions and not pragmas
• changes the duckdb_logs view back to be a bind-replace function instead of a view: this means we can efficiently scan normalized log tables as well as denormalized log storages with the same function. (using a join if normalized and a simple scan if denormalized)

How to use the new enable_logging function

This PR moves towards using the enable_logging function as the primary way of configure logging. The simplest way to use the file log storage is:

CALL enable_logging();

This will enable logging in it's default config, which is currently the memory storage with log level INFO.

To use a different storage, one can use:

CALL enable_logging(storage='file', storage_path='./my_logs');

The following will make sure DuckDB sends all log entries at the default log level or higher (by default D_INFO) are written to two csv files ./my_logs/duckdb_log_entries.csv and ./my_logs/duckdb_log_contexts.csv which are written in the same schema as the duckdb_logs() and duckdb_log_contexts() table functions.

To query the logs, the same table functions duckdb_logs() and duckdb_log_contexts() can be used, meaning that the default view duckdb_logs works as well:

FROM duckdb_logs

The above query uses bind_replace to return a scan + join over the two csv files automatically.

More ways to enable logging

The enable_logging pragma has been expanded to take various parameters, including a struct field storage_config which is passed through to the log storage to configure it in an extensible way. Let's go over some examples:

-- log all log messages of level `trace` to stdout
CALL enable_logging(level='trace', storage='stdout');
-- log only log messages of the `FileSystem` type, to a single, denormalized file `some/log/file.csv`
CALL enable_logging('FileSystem', storage='file', storage_config={'path': 'some/log/file.csv'});
-- log only log messages of the `FileSystem` type, to two, normalized files in the path `some/log/file/path`. The buffer size of the log storage is 5000
CALL enable_logging('FileSystem', storage='file', storage_config={'path': 'some/log/file/path', 'buffer_size': 5000});
-- Same as last query, but leverage some syntactic sugar to automatically forward common settings path and buffer size. Also infers storage='file' from presence of path
CALL enable_logging('FileSystem', storage_path='some/log/file/path', storage_buffer_size=5000);
-- Configure a custom log storage implemented in some extension
CALL enable_logging(storage='my_log_storage', storage_config={'my_custom_options': 'some_option'});
-- Enable logging only for log type QueryLog and FileSystem
CALL enable_logging(['QueryLog', 'FileSystem']);

Note that in the last query we can see that some log storage config params are pulled up into the enable_logging function as named parameters with a storage_ prefix. These clean up the UX for some common options. Currently available are:

• storage_path
• storage_normalize
• storage_buffer_size

Another important thing to realize is that enable_logging will completely reset the logging config. So any manually specified logging related config (e.g. using the SET variables) will be lost once called. I feel this is a desirable trait though given the complexity of the number of states the logging mechanism can be in, allowing configuring the logging only once on enabling it keeps things sane.

Buffer sizes

As seen in section before, buffer sizes are configurable. The default values can differ between log storages:

• memory: STANDARD_VECTOR_SIZE
• stdout: 1 (flush on every write)
• file: STANDARD_VECTOR_SIZE

TODOs

There are some problems around deadlocking that need further attention / could use better testing.

For example, the code can easily deadlock when a LogStorage tries to grab a lock while flushing. We need to make sure that writing a log entry is guaranteed to not grab any locks that the code calling the logger might hold. There's a hotfix coming in to fix this in v1.3-ossivalis, which needs to be applied to this PR too where we can not use the buffermanager while flushing logs. Instead we should rely on the default allocator to avoid grabbing a log when flushing.