Skip to content

DjangoUnicodeDecodeError when trying to write binary field #665

@ya-pekatoros

Description

@ya-pekatoros

Hi there!

I've faced interesting issue, when using Silk in Django projects, using pickle (https://docs.python.org/3/library/pickle.html) and also trying to save request.body as binary data in DB (MySQL) - BinaryField in Django

During tests I've face these:

Traceback (most recent call last):
  File "/venv/lib/python3.10/site-packages/django/utils/encoding.py", line 70, in force_str
    s = str(s, encoding, errors)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  --- PROJECT MODULES AND FUNCTIONS ---
    resp = self.client.post(
  File "/venv/lib/python3.10/site-packages/django/test/client.py", line 852, in post
    response = super().post(
  File "/venv/lib/python3.10/site-packages/django/test/client.py", line 441, in post
    return self.generic(
  File "/venv/lib/python3.10/site-packages/django/test/client.py", line 541, in generic
    return self.request(**r)
  File "/venv/lib/python3.10/site-packages/django/test/client.py", line 810, in request
    self.check_exception(response)
  File "/venv/lib/python3.10/site-packages/django/test/client.py", line 663, in check_exception
    raise exc_value
  File "/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "/venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/venv/lib/python3.10/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/venv/lib/python3.10/site-packages/django/views/generic/base.py", line 103, in view
    return self.dispatch(request, *args, **kwargs)
  File "/venv/lib/python3.10/site-packages/rest_framework/views.py", line 509, in dispatch
    response = self.handle_exception(exc)
  File "/venv/lib/python3.10/site-packages/rest_framework/views.py", line 469, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/venv/lib/python3.10/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
    raise exc
  File "/venv/lib/python3.10/site-packages/rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
  File "venv/lib/python3.10/site-packages/rest_framework/decorators.py", line 50, in handler
    return func(*args, **kwargs)
  File "/django/response.py", line 289, in wrapper
    return func(request, *args, **kwargs)
 --- PROJECT MODULES AND FUNCTIONS ---
  File "/venv/lib/python3.10/site-packages/django/db/models/base.py", line 812, in save
    self.save_base(
  File "/venv/lib/python3.10/site-packages/django/db/models/base.py", line 863, in save_base
    updated = self._save_table(
  File "/venv/lib/python3.10/site-packages/django/db/models/base.py", line 976, in _save_table
    updated = self._do_update(
  File "/venv/lib/python3.10/site-packages/django/db/models/base.py", line 1040, in _do_update
    return filtered._update(values) > 0
  File "/venv/lib/python3.10/site-packages/django/db/models/query.py", line 1215, in _update
    return query.get_compiler(self.db).execute_sql(CURSOR)
  File "/venv/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1822, in execute_sql
    cursor = super().execute_sql(result_type)
  File "/venv/lib/python3.10/site-packages/silk/sql.py", line 80, in execute_sql
    sql_query = q % tuple(force_str(param) for param in params)
  File "/venv/lib/python3.10/site-packages/silk/sql.py", line 80, in <genexpr>
    sql_query = q % tuple(force_str(param) for param in params)
  File "/venv/lib/python3.10/site-packages/django/utils/encoding.py", line 74, in force_str
    raise DjangoUnicodeDecodeError(s, *e.args)
django.utils.encoding.DjangoUnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte. You passed in b'\x80\x04\x95\x98\x01\x00\x00\x00\x00\x00\x00\x8c\x10requests.cookies\x94\x8c\x11RequestsCookieJar\x94\x93\x94)\x81\x94}\x94(\x8c\x07_policy\x94\x8c\x0ehttp.cookiejar\x94\x8c\x13DefaultCookiePolicy\x94\x93\x94)\x81\x94}\x94(\x8c\x08netscape\x94\x88\x8c\x07rfc2965\x94\x89\x8c\x13rfc2109_as_netscape\x94N\x8c\x0chide_cookie2\x94\x89\x8c\rstrict_domain\x94\x89\x8c\x1bstrict_rfc2965_unverifiable\x94\x88\x8c\x16strict_ns_unverifiable\x94\x89\x8c\x10strict_ns_domain\x94K\x00\x8c\x1cstrict_ns_set_initial_dollar\x94\x89\x8c\x12strict_ns_set_path\x94\x89\x8c\x10secure_protocols\x94\x8c\x05https\x94\x8c\x03wss\x94\x86\x94\x8c\x10_blocked_domains\x94)\x8c\x10_allowed_domains\x94Nub\x8c\x08_cookies\x94}\x94ub.' (<class 'bytes'>)

So Silk is trying to decode any content in SQL queries, how to switch off these for binary data?

I know about SilkyIgnorePaths, but its not very reliable to find all paths with binary data recording. I think it will be better to somehow wrap around bad decoding tries.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions