Skip to content

Serious performance issues in 12.2 (AWS S3) #351

@pySilver

Description

@pySilver

Hello,

There is a serious performance issue in current 12.2 branch. Problem is that we have ~2M thumbnails, so if execution goes here (marked line):

# /sorl/thumbnail/base.py:101 (get_thumbnail)
        # We have to check exists() because the Storage backend does not
        # overwrite in some implementations.
        if not thumbnail.exists(): # <---- This is the root of the problem!
            try:
                source_image = default.engine.get_image(source)
            except IOError:
                if settings.THUMBNAIL_DUMMY:
                    return DummyImageFile(geometry_string)
                else:
                    # if S3Storage says file doesn't exist remotely, don't try to
                    # create it and exit early.
                    # Will return working empty image type; 404'd image
                    logger.warn(text_type('Remote file [%s] at [%s] does not exist'),
                                file_, geometry_string)

                    return thumbnail

            # We might as well set the size since we have the image in memory
            image_info = default.engine.get_image_info(source_image)
            options['image_info'] = image_info
            size = default.engine.get_image_size(source_image)
            source.set_size(size)

            try:
                self._create_thumbnail(source_image, geometry_string, options,
                                       thumbnail)
                self._create_alternative_resolutions(source_image, geometry_string,
                                                     options, thumbnail.name)
            finally:
                default.engine.cleanup(source_image)

it is actually asks boto to return LIST of all stored thumbnails (without even using prefix), so appliction hangs with 100% CPU and high memory usage (well, not a surprise actually).

Wouldn't it be better to provide a prefix for lookup (constructed with the same function as used to store thumbnail) ?

In a mean time we've had to revert to 11.12.1b which works better.

This is related to a fix introduced in #92

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions