Skip to content

DynamicQuantizeLinear - Documentation example and reference mismatch #5597

@cjvolzka

Description

@cjvolzka

Bug Report

Is the issue related to model conversion?

No

Describe the bug

The implementation for y_scale for DynamicQuantizeLinear differs in two different locations

The documentation has:

y_scale = (max(x) - min(x))/(qmax - qmin)

while the reference shows

        maxx = np.float32(np.maximum(0, np.max(x)))
        minx = np.float32(np.minimum(0, np.min(x)))
        y_scale = np.float32(1.0 if maxx == minx else (maxx - minx)) / np.float32(
            qmax - qmin
        )

The reference appears to be based on https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/quantized/cpu/QuantUtils.h#L95-L99 which itself appears to be based on https://github.com/pytorch/FBGEMM/blob/main/include/fbgemm/QuantUtils.h

Assuming the references are correct, the documentation should be:

y_scale = (maximum(0,max(x)) - minimum(0,min(x)))/(qmax - qmin)

Side note: The documentation does have a bullet data range is adjusted to include 0. which alludes to this. However it is easy to miss and exactly how it affects y_scale code isn't obvious so it'd be better to include it in the code snippet.

System information

NA

Reproduction instructions

Expected behavior

Implementation of documentation example snippets leads to output that matches output of reference example

Notes

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions