Skip to content

Onnx format conversion to kmodel format bug #1193

@meimei-123

Description

@meimei-123

Describe the bug
In my execution:
compiler.import_onnx(model_content, import_options)
Encountered an error:
Process terminated. Assertion Failed
3d4e51ffe6608ca492ec5955e0b7f8b1

So I checked the model structure and annotated it line by line from back to front
Once x=self.fc6 (x) is commented out, it can be run. self.fc6=nn Linear (3072, 192), in ONNX format, is a Gemm operator.
I checked the list of onnx operators supported by nncase and found that there is Gemm.

To Reproduce
Command line or scripts to reproduce the behavior:

Expected behavior
I want to convert the pytorch model into a kmodel model, with the following model structure:

Origin model and code
model:

class Mymodel(nn.Module):
    def __init__(self, C=1024, m=0.2, s=30, n_class=2):
        super(Mymodel, self).__init__()

        self.conv1 = nn.Conv1d(80, C, kernel_size=5, stride=1, padding=2)
        self.relu = nn.ReLU()
        self.bn1 = nn.BatchNorm1d(C)
        self.layer1 = Bottle2neck(C, C, kernel_size=3, dilation=2, scale=8)
        self.layer2 = Bottle2neck(C, C, kernel_size=3, dilation=3, scale=8)
        self.layer3 = Bottle2neck(C, C, kernel_size=3, dilation=4, scale=8)
        # I fixed the shape of the output from MFA layer, that is close to the setting from ECAPA paper.
        self.layer4 = nn.Conv1d(3 * C, 1536, kernel_size=1)
        self.attention = nn.Sequential(
            nn.Conv1d(4608, 256, kernel_size=1),
            nn.ReLU(),
            nn.BatchNorm1d(256),
            nn.Tanh(),  # I add this layer
            nn.Conv1d(256, 1536, kernel_size=1),
            nn.Softmax(dim=2),
        )
        self.bn5 = nn.BatchNorm1d(3072)
        self.fc6 = nn.Linear(3072, 192)
        self.bn6 = nn.BatchNorm1d(192)

    def forward(self, x):
        x = self.conv1(x)
        x = self.relu(x)
        x = self.bn1(x)

        x1 = self.layer1(x)
        x2 = self.layer2(x + x1)
        x3 = self.layer3(x + x1 + x2)

        x = self.layer4(torch.cat((x1, x2, x3), dim=1))
        x = self.relu(x)

        t = x.size()[-1]

        global_x = torch.cat((x, torch.mean(x, dim=2, keepdim=True).repeat(1, 1, t),
                              torch.sqrt(torch.var(x, dim=2, keepdim=True).clamp(min=1e-4)).repeat(1, 1, t)), dim=1)

        w = self.attention(global_x)

        mu = torch.sum(x * w, dim=2)
        sg = torch.sqrt((torch.sum((x ** 2) * w, dim=2) - mu ** 2).clamp(min=1e-4))

        x = torch.cat((mu, sg), 1)
        x = self.bn5(x)
        x = self.fc6(x) # This is where the error occurred,after I annotate the code here and after, it will work fine
        x = self.bn6(x)
        return x

code:

    input_shape = [5, 80, 626] # The input data is a mel spectrogram converted from wav    
    dump_dir = 'tmp/export_onnx'
    if not os.path.exists(dump_dir):
        os.makedirs(dump_dir)

    # onnx simplify
    model_file = onnx_simplify(args.model, dump_dir)

    # compile_options
    compile_options = nncase.CompileOptions()
    compile_options.target = args.target
    compile_options.preprocess = False # 
    compile_options.swapRB = False
    compile_options.dump_ir = True # 是否输出ir
    compile_options.dump_asm = True # 是否输出汇编
    compile_options.dump_dir = dump_dir # 输出目录

    # compiler
    compiler = nncase.Compiler(compile_options)

    # import
    model_content = read_model_file(model_file)
    import_options = nncase.ImportOptions()
    compiler.import_onnx(model_content, import_options)

    # ptq_options
    ptq_options = nncase.PTQTensorOptions()
    ptq_options.samples_count = 6
    ptq_options.set_tensor_data(generate_data(input_shape, ptq_options.samples_count, args.dataset))
    compiler.use_ptq(ptq_options)

Environment (please complete the following information):

  • OS: [Win11 WSL2 Ubuntu20.04]
  • nncase version :nncase 2.8.1, nncase kpu 2.8.1
  • DL Framework :pytorch 2.0.1
  • other: Python 3.8.0, nncase 2.8.1, nncase kpu 2.8.1, numpy=1.19.5, onnx1.9.0, onnx simplier 0.3.6

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions