Amazing work @abhinavkulkarni. The original creators of AWQ recently removed all support of the GEMM kernel, however, after extensive testing of the GEMV kernel, I believe the GEMM kernel is the best for the highest throughput (large context, large batch size) as the GEMV kernel struggles with performance on anything but empty context and batch size > 1.

I am the author of AutoAWQ and will maintain backward compatibility with GEMM as the main packing method and GEMV as an optional packing method (about to be merged). You can either use the old commit or switch to AutoAWQ, which has a PyPi package that is easily installable:

pip install autoawq

Hey thanks a LOT for this PR.

Before diving into the details, I'll drop some benchmark I did on single A100:

f16:

awq

gptq

These tell a very good story.

Baseline latency for AWQ is 15ms/per, 25ms/token for f16 and 20ms/token for exllama.
But more importantly, the baseline holds much better for AWQ at higher throughput, giving more throughput at constant latency.

F16 seems to be managing to reach higher throughput but this regime is not the main one for usual TGI deployments, where latency is the key.

I will probably make a bit more testing on different TP sharding layout and model sizes but this looks really good.

Ok, my comments are mostly nits about code maintenance, but this PR looks great !

Thanks for it !

@Narsil @abhinavkulkarni Has this PR been tested to be compatible with the 300+ AWQ repositories that @TheBloke has pushed with AutoAWQ?

• Most notably, I want to ensure that the models can be loaded in TGI with the quant_config.json that is provided? Example: https://huggingface.co/TheBloke/Llama-2-13B-AWQ
• Is this PR compatible with Windows? The original AWQ GEMM kernels are not compatible with Windows - I modified them to support Windows in AutoAWQ.

@TheBloke: Can we get some clarity on naming the quantization config for GPTQ and AWQ?

You seem to name it as quantize_config.json for GPTQ and quant_config.json for AWQ.

TGI code expects it to be named quantize_config.json.

Is it possible for you to rename it to quantize_config.json across all your AWQ repos? Or create a symlink in these repos so that both the files are present?

I would advise against renaming to quantize_config.json as that makes all his repositories incompatible with AutoAWQ, which was used to create the 300+ repositories.

EDIT: It will also break all other repositories that have implemented AutoAWQ.

Yeah that's because AutoGPTQ and AutoAWQ create that file with a different name. AutoGPTQ creates it as quantize_config.json, but AutoAWQ creats it as quant_config.json.

I'm happy to rename it across all, but it will require AutoAWQ updating to support that name (not quite sure why a different name was used by AutoAWQ)

Okay, @TheBloke, @casper-hansen and @Narsil, please agree on one standard.

I'm happy to comply with whatever is decided. Two issues we need consensus on:

1. Wether to have quantize_config.json for GPTQ and quant_config.json for AWQ or just one for both

2. Secondly, uniformity in how we refer to quantization parameters - for e.g. in GPTQ quantize_config.json, we use bits but in quant_config.json of AWQ, we use w_bit. Same for groupsize for GPTQ vs group_size for AWQ.

Please let me know what you agree on and I'll modify the PR accordingly.

@abhinavkulkarni GPTQ doesn't use groupsize, it's group_size there too.

IMHO the best option is for AutoAWQ to update to use quantize_config.json, and match the param names of AutoGPTQ. Ultimately the hope is that AutoAWQ will be integrated into Transformers, like AutoGPTQ was. If/when that happens, I imagine the Transformers team will ask for the same param names to be used - though they won't care about the filename because the data will need to go into config.json

Here is the params that Transformers expects in config.json for a GPTQ model:

    "quantization_config": {
        "bits": 4,
        "group_size": 128,
        "damp_percent": 0.1,
        "desc_act": true,
        "sym": true,
        "true_sequential": true,
        "quant_method": "gptq"
    }

Based on this, IMHO AutoAWQ should create a quantize_config.json matching this format:

"bits": 4,
"group_size": 128,
"zero_point": true,
"version": "GEMM"

Then that will be easily integrated into Transformers' config.json -> quantization_config section in future - with the addition of a "quant_method": "awq" key as well.

For backwards compatibility, AutoAWQ could also read from quant_config.json and still support the existing w_bits key name, so that support for existing models is still provided. Though I would update all my repo config files to the new format at such a time as the change was made.

I know that Huggingface takes backward compatibility very seriously. Implementing this in AutoAWQ requires quite a bit of maintenance going forward and it will break all PyPi releases of AutoAWQ (v0.0.1, v0.0.2, v0.1.0) because it will not be able to read any Huggingface repository. Additionally, the oobabooga/text-generation-webui will also need to update their version because it will brick the functionality in the PR. I am also not sure if this breaks the vLLM support for AWQ.

I am not so sure that people will be happy with this change for the reasons that I just explained, but I would gladly accept PRs that refactor the current behavior and extract the logic into a QuantizeConfig class that implements the necessary logic for backward compatibility. However, going forward, we have to support quant_config.json for multiple versions and raise warnings in AutoAWQ before we phase the quant_config.json out (just like they do in Huggingface libraries). We can finally phase it out in the next major release: v0.2.0.

vLLM already allows for both filenames:

   @classmethod
    def get_config_filenames(cls) -> List[str]:
        return [
            "quant_config.json",  # E.g., casperhansen/vicuna-7b-v1.5-awq
            "quantize_config.json",  # E.g., abhinavkulkarni/mosaicml-mpt-7b-instruct-w4-g128-awq  # pylint: disable=line-too-long
        ]

And both key names:

 weight_bits = cls.get_from_keys(config, ["w_bit", "bits"])

HF take backwards compatibility seriously once something is in a Hugging Face project - so they definitely won't want it to change after AWQ is merged. Before it's merged I don't think they'll care one way or the other. But they may well ask for the w_bits -> bits param change to be made in order for it to be merged, so I think you're going to end up doing at least that change eventually if you want it in Transformers.

So if this change is going to be made, I would say the time to do it is now - there's never a good time for a change that might break compatibility, but earlier is always far better than later

If the config file name does change, I'm happy to have both quantize_config and quant_config.json in all my repos for a while, which would allow ongoing support for prior AutoAWQ versions and any un-updated clients. Then I could phase it out in a a few weeks, once everyone is up-to-date. A bit like I've handled the many breaking llama.cpp changes.

I agree with @TheBloke that it's better to make changes now than later.

So, how about this:

1. @TheBloke, please write a "translator" to translate AutoAWQ quant_config.json to quantize_config.json and have both of them in the model repo. Please back-process all of your published models to have both of these files.

2. @casper-hansen: Over a period of time, please deprecate quant_config.json and start publishing quantize_config.json in the format HF team expects.

Agreed?

Ok, only nits are left. I will merge this into dev and add the nits I wanted to share and add an integration tests.

Thank you very much for this contribution !

Totally agree with all the names/config discussion, which is why I suggested reusing all the code we could for awq to match gptq.

Also in this repo, unlike in transformers we might want to make the necessary changes to support the existing repos (provided they are not too hard to support as in little lines of code.)

Also looking at the fix PR : casper-hansen/AutoAWQ@e188472 (Is that correct ?) it seems the fix, is just removing the unecessary kernels.

This part is not Windows support for the GEMM kernel, those are FasterTransformer kernels.

Here I added Windows support. It's super simple. MSVC does not support __asm__ __volatile__, so just remove the underscores and it works. casper-hansen/AutoAWQ@4c39a76

Oh nice Thanks for the tip !

There is only a 10% difference in inference between llama2 7b model quantize model and basic model on a single A100 GPU. In the autoAWQ inference example, there is a fuse_layer option that can significantly affect the speed depending on whether it is set to 'true/false'. I am curious if this option is always set to true in TGI

Is flash attention v2 installed and on?

I tested using the tgi-1.1.0 Docker image, which has Flash Attention v2 installed.