It's better to explain how future developers will run equivalent tests on their local machines in this PR. Can you add a document with real code to explain how to use ONNX hub for test a model with example data? I saw limited doc for loading a model but doesn't find instructions to really execute the model with example data.

This is massif. Can we try this PR on ORT branch to make sure this fixes our pipelines?

After discussion with Liqun, we decide to temporarily disable testing real models from AWS in onnx repo to faster unblock pipeline failures. Liqun will try to move them elsewhere like Azure. Then, we will add these tests back with updated URL instead of the current AWS URL.

It's better to explain how future developers will run equivalent tests on their local machines in this PR. Can you add a document with real code to explain how to use ONNX hub for test a model with example data? I saw limited doc for loading a model but doesn't find instructions to really execute the model with example data.

Good point. IIUC, onnx repo only tests the model itself: https://github.com/onnx/onnx/blob/main/onnx/test/test_backend_test.py#L38-L42 since previously there is no runtime implementation here. So the corresponding tests will be (Take AlexNet as an example)

import itertools
import onnx
model = onnx.hub.load('alexnet', opset=9)
onnx.checker.check_model(model)
model = onnx.shape_inference.infer_shapes(model, check_type=True, strict_mode=True)
value_infos = {
    vi.name: vi
    for vi in itertools.chain(model.graph.value_info, model.graph.output)
}
for node in model.graph.node:
    for i, output in enumerate(node.output):
        if node.op_type == "Dropout" and i != 0:
            continue
        assert output in value_infos
        tt = value_infos[output].type.tensor_type
        assert tt.elem_type != onnx.TensorProto.UNDEFINED
        for dim in tt.shape.dim:
            assert dim.WhichOneof("value") == "dim_value"

I will provide the full equivalent testing method if we eventually decide to remove them for good.

This is massif. Can we try this PR on ORT branch to make sure this fixes our pipelines?

ORT pipelines will be OK for now by manually disabling these real model tests (from onnx repo) there: microsoft/onnxruntime#14606. In the near future, after we have transferred these models to a new place, we will update the URL in ONNX repo, have a patch release commit with that update and then consume that ONNX commit in ORT to recover these tests. Ideally only URLs will be updated and all other relevant stuff in ONNX will be consistent.

Instead of copying the models somewhere else, would it be possible to remove the initializers and replace them by a node ConstantOfShape. The models should be a lot smaller and could be part of the repository without disabling the tests. I tried this idea in PR #4861.

Thank you all for the input. This PR is superseded by #4865 Instead of disabling them, we have replaced the broken URL with the new URL under onnxruntime.ai. Current ONNX 1.13 still has the old broken URLs and so we will have a a patch 1.13.1 release which includes #4865.