Skip to content

get_cost_info(hp="200") returns weird value for some models in topology search space #36

@mzsrkeen10

Description

@mzsrkeen10

Describe the bug
For some architectures, get_cost_info() returns different flops/params for hp="12" and hp="200" in topology search space.
For example, the number of parameters (for CIFAR-10) of architecture |skip_connect~0|+|none~0|nor_conv_3x3~1|+|avg_pool_3x3~0|nor_conv_3x3~1|nor_conv_3x3~2| is 0.802426 for hp="12", but for hp="200", the value is 0.6403993333333333.

I've found many other architectures in which the same behavior occurs, but some specific examples are as follows:

  • |nor_conv_1x1~0|+|nor_conv_3x3~0|skip_connect~1|+|none~0|skip_connect~1|none~2|
  • |nor_conv_3x3~0|+|skip_connect~0|skip_connect~1|+|none~0|skip_connect~1|none~2|

To Reproduce

from nats_bench import create

arch = "|skip_connect~0|+|none~0|nor_conv_3x3~1|+|avg_pool_3x3~0|nor_conv_3x3~1|nor_conv_3x3~2|"
api_path = "NATS-tss-v1_0-3ffb9-simple"
search_space = "tss"
dataset = "cifar10"

api = create(api_path, search_space, fast_mode=True, verbose=False)
idx = api.query_index_by_arch(arch)
info_12 = api.get_cost_info(idx, dataset, hp="12")
info_200 = api.get_cost_info(idx, dataset, hp="200")
print(info_12)
print(info_200)
{'flops': 113.95137, 'params': 0.802426, 'latency': 0.016719988563604522, 'T-train@epoch': 21.51544686158498, 'T-train@total': 258.1853623390198, 'T-ori-test@epoch': 1.496117415882292, 'T-ori-test@total': 17.953408990587505}
{'flops': 90.35841, 'params': 0.6403993333333333, 'latency': 0.016719988563604522, 'T-train@epoch': 21.515446861584987, 'T-train@total': 4303.0893723169975, 'T-ori-test@epoch': 1.4961174158822923, 'T-ori-test@total': 299.22348317645844}
  • OS: Ubuntu 20.04.3 LTS (Focal Fossa)
  • Python version: 3.8.12
  • PyTorch version: 1.8.1+cu102

Expected behavior
I think flops/params for hp="12" and hp="200" should be the same.
(From reading the paper and your implementation in AutoDL-Projects, I could not find any factors that would make a difference.)

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions