-
Notifications
You must be signed in to change notification settings - Fork 21
Closed
Labels
bugSomething isn't workingSomething isn't working
Description
Describe the bug
For some architectures, get_cost_info()
returns different flops/params for hp="12"
and hp="200"
in topology search space.
For example, the number of parameters (for CIFAR-10) of architecture |skip_connect~0|+|none~0|nor_conv_3x3~1|+|avg_pool_3x3~0|nor_conv_3x3~1|nor_conv_3x3~2|
is 0.802426
for hp="12"
, but for hp="200"
, the value is 0.6403993333333333
.
I've found many other architectures in which the same behavior occurs, but some specific examples are as follows:
|nor_conv_1x1~0|+|nor_conv_3x3~0|skip_connect~1|+|none~0|skip_connect~1|none~2|
|nor_conv_3x3~0|+|skip_connect~0|skip_connect~1|+|none~0|skip_connect~1|none~2|
To Reproduce
from nats_bench import create
arch = "|skip_connect~0|+|none~0|nor_conv_3x3~1|+|avg_pool_3x3~0|nor_conv_3x3~1|nor_conv_3x3~2|"
api_path = "NATS-tss-v1_0-3ffb9-simple"
search_space = "tss"
dataset = "cifar10"
api = create(api_path, search_space, fast_mode=True, verbose=False)
idx = api.query_index_by_arch(arch)
info_12 = api.get_cost_info(idx, dataset, hp="12")
info_200 = api.get_cost_info(idx, dataset, hp="200")
print(info_12)
print(info_200)
{'flops': 113.95137, 'params': 0.802426, 'latency': 0.016719988563604522, 'T-train@epoch': 21.51544686158498, 'T-train@total': 258.1853623390198, 'T-ori-test@epoch': 1.496117415882292, 'T-ori-test@total': 17.953408990587505}
{'flops': 90.35841, 'params': 0.6403993333333333, 'latency': 0.016719988563604522, 'T-train@epoch': 21.515446861584987, 'T-train@total': 4303.0893723169975, 'T-ori-test@epoch': 1.4961174158822923, 'T-ori-test@total': 299.22348317645844}
- OS: Ubuntu 20.04.3 LTS (Focal Fossa)
- Python version: 3.8.12
- PyTorch version: 1.8.1+cu102
Expected behavior
I think flops/params for hp="12"
and hp="200"
should be the same.
(From reading the paper and your implementation in AutoDL-Projects, I could not find any factors that would make a difference.)
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working