Acknowledging #15400 is still unsolved, I've done some testing on comparing the memory footprint for p2p_invalid_messages on Ubuntu 18.04 vs macOS.

Checking the size of the transport buffer and rss memory after each msg_at_size shows that after the second message is sent, on macOS, the rss memory increases dramatically. Additional there are leftover bytes in the transport output buffer. tracemalloc shows that a large memory block is allocated in the call self._buffer.extend(data) in /asyncio/selector_events.py:882 where the output buffer is extended because not everything was able to be sent.

============= Before Sending Message =============
Message #1, memory footprint: 77088
Current output buffer size 1735312

============= After Sending Message =============
Message #1, memory footprint: 83544
Current output buffer size 2609568

[ TraceMalloc Output ]
/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/selector_events.py:882: size=2548 KiB, count=1, average=2548 KiB
/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py:1420: size=736 B, count=4, average=184 B
/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/selector_events.py:883: size=496 B, count=1, average=496 B
/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/selector_events.py:305: size=496 B, count=1, average=496 B
/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/events.py:88: size=496 B, count=1, average=496 B
/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/selector_events.py:903: size=488 B, count=1, average=488 B

The increase in memory from this step (~13%) accounts for most (~99%) of the overall memory increase in this test. I.e. this step is the reason for the need for a4eaaa6

In Ubuntu, the buffer is always emptied after each send and there is a minimal spike in memory footprint.

============= Before Sending Message =============
Message #1, memory footprint: 51532
Current output buffer size 0

============= After Sending Message =============
Message #1, memory footprint: 52884
Current output buffer size 0

Any thoughts on approaches to solving this would be appreciated. It seems to mainly be an initial overhead penalty in macOS as the memory footprint plateaus for the remaining 78 messages. So perhaps it is a macOS specific quirk?

qa: p2p_invalid_messages needs improvement #14814

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions