It is only used for UDP ports, so it can be replaced with a uint16.

When/If we add more overlays/SCION address types, we can use net.Addr to carry around generic addressing information anyway, and have code that cares about address internals just cast to what it supports.