Skip to content

Conversation

sipa
Copy link
Member

@sipa sipa commented Apr 9, 2020

This adds a new binary, bitcoin-asmap that can be used to construct, inspect, and query asmap files. It replaces the Python scripts I had at https://github.com/sipa/asmap with something more performant, efficient, and auditable.

There are some rough edges still (in particular documenting the code and a number of edge cases), but I'd like to offer it for comments here. I'm not sure we want this code to be part of the Bitcoin Core codebase, but it certainly has advantages (in terms of code reuse and having consistent encoder/decoder). I think it may also be beneficial that users have a way to audit these files using code that's subject to our review and testing process.

This PR includes a fuzz test that verifies (for small, reduced inputs) that encoding produces asmap files consistent with their input, that invoking the interpreter on them works as expected, and that decoding and re-encoding does not change their behavior.

Given a text file asmap.txt with lines like these:

193.189.95.0/24 AS34906
193.189.96.0/24 AS20850
193.189.98.0/23 AS33925
193.190.0.0/15 AS2611
193.19.102.0/23 AS20485
193.19.103.0/24 AS8342
193.19.106.0/23 AS3257

You can run bitcoin-asmap encode asmap.dat <asmap.txt to construct an asmap file in a few seconds. bitcoin-asmap decode asmap.dat will decode it back to the original format (note however that the result may look very different, as unassigned IP ranges get remapped for efficiency). bitcoin-asmap lookup asmap.dat 193.190.253.208 will look up one entry in it.

There is code at https://github.com/rrybarczyk/asmap-rs (by @rrybarczyk and @naumenkogs) to build input for this tool using RIPE's dumps of Internet routing tables.

@jonatack
Copy link
Member

jonatack commented Apr 9, 2020

Concept ACK (leaving aside for now the question of inclusion in the codebase).

@naumenkogs
Copy link
Member

naumenkogs commented Apr 9, 2020

Concept ACK. I think it makes sense for us to have this tool inside the repo, if we want to encourage using asmap, and improve the auditability/maintenance.

Did some slight code review, will do more if/when we get enough Concept ACKs.

Also, I will probably post this separately, but anyone who speaks rust is welcome to review our tool (https://github.com/i7i/asmap-rs) for aggregating routing info from various sources :)

@DrahtBot
Copy link
Contributor

DrahtBot commented Apr 9, 2020

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

@sipa
Copy link
Member Author

sipa commented Apr 12, 2020

Added lots of comments, and improved the encoder algorithm. I believe it will now always find the smallest encoding (in a few seconds, for realistic inputs).

@sipa sipa force-pushed the 202004_asmap_tool branch from cead708 to 3d435b1 Compare April 12, 2020 01:42
@practicalswift
Copy link
Contributor

Concept ACK

@jb55
Copy link
Contributor

jb55 commented Apr 16, 2020

+0 for having it in the repo, Concept ACK

@luke-jr
Copy link
Member

luke-jr commented Apr 23, 2020

Ideally it should be in another repo, but maybe we're not ready for that yet if it shares a lot of code.

@DrahtBot
Copy link
Contributor

DrahtBot commented May 6, 2020

🐙 This pull request conflicts with the target branch and needs rebase.

@Sjors
Copy link
Member

Sjors commented Nov 16, 2020

It might be worth building this on top of #19937 as part of bitcoin-util.

Concept ACK on including something this into the codebase, ideally along with some instructions which are a bit sparse now. Unless you suspect this needs tons of tweaking in the future, in which case perhaps a separate repo is better. Or if the output can be used by other software than Bitcoin Core.

@ajtowns ajtowns mentioned this pull request Nov 23, 2020
@lestephane
Copy link

@Sjors I had the same problem with the lack of clear instructions so I created a Vagrantfile that performs all the steps in one go.

It's located at https://github.com/lestephane/vagrant-asmap

You run vagrant up, and after a while, if your machine has the minimum specs to run the bottleneck analysis (12GB RAM seems to be needed for me), you end up with an ip_asn.map file.

@DrahtBot
Copy link
Contributor

There hasn't been much activity lately and the patch still needs rebase. What is the status here?
  • Is it still relevant? ➡️ Please solve the conflicts to make it ready for review and to ensure the CI passes.
  • Is it no longer relevant? ➡️ Please close.
  • Did the author lose interest or time to work on this? ➡️ Please close it and mark it 'Up for grabs' with the label, so that it can be picked up in the future.

1 similar comment
@DrahtBot
Copy link
Contributor

There hasn't been much activity lately and the patch still needs rebase. What is the status here?
  • Is it still relevant? ➡️ Please solve the conflicts to make it ready for review and to ensure the CI passes.
  • Is it no longer relevant? ➡️ Please close.
  • Did the author lose interest or time to work on this? ➡️ Please close it and mark it 'Up for grabs' with the label, so that it can be picked up in the future.

@dunxen
Copy link
Contributor

dunxen commented Apr 7, 2022

Concept ACK

I'm working on some tooling to help with map comparisons between releases and some user experience improvements to asmap-rs.

Although certainly we'll have an encoded map included with every release when asmap is enabled by default, I think it still makes sense to include bitcoin-asmap in this repo for easier maintenance and discoverability. It's just less hassle for node operators who may be interested generating and encoding their own mappings too.

@sipa
Copy link
Member Author

sipa commented Jun 8, 2022

Closing. I'm working on external tooling for working with asmap files in https://github.com/sipa/asmap/tree/nextgen (encode, decode, diff, bottleneck, ...).

@sipa sipa closed this Jun 8, 2022
@bitcoin bitcoin locked and limited conversation to collaborators Jun 8, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.