Skip to content

RFC: Erlang Dist Security Filtering Prototype #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

RFC: Erlang Dist Security Filtering Prototype #1

wants to merge 3 commits into from

Conversation

potatosalad
Copy link
Owner

@potatosalad potatosalad commented Jan 20, 2022 •
edited
Loading

Erlang Dist Security Filtering Prototype

20220108-Erlang-Dist-Security-This-is-Fine

The Erlang Distribution Protocol (erldist) is used extensively to connect service nodes for use with clustered message routing.

When it comes to security, however, we have found erldist to be lacking, albeit by design, as mentioned in the docs with the following warning:

Warning
The Erlang Distribution protocol is not by itself secure and does not aim to be so.

What exactly is Erlang Dist?

Erlang Dist is a bidirectional and stateful protocol between Erlang nodes (technically, other languages are also supported and are referred to as C or Hidden Nodes).

What does that actually look like in practice? Let's start two nodes in two separate terminals:

# Terminal 1
erl --name foo@127.0.0.1

# Terminal 2
erl --name bar@127.0.0.1

Initially, the nodes are not connected to any other nodes.

Let's run the following on our foo@127.0.0.1 node:

(foo@127.0.0.1)1> erlang:nodes().
[]
(foo@127.0.0.1)2> net_adm:ping('bar@127.0.0.1').
pong
(foo@127.0.0.1)3> erlang:nodes().
['bar@127.0.0.1']

Our call to net_adm:ping('bar@127.0.0.1'). resulted in a new connection between the two nodes.

What can we do with this new connection?

First, let's try manually copying a pid binary from one terminal to another.

Let's run the following on our foo@127.0.0.1 node:

(foo@127.0.0.1)4> erlang:display(erlang:term_to_binary(self())).
<<131,88,100,0,13,102,111,111,64,49,50,55,46,48,46,48,46,49,0,0,0,87,0,0,0,0,97,217,170,78>>
true

Let's copy the binary and run the following on our bar@127.0.0.1 node:

(bar@127.0.0.1)1> FooPid = erlang:binary_to_term(<<131,88,100,0,13,102,111,111,64,49,50,55,46,48,46,48,46,49,0,0,0,87,0,0,0,0,97,217,170,78>>).
<9115.87.0>
(bar@127.0.0.1)2> FooPid ! {hello_from_bar, self()}.
{hello,<0.87.0>}

Let's run receive on our foo@127.0.0.1 node and send something back (remember, erldist connections are bidirectional):

(foo@127.0.0.1)5> {hello_from_bar, BarPid} = receive Msg -> Msg end.
{hello_from_bar,<9304.87.0>}
(foo@127.0.0.1)6> BarPid ! {hello_from_foo, self()}.
{hello_from_foo,<0.87.0>}

Let's run flush(). on our bar@127.0.0.1 node:

(bar@127.0.0.1)3> flush().
Shell got {hello_from_foo,<9115.87.0>}
ok

Great, message passing is working. Now what?

What else can we do?

As it turns out: we can do a lot of things right out of the box.

We can start arbitrary functions on another node:

(bar@127.0.0.1)4> spawn_request('foo@127.0.0.1', fun() -> do_something_malicious() end).
#Ref<0.3598369981.537133059.210537>

We can call any function on another node and return the result to the caller:

(bar@127.0.0.1)5> erpc:call('foo@127.0.0.1', foo, get_secret, []).
<<"some secret">>

We can reset the group_leader of any process on another node which will do things like redirect console output (or calls to io:format/2) over the network to the calling node:

(bar@127.0.0.1)6> erlang:group_leader(erlang:group_leader(), FooPid).
true

We can send messages to any pid, registered process or port, or alias on another node:

(bar@127.0.0.1)7> {my_foo_process, 'foo@127.0.0.1'} ! hello.
hello

How is Erlang Dist implemented?

As of Erlang/OTP 24, erldist is implemented in four main parts:

  1. ERTS Dist Controller BIF functions in dist.c:
  2. Erlang net_kernel module for keeping track of connections and Dist Controller handles.
  3. Erlang dist_util module which handles the handshake and net tick portions of the protocol.
  4. Erlang proto_dist module, like inet_tcp_dist.erl, which handles the transport or "carrier" for the protocol.

NOTE: There are also other parts specific to each proto_dist, like the customizations built into the Port Driver system leveraged by inet_drv.c or the tls_sender.erl and tls_socket.erl which are leveraged by inet_tls_dist.erl. In addition, some helper functions, like erts_internal:dist_spawn_init/1 are used by dist.c.

What is Erlang Dist Security Filtering?

In short: it's a Firewall for Erlang Dist.

20220108-Erlang-Better

Filters (or rules) are configured per-node connection using either coarse-grained or fine-grained BIF calls that modify the dist_entry table within dist.c for the given connection.

Coarse-grained filters may be configured with either accept or reject (with accept being the default):

net_kernel:set_filter(Node, link, reject).
net_kernel:set_filter(Node, reg_send, reject).
net_kernel:set_filter(Node, group_leader, reject).
net_kernel:set_filter(Node, monitor, reject).
net_kernel:set_filter(Node, send, reject).
net_kernel:set_filter(Node, spawn_request, reject).
net_kernel:set_filter(Node, alias_send, reject).

Fine-grained filters may be configured for reg_send and spawn_request command types:

net_kernel:add_filter(Node, reg_send, my_secret_process, reject).
net_kernel:add_filter(Node, reg_send, my_public_process, accept).

net_kernel:add_filter(Node, spawn_request, {my_mod, my_fun, 0}, accept).
net_kernel:add_filter(Node, spawn_request, {my_mod, my_fun, 1}, reject).

Fine-grained filters may also be removed for reg_send and spawn_request command types:

net_kernel:del_filter(Node, reg_send, my_secret_process).
net_kernel:del_filter(Node, reg_send, my_public_process).

net_kernel:del_filter(Node, spawn_request, {my_mod, my_fun, 0}).
net_kernel:del_filter(Node, spawn_request, {my_mod, my_fun, 1}).

Combined filtering may be tested to see whether a given command from a node will result in an accept or reject:

1> net_kernel:test_filter(Node, reg_send, my_secret_process).
accept
2> net_kernel:add_filter(Node, reg_send, my_secret_process, reject).
accept
3> net_kernel:test_filter(Node, reg_send, my_secret_process).
reject

In addition, there is an option to enable Erlang-based filtering for reg_send, group_leader, send, spawn_request, and alias_send commands:

-module(my_dist_handler).

-export([
    reg_send/3,
    group_leader/3,
    send/3,
    dist_spawn_init/4, % spawn_request handler function
    alias_send/3
]).

-spec reg_send(Node, From, ToRegName) -> no_return() when
    Node :: node(),
    From :: pid() | '',
    ToRegName :: atom().
reg_send(Node, From, ToRegName) ->
    % Perform any pre-message-filtering here...
    receive
        Message ->
            % Perform any post-message-filtering here...
            catch ToRegName ! Message,
            exit(normal)
    end.

-spec group_leader(Node, GroupLeader, Pid) -> no_return() when
    Node :: node(),
    GroupLeader :: pid(),
    Pid :: pid().
group_leader(Node, GroupLeader, Pid) ->
    % Perform any filtering here...
    true = erlang:group_leader(GroupLeader, Pid),
    exit(normal).

-spec send(Node, From, ToPid) -> no_return() when
    Node :: node(),
    From :: pid() | '',
    ToPid :: pid().
send(Node, From, ToPid) ->
    % Perform any pre-message-filtering here...
    receive
        Message ->
            % Perform any post-message-filtering here...
            catch ToPid ! Message,
            exit(normal)
    end.

-spec dist_spawn_init(Node, Module, Function, Arguments) -> Result when
      Node :: node(),
      Module :: module(),
      Function :: atom(),
      Arguments :: [term()],
      Result :: any().
dist_spawn_init(Node, Module, Function, Arguments) ->
    % Perform any filtering here...
    erlang:apply(Module, Function, Arguments).

-spec alias_send(Node, From, ToAlias) -> no_return() when
    Node :: node(),
    From :: pid() | '',
    ToAlias :: reference().
alias_send(Node, From, ToAlias) ->
    % Perform any pre-message-filtering here...
    receive
        Message ->
            % Perform any post-message-filtering here...
            catch ToAlias ! Message,
            exit(normal)
    end.

The my_dist_handler module can be enabled per-node with:

4> net_kernel:set_handler(Node, reg_send, my_dist_handler).
undefined
5> net_kernel:set_handler(Node, group_leader, my_dist_handler).
undefined
6> net_kernel:set_handler(Node, send, my_dist_handler).
undefined
7> net_kernel:set_handler(Node, spawn_request, my_dist_handler).
undefined
8> net_kernel:set_handler(Node, alias_send, my_dist_handler).
undefined

Any inbound control commands for these types, if accepted by the coarse-grained and fine-grained filters, will spawn a new process and execute the above callbacks based on the command type.

How is Erlang Dist Filtering implemented?

The implementation is still subject to change, but as of 2021H2:

  1. New BIF functions for interacting with the dist_entry table from the net_kernel module in dist.c:
    • erlang:dist_add_filter/4
    • erlang:dist_del_filter/3
    • erlang:dist_set_filter/3
    • erlang:dist_set_handler/3
    • erlang:dist_test_filter/3
  2. Modifications to the erts_internal module to support Handler:dist_spawn_init(Node, M, F, A) callbacks for custom handlers.
  3. Modifications to dist_util and net_kernel to support calling BIF functions from (1):
    • net_kernel:add_filter/4
    • net_kernel:del_filter/3
    • net_kernel:set_filter/3
    • net_kernel:set_handler/3
    • net_kernel:test_filter/3
  4. Extensions to the dist_entry struct and erl_alloc.types to support a new dist_filter hash table in erl_node_tables.h.
  5. Minor modification to erl_create_process in erl_process.c so that spawn_reply is only enabled if the tag on the ErlSpawnOpts is equal to am_spawn_reply.

@potatosalad potatosalad force-pushed the wip-dist-filtering branch 4 times, most recently from a65a3b3 to a99b957 Compare January 26, 2022 20:07
@potatosalad potatosalad changed the title Erlang Dist Security Filtering Prototype RFC: Erlang Dist Security Filtering Prototype Jan 27, 2022
@potatosalad potatosalad force-pushed the wip-dist-filtering branch 3 times, most recently from dd2d761 to ec651d7 Compare February 2, 2022 21:01
@potatosalad potatosalad mentioned this pull request Mar 6, 2022
Merged
@potatosalad potatosalad force-pushed the wip-dist-filtering branch 2 times, most recently from 2f639b3 to d62cfe4 Compare March 14, 2022 17:57
@potatosalad potatosalad mentioned this pull request Mar 30, 2022
Closed
@rickard-green
Copy link

I posted a comment regarding the strategy in general in the erlangforums thread.

Just adding a comment here on the implementation of signal handlers for send, reg_send, etc that you've implemented in the prototype as well. These will break the signal order guarantee that the language provides since execution order of the spawned processes executing the handlers is undefined. Code that should be executing behind such filters would also have to be prepared for random signal order delivery.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@potatosalad @rickard-green