The storj-sim tool enables you to run all the components of the Storj network (Satellite, Storage Nodes, Console and Gateway) and test them on your local machine.

In every day Storj usage, the Satellite, Storage Node, and Uplink are run on separate servers and computers, but for the purposes of the test network, all of the components are run locally.

You need to install:

1. Go 1.18
2. C tooling
3. Postgres
4. Redis

Go 1.18 can be installed by following instructions at golang.org.

Installing C tooling:

• Linux: apt install build-essential. Adjust for your distro.
• Mac: Install XCode and then xcode-select --install.
• Windows: tdm-gcc is the easiest to get started with. mingw2, MSYS2 and similar would work as well, however their setup is slightly more complicated.

You can setup postgres through docker. See Using Postgres section for more information. Otherwise, find the appropriate installation guide for your system.

Installing redis:

• Linux: apt install redis-server. Adjust for your distro.
• Mac: brew install redis.
• Windows: Download [https://github.com/ServiceStack/redis-windows/raw/master/downloads/redis-latest.zip]. Extract to any location and add it to your Environment Variable PATH.

git clone https://github.com/storj/storj.git storj
cd storj
make install-sim

Ensure that storj folder is outside of GOPATH, otherwise you may see errors.

This will install the storj-sim satellite storage node gateway and uplink binaries to wherever Go is configured to output binaries on your system. By default, this is $HOME/go/bin. If this location is not in your $PATH add it by running `export PATH="$HOME/go/bin:$PATH".

Next, run setup:

Postgres is required for storj-sim. There needs to be a postgres instance running and the connection string needs to be provided to storj-sim on setup. See section on using postgres below for details.

storj-sim network setup --postgres=postgres://[user]@[addr][:port]/[dbname]?sslmode=disable

You now have a configured Storj test network with default configuration options.

You might also want to take a look at the config by navigating to the root directory --config-dir where all the configs are specified. You can tweak the configuration settings there as needed.

For insight into what is happening under the hood you can use -x or --print-commands to see how the processes are started.

The next step is to run it!

Now that the configuration has been completed, we can fire up the test network with:

storj-sim network run

Your test network is now running. You should see output containing your Amazon S3 gateway access and secret keys, which you will need to connect Amazon S3 compatible clients.

At the moment it's assinging ports in the following way:

The port format is: "1PXXE", where P is the peer class, XX is the index of the instance, and E is the endpoint.

• Gateways start from port 11000
• Version control is at port 12000
• Bootstrap server is at port 13000
• Satellites start from port 10000
• Satellite Console starts on port 10002
• Storage Nodes public ports start from port 14000
• Storage Nodes private ports start from port 14001

See storj-sim network source code for more details.

To get access to a gateway and test your keys, you open http://127.0.0.1:11000 in a web browser.

You can access a storage node dashboard using the storage command. For example for accessing storage node 4 dashboard using the default configuration:

storagenode dashboard --config-dir ~/.local/share/storj/local-network/storagenode/4/ --identity-dir ~/.local/share/storj/local-network/storagenode/4 --address :13004 --color

storj-sim network test <command> can be used to run tests.

storj-sim will start up the network and once it's up and running it will execute the specified <command>.

The information about the network is exposed via environment flags. All the flags start with a prefix and an index.

• Address: STORAGENODE_0_ADDR, SATELLITE_0_ADDR, GATEWAY_0_ADDR
• Keys: GATEWAY_0_ACCESS_KEY, GATEWAY_0_SECRET_KEY
• Directory: STORAGENODE_0_DIR, SATELLITE_0_DIR, GATEWAY_0_DIR

You can obtain the list of environment flags by running:

storj-sim network env

For a real-world example you can take a look in test-sim.sh and test-sim-aws.sh.

For using the uplink command-line client with the storj-sim network you have to reuse the configuration of the gateway generated by the network setup. The gateway configuration file is very similar to the one used by uplink and it contains what's needed for accessing to the test network.

Because the gateway configuration have extra fields that uplink doesn't expect, you'll see some INFO Invalid configuraiton file... messages that you can ignore them, for example:

2019-11-22T11:55:06.442+0100    INFO    Invalid configuration file key: minio.access-key
2019-11-22T11:55:06.442+0100    INFO    Invalid configuration file key: minio.secret-key
2019-11-22T11:55:06.442+0100    INFO    Invalid configuration file key: server.address

To use the uplink with the gateway configuration file, you always have to pass the --config-dir parameter to uplink command-line tool with the full path to the directory containing the file, for example

 uplink --config-dir /home/user/.local/share/storj/local-network/gateway/0 ls sj://

storj-sim network destroy can be used to wipe the network easily. However postgres database is not wiped.

While developing it's often nice to be able to delete the network and set it up from scratch.

For convenience, you may run the command in a single line, like so:

storj-sim network destroy && storj-sim network setup && storj-sim network test bash my-test-script.sh

Here are the steps to run storj-sim with postgres:

Step 1: Start a postgres instance.

One way to do this is to run a postgres container locally. For example, the following commands will run a postgres docker container locally:

The following assumes that docker and psql are already installed.

// pull down official docker image
$ docker pull postgres

$ docker run --rm -e POSTGRES_HOST_AUTH_METHOD=trust -p 5432:5432 --name postgres postgres

// in a different tab run this command to create a database for storj-sim to use
$ docker exec -it postgres createdb -U postgres teststorj

// if you want to run your own postgres queries, run this command to log into the postgres
// interactive terminal
$ psql -h localhost -U postgres

Step 2: Run storj sim with postgres

// setup storj-sim network with postgres connection string. You can supply any password
// and database name as long as they match the postgres instance running above
$ storj-sim network setup --postgres=postgres://postgres@localhost/<dbName>?sslmode=disable

$ storj-sim network run

A few different processes can use redis as their database. Live accounting cache and revocation db both need to use redis on the Satellite. They will exist as separate databases on the same redis instance. For example, live accouning may use redis://127.0.0.1:6379?db=0 while revocation may use redis://127.0.0.1:6379?db=1.

By default, storj-sim will start a redis server for each Satellite that is run and write the respective connection string for each database to a redis.conf file. You must have redis-server installed locally for this process to run. You can download redis at https://redis.io/download. (On macOS you can install redis with homebrew brew install redis)

For example, if you're running 2 satellites...

storj-sim network --satellites 2 setup
storj-sim network --satellites 2 run

...a configuration like the following may be generated.

// satellite 0
--live-accounting.db: redis://127.0.0.1:10004?db=0
--server.revocation-dburl: redis://127.0.0.1:10004?db=1

// satellite 1
--live-accounting.db: redis://127.0.0.1:10014?db=0
--server.revocation-dburl: redis://127.0.0.1:10014?db=1

Alternatively, you can set up your own redis server and connect storj-sim directly.

For example, you may run redis-server (default port = 6379) or redis-server --port <port> then storj-sim network setup --redis 127.0.0.1:<port>. You can also set an environment variable for $STORJ_SIM_REDIS, e.g. export STORJ_SIM_REDIS=127.0.0.1:<port> then simply run storj-sim network setup without passing in a flag. All Satellites will use the same server but different dbs if you run a redis server yourself.

Example for 2 satellites:

storj-sim network --satellites 2 --redis 127.0.0.1:6379 setup
storj-sim network --satellites 2 run

OR

export STORJ_SIM_REDIS=127.0.0.1:6379
storj-sim network --satellites 2 setup
storj-sim network --satellites 2 run

results in:

// satellite 0
--live-accounting.db: redis://127.0.0.1:6379?db=0
--server.revocation-dburl: redis://127.0.0.1:6379?db=1

// satellite 1
--live-accounting.db: redis://127.0.0.1:6379?db=2
--server.revocation-dburl: redis://127.0.0.1:6379?db=3

If you're running your own redis server, you can designate the database value to start with flag --redis-startdb which defaults to 0. For example, storj-sim network --satellites 2 --redis 127.0.0.1:6379 --redis-startdb 3 setup will start incrementing the dbs from 3 (db=3,4,5,6 for 2 satellites). It assumes all databases from the start value onward are available.

Please see the Uplink CLI or S3 Gateway tutorial for how to upload and download data to the test network.

Let's 'be the cloud' and decentralize all the things together!