Feature dynamic hba #412

t0k4rt · 2018-01-15T15:39:16Z

Hello,
We want to deploy Stolon in our architecture, but we noticed that connection to su and repl users are open to the world which is (imho) not good for a secure cluster.
We can deal with this with a firewall rule but I would prefer this to be setup correctly in the pg_hba.conf.
I saw a todo in your code concerning this issue so I decide to propose a fix. I'm not goloang developer nor sure it will match your coding standard or code organization wishes ... but it's a start.

The code simply read the cluster data generate a list of pghba entries and merges it with the db.Spec.PGHBA.

I found this to be the easiest solution.

Important to notice, it does not compile, I've got the current error:

# github.com/sorintlab/stolon/vendor/github.com/sorintlab/tcpkeepalive
gopath/src/github.com/sorintlab/stolon/vendor/github.com/sorintlab/tcpkeepalive/tcpkeepalive.go:14:10: undefined: setIdle
gopath/src/github.com/sorintlab/stolon/vendor/github.com/sorintlab/tcpkeepalive/tcpkeepalive.go:22:10: undefined: setCount
gopath/src/github.com/sorintlab/stolon/vendor/github.com/sorintlab/tcpkeepalive/tcpkeepalive.go:29:10: undefined: setInterval

I also changed the default su and repl connection to localhost.

I hope it can be useful for you !
Don't hesitate to tell me how I can improve this.
Thanks for this product !
Alexandre

t0k4rt · 2018-01-15T17:19:55Z

./test does not launch on OSX either

sgotti

@t0k4rt Thanks for the PR!

We develop and test stolon on GNU/Linux so it isn't tested on macosx (but github.com/sorintlab/tcpkeepalive/tcpkeepalive should be fixed to at least compile on macosx).

As an initial review:

you should drop the updated vendor mega commit which add tons of files without explaining the reasons. If you have to update some vendored dependencies please open another different issue/PR explaining the reasons.
Changing the generated hba entries will probably break the current behavior for someone reyling on it also for external replication. It'll be better to enable it only if a cluster spec option is set. For example we could define an option called DefaultReplAccess that can have two values: all (default) and onlyfollowers.
See the other inline comments.

sgotti · 2018-01-19T13:50:16Z

cmd/keeper/keeper.go

@@ -1512,7 +1513,8 @@ func (p *PostgresKeeper) postgresKeeperSM(pctx context.Context) {

 	if !reflect.DeepEqual(db.Spec.PGHBA, pgm.CurHba()) {
 		log.Infow("postgres hba entries changed, reloading postgres instance")
-		pgm.SetHba(db.Spec.PGHBA)
+		// Dynamicly generate hba auth from clusterData
+		pgm.SetHba(append(db.Spec.PGHBA, p.generateHBA(cd)...))


The generated hba entries should be inserted before the user provided hba entries.

sgotti · 2018-01-19T13:51:11Z

cmd/keeper/keeper.go

@@ -1512,7 +1513,8 @@ func (p *PostgresKeeper) postgresKeeperSM(pctx context.Context) {

 	if !reflect.DeepEqual(db.Spec.PGHBA, pgm.CurHba()) {
 		log.Infow("postgres hba entries changed, reloading postgres instance")
-		pgm.SetHba(db.Spec.PGHBA)
+		// Dynamicly generate hba auth from clusterData
+		pgm.SetHba(append(db.Spec.PGHBA, p.generateHBA(cd)...))


don't append again here but do the append directly inside generateHBA.

t0k4rt · 2018-01-19T14:10:42Z

No Problems, I'll change that !
I'm not experienced at all with go nor go vendors management, so please excuse this poor (yet) PR.

sgotti · 2018-01-19T14:21:46Z

I'm not experienced at all with go nor go vendors management, so please excuse this poor (yet) PR.

don't worry and welcome to the wonderful world of Go! Feel free to ask for any doubt (here on on gitter).

t0k4rt · 2018-01-23T16:39:06Z

Hi, I updated the code as requested.
I added an optional switch defaultReplAccess that can take two options: "all" (default value) or "standby"

In case user select all:

master will have its default hba with:
- repl access to all ips (0.0.0.0/0)
- su access to all ips (0.0.0.0/0)
slave will have its default hba with:
- su access to all ips (0.0.0.0/0)
- no replication access since slave does not need it

In case user select standby:

master will have its default hba with:
- repl access to all standby ips (x.x.x.x/32)
- su access to localhost (127.0.0.1/32)
- su access to all standby ips (x.x.x.x/32)
slave will have its default hba with:
- su access to localhost (127.0.0.1/32)
- no replication access since slave does not need it

I tested the binary on a linux server, it works as expected.
Maybe I should seperate repl and su options and create defaultReplAccess / defaultSuAccess ?

sgotti

Thanks. Updated comments. If you have some problems/doubts on how to add a parameter to the clusterspec let me know.

sgotti · 2018-01-24T10:31:26Z

cmd/keeper/keeper.go

@@ -89,6 +89,7 @@ type config struct {
 	pgSUPasswordFile        string
 	pgInitialSUUsername     string
 	pgInitialSUPasswordFile string
+	defaultReplAccess       string


Sorry, I meant to set it as a cluster spec option, so it could be set globally: https://github.com/sorintlab/stolon/blob/master/pkg/cluster/cluster.go#L186

Not a problem I'll change that

Changes done

sgotti · 2018-01-24T10:34:30Z

cmd/keeper/keeper.go

+			fmt.Sprintf("host all %s %s %s\n", p.pgSUUsername, "::0/0", p.pgSUAuthMethod),
+		)
+		// allow only for master server replication access
+		if role == common.RoleMaster {


This is a change from the current behaviour and it's breaking the standby cluster feature and related tests. So it should be removed.

I don't understand why it breaks.
Master pg_hba should only allow replication user to connect from standby servers.
Disbaled or enabling replication user on standby server should not have any effects.
When a standby server is promoted, its pg_hba should be automatically updated and would contain the right rules.

Stolon can also be in standby cluster mode. In this case the master keeper manages a standby instance that will sync with an external instance. So the other keepers will sync from it (cascading replication) and the above check will break.

I suggested to just remove it because to keep backward compatibility with "all" mode we have to keep the previous rules.

Ok, changes done. It should be ok !

t0k4rt · 2018-01-25T15:07:09Z

I got during tests this issue, So I'm going to do more tests

utils.go:121: [keeper c594fd73]: panic: runtime error: invalid memory address or nil pointer dereference
	utils.go:121: [keeper c594fd73]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0xadb63f]

sgotti · 2018-01-25T15:08:11Z

pkg/cluster/cluster.go

+	// Allow hba access to all members (0.0.0.0/0)
+	PgHBAAccessAll PgHBAAccessMode = "all"
+	// Allow hba access to standby server only
+	PgHBAAccessStandby PgHBAAccessMode = "standby"


Stale field?

sgotti · 2018-01-25T15:09:56Z

cmd/keeper/keeper.go

+	// Dynamically generate hba auth from clusterData
+	var hbaBase []string
+
+	if cluster.PgHBAAccessAll == *cd.Cluster.Spec.PgHBAAccess {


Use cd.Cluster.DefSpec().PgHBAAccess or you'll miss the default values when not provided and it'll panic like done now on semaphore.

sgotti · 2018-01-25T15:10:24Z

cmd/keeper/keeper.go

+	// Dynamically generate hba auth from clusterData
+	var hbaBase []string
+
+	if cluster.PgHBAAccessAll == *cd.Cluster.Spec.PgHBAAccess {


use a switch statement here?

sgotti · 2018-01-25T15:23:57Z

pkg/cluster/cluster.go

@@ -243,6 +259,10 @@ type ClusterSpec struct {
 	ExistingConfig *ExistingConfig `json:"existingConfig,omitempty"`
 	// Standby setting when role is standby
 	StandbySettings *StandbySettings `json:"standbySettings,omitempty"`
+	// Set default hba auth for replication user and superuser.
+	// Values can be "all" or "standby", "all" allow access to all ips, "standby" restrict master access to standby servers ips.


standby or strict?

IMHO it's better strict

sgotti · 2018-01-25T15:24:42Z

pkg/cluster/cluster.go

+	// Set default hba auth for replication user and superuser.
+	// Values can be "all" or "standby", "all" allow access to all ips, "standby" restrict master access to standby servers ips.
+	// default is "all"
+	PgHBAAccess *PgHBAAccessMode `json:"pgHBAAccess,omitempty"`


I'll call this DefaultSUReplAccessMode, longer but a bit clearer. Other Suggestions?

We could use:

DefaultSUReplAccessMode as option

SUReplAccessMode as type

DefaultSUReplAccess as default value

SUReplAccessAll and SUReplAccessStrict as option names
Is it ok for you ?

Seems Good.

sgotti · 2018-01-25T16:32:02Z

cmd/keeper/keeper.go

@@ -1510,9 +1511,11 @@ func (p *PostgresKeeper) postgresKeeperSM(pctx context.Context) {
 		log.Infow("postgres parameters not changed")
 	}

-	if !reflect.DeepEqual(db.Spec.PGHBA, pgm.CurHba()) {
+	// Dynamicly generate hba auth from clusterData
+	new_hba := p.generateHBA(cd, localRole, db.Spec.PGHBA)


s/new_hba/newHBA/

t0k4rt · 2018-01-25T16:32:45Z

@sgotti I've got an issue when I try to update my ClusterSpec,
the key "defaultSUReplAccessMode": "strict" is properly set but not persisted.
Any ideas ?

sgotti · 2018-01-25T18:54:02Z

pkg/postgresql/postgresql.go

-	// TODO(sgotti) Configure this dynamically based on our followers provided by the clusterview
-	f.WriteString(fmt.Sprintf("host replication %s %s %s\n", p.replUsername, "0.0.0.0/0", p.replAuthMethod))
-	f.WriteString(fmt.Sprintf("host replication %s %s %s\n", p.replUsername, "::0/0", p.replAuthMethod))
-
 	if p.hba != nil {
 		for _, e := range p.hba {
 			f.WriteString(e + "\n")


You have forgot to handle the "else" case below inside generateHBA. Here it'll always be skipped since p.hba is never empty.

Ok, this means that previously, when p.hba was empty, the generated hba was:

host replication repluser 0.0.0.0/0 md5 host replication repluser ::0/0 md5 host all superuser 0.0.0.0/0 md5 host all superuser ::0/0 md5 host all all 0.0.0.0/0 md5 host all all ::0/0 md5

which is same as this:

host all all 0.0.0.0/0 md5 host all all ::0/0 md5

but simpler so I think this should be the result in "all" access mode.

In "strict" access mode we should not use this rule.

Not always, the first lines can have a different auth method than md5 (currently only trust but in future other methods could be added), while the last lines are added only if there's no user provided pgHBA and always have md5. I'd prefer to not change the current logic.

Yes you're right !

For the strict mode, I still would prefer not to add this rule.

It may be confusing but these are two separate things. The first lines (the one affected by this patch to set the DefaultSuReplAccessMode) are for instance replication, the latter is a default when the user doesn't provide a PGHBA. So I'll leave them distinct and not make DefaultSuReplAccessMode change this.

Ok I understand !
Can I plan a DefaultUserAccessMode feature ?

@t0k4rt you can already do the same providing an empty (not nil) pgHBA in the clusterspec. No need for a new options.

sgotti · 2018-01-25T18:57:06Z

cmd/keeper/keeper.go

+		hbaBase = append(
+			hbaBase,
+			fmt.Sprintf("host all %s %s %s\n", p.pgSUUsername, "0.0.0.0/0", p.pgSUAuthMethod),
+			fmt.Sprintf("host all %s %s %s\n", p.pgSUUsername, "::0/0", p.pgSUAuthMethod),


remove \n in all the lines or an empty line will be added

sgotti · 2018-01-25T18:57:57Z

the key "defaultSUReplAccessMode": "strict" is properly set but not persisted.

What do you mean by "not persisted"? I tried a stolonctl patch and it has been applied to the cluster spec and the pghba file has been updated (with some problems, see my comments) by the keepers.

t0k4rt · 2018-01-26T08:43:11Z

@sgotti Ok the not persisted issue is because I've got a cluster running two different versions of the keeper. The other keeper does not know the new key so it might removes it.

sgotti · 2018-01-26T09:00:28Z

@sgotti Ok the not persisted issue is because I've got a cluster running two different versions of the keeper. The other keeper does not know the new key so it might removes it.

The unique component that writes the cluster data is the sentinel (and stolonctl) so perhaps you had an old sentinel.

sgotti · 2018-01-26T09:45:27Z

pkg/postgresql/postgresql.go

@@ -713,30 +713,14 @@ func (p *Manager) writePgHba() error {

 	// Minimal entries for local normal and replication connections needed by the stolon keeper
 	// Matched local connections are for postgres database and suUsername user with md5 auth
-	// Matched local replicaton connections are for replUsername user with md5 auth
+	// Matched local replication connections are for replUsername user with md5 auth
 	f.WriteString(fmt.Sprintf("local postgres %s %s\n", p.suUsername, p.suAuthMethod))
 	f.WriteString(fmt.Sprintf("local replication %s %s\n", p.replUsername, p.replAuthMethod))


In the meantime I'll also move these lines in generateHBA and make writePgHba() a function that just writes the pgHBA file without custom logic inside.

sgotti · 2018-01-26T12:00:55Z

cmd/keeper/keeper.go

+			"host all all 0.0.0.0/0 md5",
+			"host all all ::0/0 md5",
+		)
+	}


this must be at the end. After the stolon generated rules.

sgotti · 2018-01-26T12:01:53Z

cmd/keeper/keeper.go

+	}
+
+	// when no custom hba is set, we default to open to all
+	if len(userHba) == 0 {


don't use len but check if it's nil or not like currently done. So you can distinguis between a not provided (nil) or a provided but empty pgHBA.

sgotti · 2018-01-26T12:06:03Z

pkg/cluster/cluster.go

@@ -243,6 +257,10 @@ type ClusterSpec struct {
 	ExistingConfig *ExistingConfig `json:"existingConfig,omitempty"`
 	// Standby setting when role is standby
 	StandbySettings *StandbySettings `json:"standbySettings,omitempty"`
+	// Set default hba auth for replication user and superuser.
+	// Values can be "all" or "strict", "all" allow access to all ips, "strict" restrict master access to standby servers ips.


s/to/from/g

sgotti · 2018-01-26T12:08:55Z

pkg/cluster/cluster.go

@@ -243,6 +257,10 @@ type ClusterSpec struct {
 	ExistingConfig *ExistingConfig `json:"existingConfig,omitempty"`
 	// Standby setting when role is standby
 	StandbySettings *StandbySettings `json:"standbySettings,omitempty"`
+	// Set default hba auth for replication user and superuser.


Define the mode of the default hba rules needed for replication by standby keepers (the su and repl auth methods will be the one provided in the keeper command line options)

sgotti

Thanks for the patience. A batch of some other comments.

Can you also update the clusterspec doc to add the new parameter?

sgotti · 2018-01-26T15:10:12Z

cmd/keeper/keeper.go

+		// allow only for master replication access
+		if role == common.RoleMaster {
+			// we should not allow replication acces for 127.0.0.1 since it can be then forwarded through proxy.
+			for _, db := range cd.DBs {


Unrelated comment?

sgotti · 2018-01-26T15:19:50Z

cmd/keeper/keeper.go

+		)
+		// allow only for master replication access
+		if role == common.RoleMaster {
+			// we should not allow replication acces for 127.0.0.1 since it can be then forwarded through proxy.


This will break the standby cluster cascading replication, we should also match a master keeper that has a standby role in a standby cluster .

I suggest to add a function like (similar to

stolon/cmd/sentinel/sentinel.go

Line 452 in 70464d3

func (s *Sentinel) dbType(cd *cluster.ClusterData, dbUID string) dbType {

) (not tested):

// IsMaster return if the db is the cluster master db. // A master is a db that: // * Has a master db role // or // * Has a standby db role with followtype external func IsMaster(db *cluster.DB) bool { switch db.Spec.Role { case common.RoleMaster: return true case common.RoleStandby: if db.Spec.FollowConfig.Type == cluster.FollowTypeExternal { return true } return false default: panic("invalid db role in db Spec") } }

Then change generateHBA role common.Role parameter to master bool and use it in the above check

BTW It's not triggered by the tests because the new SUReplAccessStrict isn't tested anywhere. We should add/change some integration tests to also test this option. Perhaps I can do this in another PR.

No problems, I'll check that !
I'll update the cluster spec too
Concerning an integration test I'll check too but not sure I'll managed to do something good enough ...

Concerning an integration test I'll check too but not sure I'll managed to do something good enough ...

Don't worry, I could add something later. Unfortunately integration tests of distributed system is quite tedious and complex...

sgotti · 2018-01-26T15:20:42Z

cmd/keeper/keeper.go

+		if role == common.RoleMaster {
+			// we should not allow replication acces for 127.0.0.1 since it can be then forwarded through proxy.
+			for _, db := range cd.DBs {
+				// allow slaves only to connect as replication user to master


better to put the comments above their related code line.

t0k4rt · 2018-02-02T09:12:53Z

@sgotti Hello, everything should be fixed.
Have you any other feedback ?
thanks !

sgotti

I created in this commit: sgotti@cfc5a03 an unit test for generataHBA. There're also some changes explained in the review comments and discovered thanks to this test. You can just cherry pick it and put it in your branch.

sgotti · 2018-02-08T14:19:16Z

cmd/keeper/keeper.go

+		// we should not allow replication acces for 127.0.0.1 since it can be then forwarded through proxy.
+		computedHBA = append(
+			computedHBA,
+			fmt.Sprintf("host all %s %s %s", p.pgSUUsername, "127.0.0.1/32", p.pgSUAuthMethod),


shouldn't this be 127.0.0.0/8 ?

yes that's better !

sgotti · 2018-02-08T14:19:33Z

cmd/keeper/keeper.go

+	computedHBA := []string{
+		fmt.Sprintf("local postgres %s %s\n", p.pgSUUsername, p.pgSUAuthMethod),
+		fmt.Sprintf("local replication %s %s\n", p.pgReplUsername, p.pgReplAuthMethod),
+	}


remove \n

sgotti · 2018-02-08T14:20:45Z

cmd/keeper/keeper.go

+	// Case strict access mode
+	case cluster.SUReplAccessStrict:
+		// allow for all servers su access
+		// we should not allow replication acces for 127.0.0.1 since it can be then forwarded through proxy.


Sorry but I forgot why this is needed. The comment isn't really clear and doesn't looks related to the added rules.

True, it's changed

sgotti · 2018-02-15T11:14:57Z

cmd/keeper/keeper.go

+		computedHBA = append(
+			computedHBA,
+			fmt.Sprintf("host all %s %s %s", p.pgSUUsername, "127.0.0.0/8", p.pgSUAuthMethod),
+			fmt.Sprintf("host all %s %s %s", p.pgSUUsername, "::1/128", p.pgSUAuthMethod),


Sorry but I don't remember why these line are needed. If really needed a comment explaining the reason will be useful.

Actually It's not mandatory since we already have
fmt.Sprintf("local postgres %s %s", p.pgSUUsername, p.pgSUAuthMethod)
I can drop this

sgotti · 2018-02-15T11:17:09Z

@t0k4rt Can you import the unit test I made in sgotti@cfc5a03 and squash and rebase against the current master? Thanks!

t0k4rt · 2018-02-21T14:49:13Z

@sgotti Hello, I couldn't manage to cherry pick your commit so I included the test manually. I hope it's okay.

sgotti · 2018-02-23T10:33:04Z

@t0k4rt unfortunately you have to rebase you branch on the current master and it'll be better if you squash all your commits in only one commit. If you have some problems doing it let me know.

t0k4rt · 2018-02-26T15:00:54Z

@sgotti I think this should be ok, could you check ?

sgotti · 2018-02-26T15:10:25Z

@t0k4rt can you please squash your commits in an unique one?

t0k4rt · 2018-02-26T15:29:37Z

@sgotti I'm sorry, I merged my branch with your master to resolve conflict and squashed commits with rebase. But there are still conflict. Could you help me ?

sgotti · 2018-02-26T16:00:44Z

@t0k4rt I rebased your branch using git --rebase origin/master, fixed two conflicts and did some minor changes to the doc, some comments and the commit title.

Take it here: sgotti@1f4d429

t0k4rt · 2018-02-26T17:34:36Z

@sgotti Is it better now ? I picked your commit into my branch.

sgotti · 2018-02-26T17:53:31Z

@t0k4rt sorry but it's not just my commit (I see 1157 changed files) so it's not rebased on current origin/master.

You should do something like this:

on your local branch:

git fetch
git reset --hard origin/master

then cherry pick my commit. The if you run git rebase origin/master you shouldn't see any error and change.

Then force push your branch to github to update the PR.

t0k4rt · 2018-02-27T09:25:41Z

@sgotti is it okay now ? I started again from sorintlab/stolon master branch.

sgotti · 2018-02-27T10:17:43Z

@t0k4rt We're almost there.

I just noticed that the new functions in keeper.go are at the end of the file. I'll prefer to move them before but it's not a big deal.

I also see a difference in two comments:

--- a/pkg/cluster/cluster.go
+++ b/pkg/cluster/cluster.go
@@ -187,9 +187,9 @@ type StandbySettings struct {
 type SUReplAccessMode string
 
 const (
-       // Allow access from every host
+       // Allow hba access to all members (0.0.0.0/0)
        SUReplAccessAll SUReplAccessMode = "all"
-       // Allow access from standby server IPs only
+       // Allow hba access to standby server only (strict mode)
        SUReplAccessStrict SUReplAccessMode = "strict"
 )

add a cluster spec `defaultSUReplAccessMode` option that accepts two values: * all (default for backward compatibility) * strict In "strict" mode the generated pg_hba on the master will provide access for superuser and replication connections only from the keepers IPs having a configured standby db. In "all" mode access is granted from every host like always done before this patch.

t0k4rt · 2018-02-27T10:23:38Z

@sgotti I put all *PostgresKeeper functions together and put the right comments. This should look good !

sgotti · 2018-02-27T12:14:53Z

@t0k4rt Great! Thanks a lot of work and patience! Merging.

Feature dynamic hba

t0k4rt · 2018-02-27T12:38:08Z

@sgotti Grazie mille !

t0k4rt force-pushed the feature-dynamic-hba branch 2 times, most recently from aeafb6b to 28caa8f Compare January 15, 2018 18:25

sgotti requested changes Jan 19, 2018

View reviewed changes

t0k4rt force-pushed the feature-dynamic-hba branch from ed08ded to 470480e Compare January 23, 2018 13:43

sgotti requested changes Jan 24, 2018

View reviewed changes

sgotti reviewed Jan 25, 2018

View reviewed changes

sgotti reviewed Jan 26, 2018

View reviewed changes

sgotti requested changes Jan 26, 2018

View reviewed changes

t0k4rt force-pushed the feature-dynamic-hba branch from a31fc9f to c983d48 Compare January 29, 2018 14:13

sgotti requested changes Feb 8, 2018

View reviewed changes

sgotti reviewed Feb 15, 2018

View reviewed changes

t0k4rt force-pushed the feature-dynamic-hba branch from 0a6d0ac to ed2bf96 Compare February 26, 2018 14:43

t0k4rt force-pushed the feature-dynamic-hba branch 2 times, most recently from e976ee8 to b173d62 Compare February 26, 2018 15:27

t0k4rt force-pushed the feature-dynamic-hba branch 2 times, most recently from 9565ae8 to c2db550 Compare February 26, 2018 17:32

t0k4rt force-pushed the feature-dynamic-hba branch 2 times, most recently from 5d8a07f to 8cba671 Compare February 27, 2018 09:18

t0k4rt force-pushed the feature-dynamic-hba branch from 8cba671 to b58363e Compare February 27, 2018 10:21

sgotti approved these changes Feb 27, 2018

View reviewed changes

sgotti merged commit b58363e into sorintlab:master Feb 27, 2018

sgotti added a commit that referenced this pull request Feb 27, 2018

Merge pull request #412 from t0k4rt/feature-dynamic-hba

717a7c2

Feature dynamic hba

sgotti added this to the 0.10.0 milestone Mar 21, 2018

sgotti mentioned this pull request Jun 19, 2018

fixed listen address in SUReplAccessStrict #520

Merged

Feature dynamic hba #412

Feature dynamic hba #412

Uh oh!

Conversation

t0k4rt commented Jan 15, 2018

Uh oh!

t0k4rt commented Jan 15, 2018

Uh oh!

sgotti left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

t0k4rt commented Jan 19, 2018

Uh oh!

sgotti commented Jan 19, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

t0k4rt commented Jan 23, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sgotti left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

t0k4rt commented Jan 25, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sgotti Jan 25, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

t0k4rt commented Jan 25, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

sgotti commented Jan 19, 2018 •

edited

Loading

t0k4rt commented Jan 23, 2018 •

edited

Loading

sgotti Jan 25, 2018 •

edited

Loading

t0k4rt commented Jan 25, 2018 •

edited

Loading

sgotti Jan 26, 2018 •

edited

Loading