lua: add fire-and-forget functionality to http call #10145

mindyor · 2020-02-24T18:38:05Z

Description:
Envoy's Lua function httpCall is synchronous - Envoy waits until the upstream request completes before continuing.
This PR adds an optional flag to httpCall enabling the request to be asynchronous - Envoy does not wait until the upstream service responds, and in fact ignores the response.
httpCall continues to be synchronous by default.

Risk Level: Medium?
Testing: added a unit test and an integration test
Docs Changes: added doc change to lua_filter.rst
Release Notes: added release note to version_history.rst

Signed-off-by: mindyor <or.mindy@gmail.com>

…er constructor Signed-off-by: mindyor <or.mindy@gmail.com>

Signed-off-by: mindyor <or.mindy@gmail.com>

jmarantz · 2020-02-26T19:08:27Z

docs/root/configuration/http/http_filters/lua_filter.rst

+
+.. code-block:: lua
+
+  headers, body = handle:httpCallAsync(cluster, headers, body, timeout)


drive-by comment; I know nothing about Lua but I would've expected, and would hope you would not return headers/body from an async function, but would instead have the user supply another optional argument, which would be some sort of closure or lambda to run when the async fetch was complete, passing the headers and body to that function.

Even better, I think 2 lambdas should be supplied: one that is passed the headers when available. The second is called with body-chunks, until there are no more. It would receive a bool param that would be 'true' when the request is complete.

Actually I haven't looked deeply at the Envoy infrastructure for HTTP Fetches, but modeling the Lua interface based on that pattern (which might differ from my suggestion above) would be even better.

The phrase 'fire and forget' doesn't seem good to me from a resource usage perspective.

Ah, you're correct, this does not return headers/body. Pushed a change. Thanks for noticing that!

Resource/memory wise, it looks like there are checks in the pipeline for this (I ran into it in previous pushes, and resolved them).

@jmarantz and I talked about this offline. We will rename this httpCallNonblocking and proceed without callbacks.

Signed-off-by: mindyor <or.mindy@gmail.com>

mindyor · 2020-02-27T00:25:49Z

/retest

repokitteh-read-only · 2020-02-27T00:25:53Z

🔨 rebuilding ci/circleci: coverage (failed build)

🐱

Caused by: a #10145 (comment) was created by @mindyor.

see: more, trace.

Signed-off-by: mindyor <or.mindy@gmail.com>

dio

Thanks for working on this! Overall it is good. Flushing some comments, mostly nits.

dio · 2020-02-27T22:06:32Z

source/extensions/filters/http/lua/lua_filter.h

@@ -285,6 +297,38 @@ class StreamHandleWrapper : public Filters::Common::Lua::BaseLuaObject<StreamHan
  State state_{State::Running};
  std::function<void()> yield_callback_;
  Http::AsyncClient::Request* http_request_{};
+  FireAndForgetWriter* fireAndForgetWriter_;


WDYT of making this a unique_ptr?

dio · 2020-02-27T22:07:37Z

source/extensions/filters/http/lua/lua_filter.h

+};
+
+/**
+ * Implementation that takes incoming requests and implements


Sorry, could you expand this a little bit? I'm a bit confused by "takes incoming requests".

dio · 2020-02-27T22:08:42Z

source/extensions/filters/http/lua/lua_filter.h

+};
+
+/**
+ * A class with shared code for building and making http calls


A super nit,

Suggested change

* A class with shared code for building and making http calls

* A class with shared code for building and making HTTP calls.

dio · 2020-02-27T22:09:35Z

test/extensions/filters/http/lua/lua_integration_test.cc

@@ -355,6 +355,52 @@ name: envoy.lua
  EXPECT_EQ("nope", response->body());
 }

+// Upstream fire and forget async call


Suggested change

// Upstream fire and forget async call

// Upstream fire-and-forget asynchronous call.

dio · 2020-02-27T22:09:52Z

test/extensions/filters/http/lua/lua_filter_test.cc

@@ -799,6 +799,59 @@ TEST_F(LuaHttpFilterTest, HttpCall) {
  callbacks->onSuccess(std::move(response_message));
 }

+// Basic Asynchronous, Fire and Forget HTTP request flow.


Suggested change

// Basic Asynchronous, Fire and Forget HTTP request flow.

// Basic asynchronous, fire-and-forget HTTP request flow.

dio · 2020-02-27T22:14:53Z

source/extensions/filters/http/lua/lua_filter.cc

+int FireAndForgetWriter::luaHttpCallNonblocking(lua_State* state) {
+  if (LuaFilterUtil::makeHttpCall(state, filter_, *this)) {
+    return 0;
+  } else {


Superfluous else? We can remove this, but I don't feel strongly about it.

dio · 2020-02-27T22:20:48Z

source/extensions/filters/http/lua/lua_filter.h

+ * Implementation that takes incoming requests and implements
+ * "fire and forget" behavior using an async client.
+ */
+class FireAndForgetWriter : public Filters::Common::Lua::BaseLuaObject<FireAndForgetWriter>,


Since this writer currently is an HTTP async client "writer". Can we name it as FireAndForgetHttpWriter or HttpAsyncClientWriter or similar?

Signed-off-by: mindyor <or.mindy@gmail.com>

snowp

This looks like a great start, thanks for working on this!

snowp · 2020-02-28T02:48:50Z

docs/root/configuration/http/http_filters/lua_filter.rst

+
+  handle:httpCallNonblocking(cluster, headers, body, timeout)
+
+Makes an HTTP call to an upstream host. Same behavior as httpCall, except that Envoy will fire and forget.


I would suggest spelling out what fire and forget means here wrt continuation of the filter chain

snowp · 2020-02-28T03:04:18Z

source/extensions/filters/http/lua/lua_filter.cc

@@ -23,7 +23,10 @@ StreamHandleWrapper::StreamHandleWrapper(Filters::Common::Lua::Coroutine& corout
        if (state_ == State::Running) {
          throw Filters::Common::Lua::LuaException("script performed an unexpected yield");
        }
-      }) {}
+      }),
+      fireAndForgetHttpWriter_(new FireAndForgetHttpWriter(filter)) {}


no reason to heap allocate this, just use it as a non-pointer value

FireAndForgetHttpWriter fire_and_forget_http_writer_;

and just fire_and_forget_http_writer_(filter) in the constructor

This actually does need to be heap allocated because the success/failure callbacks can live beyond the filter, but not in the way it is done here. The Lua filter config needs to have a fire and forget writer that works identically to the router's shadow writer. Take a look at the pattern that is used there.

(Or yes per other comment just pass static callbacks and that should be fine also).

snowp · 2020-02-28T03:04:51Z

source/extensions/filters/http/lua/lua_filter.cc

+      }),
+      fireAndForgetHttpWriter_(new FireAndForgetHttpWriter(filter)) {}
+
+StreamHandleWrapper::~StreamHandleWrapper() { delete fireAndForgetHttpWriter_; }


if you are going to be doing heap allocations please use std::unique_ptr/std::shared_ptr to manage the memory

snowp · 2020-02-28T03:06:45Z

source/extensions/filters/http/lua/lua_filter.cc

+
+Http::AsyncClient::Request*
+LuaFilterUtil::makeHttpCall(lua_State* state, Filter& filter,
+                            Http::AsyncClient::Callbacks& callbacksListener) {


nit: callbacksListener -> callbacks (or at the very least snake case parameter names)

snowp · 2020-02-28T03:09:32Z

source/extensions/filters/http/lua/lua_filter.cc

@@ -485,6 +430,81 @@ int StreamHandleWrapper::luaImportPublicKey(lua_State* state) {
  return 1;
 }

+FireAndForgetHttpWriter::FireAndForgetHttpWriter(Filter& filter) : filter_(filter) {}


i don't think this class buys us a whole lot, how about just having a NoopCallbacks that we pass to makeHttpCall?

Good idea. Pushed.

snowp · 2020-02-28T03:11:36Z

source/extensions/filters/http/lua/lua_filter.h

+/**
+ * A class with shared code for building and making HTTP calls.
+ */
+class LuaFilterUtil : public Filters::Common::Lua::BaseLuaObject<LuaFilterUtil> {


Instead of making these utility functions exposed outside of this class/translation unit i'd suggest putting them in the anonymous namespace in the .cc file. This tends to be the preferred way of doing helper functions that are only used in one .cc file and that doesn't benefit from being a member function

you can look for namespace { in existing code to see what an anonymous namespace looks like

snowp · 2020-02-28T03:12:28Z

test/extensions/filters/http/lua/lua_integration_test.cc

+
+  waitForNextUpstreamRequest();
+
+  upstream_request_->encodeHeaders(default_response_headers_, true);


i know this is tested in the unit test but it would be nice to assert that the headers/body of the upstream request matches what we specified in the call

snowp · 2020-02-28T03:13:34Z

source/extensions/filters/http/lua/lua_filter.h

+class FireAndForgetHttpWriter : public Filters::Common::Lua::BaseLuaObject<FireAndForgetHttpWriter>,
+                                public Http::AsyncClient::Callbacks {
+public:
+  FireAndForgetHttpWriter(Filter& filter);


use explicit for single arg ctors: https://google.github.io/styleguide/cppguide.html#Implicit_Conversions

snowp · 2020-02-28T03:28:24Z

docs/root/configuration/http/http_filters/lua_filter.rst

@@ -244,6 +244,23 @@ data to send. *timeout* is an integer that specifies the call timeout in millise
 Returns *headers* which is a table of response headers. Returns *body* which is the string response
 body. May be nil if there is no body.

+httpCallNonblocking()


i'm not a huge fan of this name (it sounds a bit clunky/verbose) and it worries me a bit if we are going to end up with one new function per variation of a HTTP call (httpCallNonblockingWithCallbacks kinda situation)

are we able to/can we add parameters to existing LUA filters? it'd be nice to either have one or two (e.g. httpCall and asyncHttpCall) functions with options for whether the response should be dropped, callbacks to invoke on response etc.

@mattklein123 @jmarantz @dio

Are we able to/can we add parameters to existing LUA filters?

I think we can, yeah. This is a reasonable/great ask, yeah we should do this.

Yes I think we can do this and SGTM

Makes sense.

Removed the new function, adding a parameter asynchronous. Hope that looks cleaner!

Signed-off-by: mindyor <or.mindy@gmail.com>

snowp

Thanks for iterating on this, its looking pretty good

snowp · 2020-03-03T01:20:03Z

source/extensions/filters/http/lua/lua_filter.cc

@@ -23,7 +90,10 @@ StreamHandleWrapper::StreamHandleWrapper(Filters::Common::Lua::Coroutine& corout
        if (state_ == State::Running) {
          throw Filters::Common::Lua::LuaException("script performed an unexpected yield");
        }
-      }) {}
+      }),
+      noopCallbacks_(new NoopCallbacks()) {}


Since NoopCallbacks doesn't hold any state, I think it would be clearer to define it as a ConstSingleton and just pass the reference to the singleton object. That way we avoid having to deal with this memory management.

vc'd. created static object instead.

snowp · 2020-03-03T01:21:16Z

source/extensions/filters/http/lua/lua_filter.h

+   * @param 4 (int): Timeout in milliseconds for the call.
+   * @return headers (table), body (string/nil)
+   */
+  DECLARE_LUA_FUNCTION(StreamHandleWrapper, luaHttpCallBlocking);


is still necessary now that we're using the parameterized function instead? as in, they don't need to be LUA functions

snowp · 2020-03-03T01:22:02Z

source/extensions/filters/http/lua/lua_filter.cc

@@ -485,6 +507,8 @@ int StreamHandleWrapper::luaImportPublicKey(lua_State* state) {
  return 1;
 }

+NoopCallbacks::NoopCallbacks() {}


no reason to explicitly define this, its the same as the default constructor

snowp · 2020-03-03T01:24:54Z

source/extensions/filters/http/lua/lua_filter.h

@@ -156,10 +159,31 @@ class StreamHandleWrapper : public Filters::Common::Lua::BaseLuaObject<StreamHan
   * @param 2 (table): A table of HTTP headers. :method, :path, and :authority must be defined.
   * @param 3 (string): Body. Can be nil.
   * @param 4 (int): Timeout in milliseconds for the call.
+   * @param 5 (bool): Flag. If true, filter continues without waiting for HTTP response from upstream service.


this is optional right? probably want to explicitly specify the default behavior

snowp · 2020-03-03T01:28:39Z

source/extensions/filters/http/lua/lua_filter.cc

-  if (timeout_ms > 0) {
-    timeout = std::chrono::milliseconds(timeout_ms);
+  bool asynchronous = lua_toboolean(state, 6);
+  if(asynchronous) {


it'd nice if we're consistent with naming here, something like

if (async) { asyncHttpCall(state); } else { syncHttpCall(state); }

instead of using blocking/async interchangeably. it'd be nice if we had this consistency both here and elsewhere (e.g. in the tests)

snowp · 2020-03-03T01:38:04Z

docs/root/intro/version_history.rst

@@ -9,6 +9,7 @@ Version history
  minimum.
 * config: use type URL to select an extension whenever the config type URL (or its previous versions) uniquely identify a typed extension, see :ref:`extension configuration <config_overview_extension_configuration>`.
 * http: fixing a bug in HTTP/1.0 responses where Connection: keep-alive was not appended for connections which were kept alive.
+* lua: added `httpCallNonblocking` function. Same as `httpCall` but fire-and-forget.


this needs to be updated

snowp · 2020-03-03T01:40:11Z

test/extensions/filters/http/lua/lua_filter_test.cc

+  Http::TestRequestTrailerMapImpl request_trailers{{"foo", "bar"}};
+  EXPECT_EQ(Http::FilterTrailersStatus::Continue, filter_->decodeTrailers(request_trailers));
+
+  Http::ResponseMessagePtr response_message(new Http::ResponseMessageImpl(


this seems unused?

snowp · 2020-03-03T01:40:43Z

test/extensions/filters/http/lua/lua_filter_test.cc

@@ -799,6 +799,123 @@ TEST_F(LuaHttpFilterTest, HttpCall) {
  callbacks->onSuccess(std::move(response_message));
 }

+// Basic HTTP request flow. Async flag set to false


nit: proper punctuation in comments (end with period)

…e async/sync over blocking terminology Signed-off-by: mindyor <or.mindy@gmail.com>

Signed-off-by: mindyor <or.mindy@gmail.com>

snowp

Thanks, this is really close to done imo

snowp · 2020-03-04T01:35:18Z

docs/root/intro/version_history.rst

@@ -20,6 +20,7 @@ Version history
 * listener filters: listener filter extensions use the "envoy.filters.listener" name space. A
  mapping of extension names is available in the :ref:`deprecated <deprecated>` documentation.
 * listeners: fixed issue where :ref:`TLS inspector listener filter <config_listener_filters_tls_inspector>` could have been bypassed by a client using only TLS 1.3.
+* lua: added optional asynchronous behavior `httpCall` function. Makes fire-and-forget call upstream- ignores response.


i'd do something like * lua: added a parameter to `httpCall` that makes it possible to have the call be asynchronous.

snowp · 2020-03-04T01:35:56Z

source/extensions/filters/http/lua/lua_filter.cc

-  absl::optional<std::chrono::milliseconds> timeout;
-  if (timeout_ms > 0) {
-    timeout = std::chrono::milliseconds(timeout_ms);
+  bool asynchronous = lua_toboolean(state, 6);


nit: const bool

snowp · 2020-03-04T01:38:08Z

test/extensions/filters/http/lua/lua_integration_test.cc

+
+  waitForNextUpstreamRequest();
+
+  EXPECT_THAT(lua_request_->headers(), HeaderValueOf(Http::Headers::get().Method, "POST"));


id move these next to the above ASSERT_TRUEs. also maybe a small comment like // Sanity checking that we sent the expected data.

Signed-off-by: mindyor <or.mindy@gmail.com>

dio · 2020-03-04T05:23:06Z

source/extensions/filters/http/lua/lua_filter.cc

-  absl::optional<std::chrono::milliseconds> timeout;
-  if (timeout_ms > 0) {
-    timeout = std::chrono::milliseconds(timeout_ms);
+  const bool asynchronous = lua_toboolean(state, 6);


Should we test with lua_isboolean here? Since lua_toboolean returns 1 for any Lua value different from false and nil. E.g. typo like faIse (yes that's capital "I") can turn it out to be true. WDYT?

Fair point. I'll add that to my next push.

Signed-off-by: mindyor <or.mindy@gmail.com>

snowp

LGTM, thanks for adding this!

@dio any more feedback?

@mattklein123 Wanna take a look as well? In particular the way the API turned out

mindyor · 2020-03-05T01:50:07Z

/retest

repokitteh-read-only · 2020-03-05T01:50:10Z

🔨 rebuilding ci/circleci: coverage (failed build)

🐱

Caused by: a #10145 (comment) was created by @mindyor.

see: more, trace.

mindyor · 2020-03-05T17:50:43Z

/retest to understand the coverage job.

repokitteh-read-only · 2020-03-05T17:50:47Z

🔨 rebuilding ci/circleci: coverage (failed build)

🐱

Caused by: a #10145 (comment) was created by @mindyor.

see: more, trace.

mattklein123 · 2020-03-05T17:52:25Z

Coverage is broken right now and being worked on. Please stop retesting. I will take a look at the PR soon.

mattklein123

Nice, great work! Just a few nits and LGTM! Thank you!

/wait

mattklein123 · 2020-03-05T18:05:06Z

source/extensions/filters/http/lua/lua_filter.cc

-  if (body != nullptr) {
-    message->body() = std::make_unique<Buffer::OwnedImpl>(body, body_size);
-    message->headers().setContentLength(body_size);
+  const int asyncFlagIndex = 6;


nit: async_flag_index

mattklein123 · 2020-03-05T18:06:26Z

source/extensions/filters/http/lua/lua_filter.h

@@ -86,6 +86,8 @@ class FilterCallbacks {

 class Filter;

+class NoopCallbacks;


I don't think this needs forward declaration?

mattklein123 · 2020-03-05T18:06:57Z

source/extensions/filters/http/lua/lua_filter.h

+public:
+  // Http::AsyncClient::Callbacks
+  void onSuccess(Http::ResponseMessagePtr&&) override {}
+


nit: del newline

Signed-off-by: mindyor <or.mindy@gmail.com>

mattklein123

Thanks!

mindyor added 5 commits February 23, 2020 12:02

extract http call functions into a new class, LuaFilterLibrary

871ee1b

Signed-off-by: mindyor <or.mindy@gmail.com>

create fire and forget listener; lua function httpCallAsync

df9df09

Signed-off-by: mindyor <or.mindy@gmail.com>

group class functions together

541bc4b

Signed-off-by: mindyor <or.mindy@gmail.com>

add documentation for httpCallAsync

8b5b84d

Signed-off-by: mindyor <or.mindy@gmail.com>

add release note to version history

323d728

Signed-off-by: mindyor <or.mindy@gmail.com>

dio self-assigned this Feb 24, 2020

mindyor added 8 commits February 24, 2020 20:32

formatting

e892eaa

Signed-off-by: mindyor <or.mindy@gmail.com>

naming. s/LuaFilterLibrary/LuaFilterUtil/

29464ba

Signed-off-by: mindyor <or.mindy@gmail.com>

formatting: lengthen underline

b566eab

Signed-off-by: mindyor <or.mindy@gmail.com>

move instantiation of fire and forget writer into stream handle wrapp…

4cb0ecc

…er constructor Signed-off-by: mindyor <or.mindy@gmail.com>

makeHttpCall static function

27a0f3c

Signed-off-by: mindyor <or.mindy@gmail.com>

whitespace

11c1301

Signed-off-by: mindyor <or.mindy@gmail.com>

alphabetic order

21aa51e

Signed-off-by: mindyor <or.mindy@gmail.com>

delete fire and forget writer in destructor

bf9998d

Signed-off-by: mindyor <or.mindy@gmail.com>

mindyor changed the title ~~WIP - lua: create async http call~~ lua: create async http call Feb 26, 2020

jmarantz reviewed Feb 26, 2020

View reviewed changes

correct docs: httpCallAsync return

3e0f6c4

Signed-off-by: mindyor <or.mindy@gmail.com>

mindyor changed the title ~~lua: create async http call~~ lua: create nonblocking http call Feb 27, 2020

rename functions Nonblocking as implementation does not use callbacks

38a8a99

Signed-off-by: mindyor <or.mindy@gmail.com>

dio reviewed Feb 27, 2020

View reviewed changes

mindyor added 5 commits February 27, 2020 14:33

rename FireAndForgetWriter to FireAndForgetHttpWriter

67aee6e

Signed-off-by: mindyor <or.mindy@gmail.com>

comment nits

e5bee45

Signed-off-by: mindyor <or.mindy@gmail.com>

comment clarity for fire and forget http wrier

ac562dd

Signed-off-by: mindyor <or.mindy@gmail.com>

remove superfluous else

aa04eb8

Signed-off-by: mindyor <or.mindy@gmail.com>

formatting

d838492

Signed-off-by: mindyor <or.mindy@gmail.com>

snowp suggested changes Feb 28, 2020

View reviewed changes

mindyor added 2 commits February 28, 2020 13:13

put helper methods in anonymous namespace

7b234f3

Signed-off-by: mindyor <or.mindy@gmail.com>

rename callbacksListener to callbacks

f4cd746

Signed-off-by: mindyor <or.mindy@gmail.com>

test covering async = false case

5ad50d1

Signed-off-by: mindyor <or.mindy@gmail.com>

snowp suggested changes Mar 3, 2020

View reviewed changes

mindyor added 6 commits March 2, 2020 21:36

remove lua declaration of functions; update naming and comments to us…

55d1682

…e async/sync over blocking terminology Signed-off-by: mindyor <or.mindy@gmail.com>

Merge branch 'master' into async-lua-http-call

1ebaac1

use static noopcallback - thanks snowp

f279ad2

Signed-off-by: mindyor <or.mindy@gmail.com>

remove unused lines in test

1594207

Signed-off-by: mindyor <or.mindy@gmail.com>

integration test assert on header values - thanks snowp

9945ac5

Signed-off-by: mindyor <or.mindy@gmail.com>

comment tweak

1d4ba85

Signed-off-by: mindyor <or.mindy@gmail.com>

mindyor changed the title ~~lua: create nonblocking http call~~ lua: create asynchronous http call Mar 3, 2020

mindyor changed the title ~~lua: create asynchronous http call~~ lua: add fire-and-forget functionality to http call Mar 3, 2020

snowp suggested changes Mar 4, 2020

View reviewed changes

pr comments

6dcdcf5

Signed-off-by: mindyor <or.mindy@gmail.com>

dio reviewed Mar 4, 2020

View reviewed changes

mindyor added 2 commits March 4, 2020 12:25

type check for async flag

8459416

Signed-off-by: mindyor <or.mindy@gmail.com>

formatting

d190e36

Signed-off-by: mindyor <or.mindy@gmail.com>

snowp previously approved these changes Mar 4, 2020

View reviewed changes

mattklein123 self-assigned this Mar 4, 2020

mattklein123 requested changes Mar 5, 2020

View reviewed changes

repokitteh-read-only bot added the waiting label Mar 5, 2020

review feedback nits

9a1dbd7

Signed-off-by: mindyor <or.mindy@gmail.com>

mindyor dismissed snowp’s stale review via 9a1dbd7 March 5, 2020 19:15

repokitteh-read-only bot removed the waiting label Mar 5, 2020

mattklein123 approved these changes Mar 5, 2020

View reviewed changes

mattklein123 merged commit 6e9ed57 into envoyproxy:master Mar 5, 2020


		.. code-block:: lua

		headers, body = handle:httpCallAsync(cluster, headers, body, timeout)

	* A class with shared code for building and making http calls
	* A class with shared code for building and making HTTP calls.

	// Upstream fire and forget async call
	// Upstream fire-and-forget asynchronous call.

	// Basic Asynchronous, Fire and Forget HTTP request flow.
	// Basic asynchronous, fire-and-forget HTTP request flow.


		handle:httpCallNonblocking(cluster, headers, body, timeout)

		Makes an HTTP call to an upstream host. Same behavior as httpCall, except that Envoy will fire and forget.


		waitForNextUpstreamRequest();

		upstream_request_->encodeHeaders(default_response_headers_, true);


		waitForNextUpstreamRequest();

		EXPECT_THAT(lua_request_->headers(), HeaderValueOf(Http::Headers::get().Method, "POST"));

		@@ -86,6 +86,8 @@ class FilterCallbacks {

		class Filter;

		class NoopCallbacks;

lua: add fire-and-forget functionality to http call #10145

lua: add fire-and-forget functionality to http call #10145

Uh oh!

Conversation

mindyor commented Feb 24, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mindyor commented Feb 27, 2020

Uh oh!

repokitteh-read-only bot commented Feb 27, 2020

Uh oh!

dio left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

snowp left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dio Feb 28, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mindyor Mar 3, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

snowp left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mindyor commented Feb 24, 2020 •

edited

Loading

dio Feb 28, 2020 •

edited

Loading

mindyor Mar 3, 2020 •

edited

Loading