multi: implement a market-maker bot #1738

buck54321 · 2022-07-27T21:46:46Z

core:
Add a market-maker bot to **core**. The bot uses some internal and
(optionally) external signals to calculate a ideal buy and sell
price and a "break-even spread", which is the spread at which
a buy-sell sequence's tx fees equal its profit. These values can
be used as inputs into one of five "gap strategies", which determine
the target spread.

The bot can be created, started, updated, paused, and retired. Each
of those actions generates a notification.

db:
New BotProgram type is not specific to market-maker bots. Everything
is structured with a mind towards future expansion with new automated
trading routines.

ui:
New /mm view enables creation, re-configuration, and monitoring of
existing market-maker bots.

buck54321 · 2022-07-27T21:52:03Z

buck54321 · 2022-08-13T12:53:01Z

I've made the simnet test into a full-fledged cli bot. This means I could've foregone the simharness package move, but I do plan to use that package other places, so I'd love to keep that change.

client/db/bolt/db.go

client/cmd/mmbot/main.go

chappjc · 2022-08-13T20:34:55Z

I've made the simnet test into a full-fledged cli bot.

Glad to see it.

This means I could've foregone the simharness package move, but I do plan to use that package other places, so I'd love to keep that change.

In that case, will you please put the simharness.go creation and the loadbot changes in a separate commit? There's no clear connection to the mm bot work.

chappjc · 2022-08-13T20:39:14Z

client/core/bookie.go

@@ -385,7 +385,7 @@ func (dc *dexConnection) bookie(marketID string) *bookie {
 // syncBook subscribes to the order book and returns the book and a BookFeed to
 // receive order book updates. The BookFeed must be Close()d when it is no
 // longer in use. Use stopBook to unsubscribed and clean up the feed.
-func (dc *dexConnection) syncBook(base, quote uint32) (BookFeed, error) {
+func (dc *dexConnection) syncBook(base, quote uint32) (*orderbook.OrderBook, BookFeed, error) {


This is funny because originally we have the *OrderBook returned, then we removed it and had the orderbook snap as the first send on the feed (FreshBookAction I believe). It is certainly most convenient this way.

client/core/core.go

buck54321 · 2022-08-13T21:16:51Z

client/core/bot.go

+func basisPrice(host string, mkt *Market, oracleBias, oracleWeighting float64, midGap uint64, bp basisPricer, log dex.Logger) uint64 {
+	basisPrice := float64(midGap) // float64 message-rate units
+
+	log.Tracef("basisPrice: mid-gap price = %d", midGap)
+


We should be a little more careful with using the current mid-gap here. If the book is sparse, someone could manipulate the bot into providing a favorable price by placing orders far from the gap. We should try to get some metric of order book health before using the mid-gap as an input, and we should probably only allow the mid-gap to vary so much from the oracle rate.

martonp · 2022-08-14T05:30:37Z

Looks really cool. I'm just starting to look through, but I found an issue.. if you press enter instead of clicking the submit button when creating a bot, it will create two bots instead of just one. If you click the button, only one button is created.

Also, I created a bot, and I have funds in both wallets, but it is only creating orders on the buy side, not the sell side. Isn't it supposed to create for both?

Also, there should be a header on that table with the CEX info. And a K after the volume values.

buck54321 · 2022-08-15T02:26:44Z

if you press enter instead of clicking the submit button when creating a bot, it will create two bots instead of just one.

When you say "press enter", do you mean in the lot number input? I've tried on Chrome and Firefox, and can't seem to repro, which makes sense because it's not part of a form and there's no handler on that input to submit.

Maybe I've misunderstood. Please verify.

but it is only creating orders on the buy side

Not really sure about this, but I squashed some minor bugs that may have contributed. If you still see it, please run with
--log=BOT=trace,CORE=trace
and see if anything seems off.

martonp · 2022-08-15T06:31:33Z

When you say "press enter", do you mean in the lot number input? I've tried on Chrome and Firefox, and can't seem to repro, which makes sense because it's not part of a form and there's no handler on that input to submit.

I meant press enter when submitting the password.

I'll check about the other issue later.

chappjc

A first pass.

The new core/simharness package looks like a cool reusability refactor of some of the code in loadbot and elsewhere, but I don't see how it's related to this PR. Shouldn't it be delayed for when we're ready to put it into action in loadbot and eth/node_harness_test.go?

The cmd/mmbot app looks fine. I'm fairly certain that in the near future we'll want to hack in some paths for automated init/wallet-creation/register to suit existing workflows.
Also, it looks like it won't be a problem to start multiple bot programs within the one app, but if I'm not seeing an possible issue with that, let me know.

client/webserver/locales/en-us.go

client/asset/btc/btc.go

client/core/simharness/simharness.go

chappjc · 2022-09-14T19:35:41Z

client/cmd/mmbot/main.go

+var (
+	userDir, _            = os.UserHomeDir()
+	defaultConfigFileName = "mm.json"
+	defaultCoreDir        = filepath.Join(userDir, ".dexc")


Why not dcrutil.AppData? Using /.dexc isn't going to jive with Windows and probably macos

client/cmd/mmbot/main.go

JoeGruffins

I was able to get banned on btc/dcr on simnet because the btc native wallet was too slow seemingly and missed a lot of redemptions. But anyway, still says running after I was banned.

The Oracle Bias slider is visible when weight is zero if you move the bar on another bot and then go to a bot with zero. Probably unintended.

Working well and looks good to me.

client/cmd/mmbot/main.go

JoeGruffins · 2022-09-14T07:44:15Z

client/cmd/mmbot/main.go

+	signal.Notify(killChan, os.Interrupt)
+	go func() {
+		for range killChan {
+			if c != nil && pgmID > 0 {


These two checks look like a race.

client/core/bot.go

JoeGruffins · 2022-09-15T07:52:19Z

client/core/exchangeratefetcher.go

+	return strings.ToLower(strings.ReplaceAll(slug, " ", "-"))
+}
+
+func getInto(ctx context.Context, url string, thing interface{}) error {


Is this method name correct as get into?

client/core/simharness/simharness.go

JoeGruffins · 2022-09-16T04:30:03Z

client/asset/btc/btc.go

@@ -1563,15 +1577,9 @@ func (btc *baseWallet) PreSwap(req *asset.PreSwapForm) (*asset.PreSwap, error) {
 	}


Unrelated to these changes but there is an odd fmt.Printf that should maybe be a log.

btc.go:1397: fmt.Printf("btc.apiFeeFallback: %+v\n\n", btc.apiFeeFallback())

client/webserver/site/src/js/mm.ts

JoeGruffins · 2022-09-16T06:05:05Z

client/core/bot.go

+	GapStrategyAbsolutePlus GapStrategy = "absolute-plus"
+	// GapStrategyPercent sets the spread as a ratio of the mid-gap rate.
+	// 0 <= r <= 0.1
+	GapStrategyPercent GapStrategy = "percent"
+	// GapStrategyPercentPlus sets the spread as a ratio of the mid-gap rate
+	// plus the break-even gap.
+	GapStrategyPercentPlus GapStrategy = "percent-plus"


I don't have a great grasp on what is needed here, but seems like Plus categories could just be an option and don't need separate strategies. Also, why add the break-even? Could set to break even if gap is lower also work?

Asking for my understanding.

Gaps above the break-even are "profitable". A gap below the break-even guarantees losses. Having break-even as a minimum allows the bot to adapt to changing market conditions, particularly changes in tx fee rates, without user input.

Could set to break even if gap is lower also work?

I'm not sure I understand this question.

I'm not sure I understand this question.

If break even is .07 and the program would set to .1 adding sets it to .17 but really .1 was already enough to bring it above.

buck54321 · 2022-09-22T18:33:18Z

I thought I had deleted the simharness stuff already, but I guess I never git rm'd. It's gone now.

buck54321 · 2022-09-22T18:48:38Z

The cmd/mmbot app looks fine. I'm fairly certain that in the near future we'll want to hack in some paths for automated init/wallet-creation/register to suit existing workflows.
Also, it looks like it won't be a problem to start multiple bot programs within the one app, but if I'm not seeing an possible issue with that, let me know.

I was actually kinda hoping you would iterate on the cli app after this. Here are the three big enhancements I'd like to see going forward.

CLI App
Inventory management
Arbitrage

all of which I think deserve their own dedicated efforts.

chappjc · 2022-09-22T18:52:23Z

I was actually kinda hoping you would iterate on the cli app after this.

Gladly.

Here are the three big enhancements I'd like to see going forward.

Yes, no question. We need these, particularly arb.

martonp

Looking really nice.

A few UI issues:

After pausing a bot, it still shows running
The settings icon on the top right corner is not visible when starting the app. If I type /mm into the address bar, the settings icon shows up.
The drift tolerance minimum is actually 0.1%, but you can enter 0 in the option selector.

martonp · 2022-10-16T13:15:48Z

client/core/bot.go

+
+// validateProgram checks the sensibility of a *MakerProgram's values and sets
+// some defaults.
+func validateProgram(pgm *MakerProgram) error {


EmptyMarketRate is not checked in validateProgram.

We just can't know here whether it's required or not. For many or most markets, it probably isn't needed.

martonp · 2022-10-16T13:21:52Z

client/core/bot.go

+//  5. Calculate how many lots are needed to be ordered in order to meet the
+//     2 x L commitment. If low balance restricts the maintenance of L lots on
+//     one side, allow the difference in lots to be added to the opposite side.
+//  6. Place orders, cancels first, then buys and sells.


Does the order that they are placed matter if all of the orders are shuffled while matching?

This was more about just mapping out the bot game plan. You're correct that we can't control the matching sequence of orders placed in the same epoch.

client/core/bot.go

martonp · 2022-10-16T13:45:44Z

client/core/bot.go

+	log    dex.Logger
+	book   *orderbook.OrderBook
+
+	running uint32


// atomic

martonp · 2022-10-16T13:45:54Z

client/core/bot.go

+	wg      sync.WaitGroup
+	die     context.CancelFunc
+
+	rebalanceRunning uint32


// atomic

martonp · 2022-10-16T14:23:58Z

client/core/bot.go

+			basisPrice = msgOracleRate
+			log.Tracef("basisPrice: using basis price %.0f from oracle because no mid-gap was found in order book", basisPrice)
+		} else if oracleWeighting > 0 {
+			basisPrice = msgOracleRate*oracleWeighting + basisPrice*(1-oracleWeighting)


If the oracle price is completely incorrect due to some bug with the oracle, the basisPrice would be way off here as well, potentially creating some orders that would lose a lot of money. Maybe there should be a setting to not place any orders in case of a big difference between the mid-gap and the oracle price? Also it would be good to exclude outlier oracle prices.

What is the reason for using an oracle price other than making sure that the order book is not being manipulated? If this bot isn't being used to do arbitrage with another exchange, why should the price that is used by the bot be adjusted based on some external price?

What is the reason for using an oracle price other than making sure that the order book is not being manipulated?

For a low-depth book, it would be super easy to drain a bot at a favorable price by manipulating the mid-gap rate.

So the oracle weight makes it harder to do this type of manipulation, but if someone is able to do a major manipulation, then even a 50% weight wouldn't save the victim bots. I think it would make sense to have the oracle weight combined with another setting that says if the difference between the mid-gap rate and the oracle rate is greater than some percentage, then don't place any orders.

I think I've captured at least part of what you're asking with maxOracleMismatch, but I'm not disallowing ordering, because that could basically brick the bot e.g. if the book is empty expect for a single order way off-market. Instead, I'm setting a limit to the difference, and using using an adjusted mid-gap if the difference is too large. In practice, a user should use a very high if not 100% oracle weighting when the book is spOracleBiasarse, but this feature should protect the user regardless.

Some other features that I think would be good for followup.

Using 24-hour volume for price-weighting is kinda hacky. We could improve it by, for example, creating a metric for book depth near market price and using that value instead or mixing it in.

Outlier filtering, as you mentioned.

Automatic oracle weighting based on which markets are busy and which are dead. If DEX is super-busy, we need less oracle weighting.

I think I've captured at least part of what you're asking with maxOracleMismatch

Ah yeah that's right.

Another feature could be for the bot to turn off if it keeps losing money. I'd definitely have more peace of mind to leave it on after going to sleep like that.

martonp · 2022-10-16T17:07:35Z

client/core/bot.go

+	var halfSpread uint64
+	switch pgm.GapStrategy {
+	case GapStrategyMultiplier:
+		halfSpread = uint64(math.Round(float64(breakEven) * (1 + pgm.GapFactor)))


Why is this 1 + pgm.GapFactor? It seems correct if it is exactly break even when the GapFactor is 1.

martonp

I think it's looking really good. I see no issues when the bot is trading against the loadbot.

I think it would be nice to have a report of all of a bot's actions and how it performed. Also, something like the loadbot that simulates different market conditions would be cool.

martonp · 2022-10-29T20:48:29Z

docs/images/break-even-half-gap.tex

+In a buy\textrightarrow sell sequence with a positive gap, we end up with more of the quote asset and less of the base asset. Setting our loss ratio equal to our profit ratio, we get
+
+\[
+\frac{F_B}{L} = \frac{p_sL - p_bL - F_Q}{p_bL}


Nbd, but it seems more intuitive to start with (Fb * pb) + Fq = psL - pbL, saying the fees are equal to the profits.

client/core/bot.go

martonp · 2022-10-29T21:52:32Z

client/core/bot.go

+	} else {
+		m.log.Errorf("Bot MaxBuy error: %v", err)
+	}
+	maxSell, err := c.MaxSell(pgm.Host, pgm.BaseID, pgm.QuoteID)


What if while a bot is being created, another bot has not yet placed its orders? Then maybe there won't be enough funds for both bots to place all the orders they need.

I think this should be part of a larger effort to handle "inventory management". Essentially, bots need to have have limits to how much funds they can access. I think Core will unfortunately need to be involved and it might be a pita.

Yeah I was thinking the same thing, definitely sounds like a pita.

client/core/bot.go

martonp · 2022-10-30T16:57:30Z

client/core/bot.go

+type Spreader func(ctx context.Context, baseSymbol, quoteSymbol string) (sell, buy float64, err error)
+
+var spreaders = map[string]Spreader{
+	"binance.com":  fetchBinanceSpread,


Should we add OKEX and gate.io? They are no.2 and no.3 for DCR volume.

Pls skip gate, not hi fidelity numbers.

Not adding any more here, but can do after.

core: Add a market-maker bot to **core**. The bot uses some internal and (optionally) external signals to calculate a ideal buy and sell price and a "break-even spread", which is the spread at which a buy-sell sequence's tx fees equal its profit. These values can be used as inputs into one of five "gap strategies", which determine the target spread. The bot can be created, started, updated, paused, and retired. Each of those actions generates a notification. db: New BotProgram type is not specific to market-maker bots. Everything is structured with a mind towards future expansion with new automated trading routines. ui: New /mm view enables creation, re-configuration, and monitoring of existing market-maker bots.

buck54321 force-pushed the mmbot branch from cb9168e to d05f2ee Compare July 27, 2022 23:38

buck54321 force-pushed the mmbot branch from d05f2ee to 0ba1a8d Compare August 5, 2022 00:02

chappjc added this to the 0.6/1.0 milestone Aug 12, 2022

buck54321 force-pushed the mmbot branch 2 times, most recently from 59cdc17 to b72b938 Compare August 13, 2022 03:12

buck54321 force-pushed the mmbot branch from b72b938 to fbcf6ac Compare August 13, 2022 16:10

buck54321 marked this pull request as ready for review August 13, 2022 16:18

chappjc reviewed Aug 13, 2022

View reviewed changes

client/db/bolt/db.go Outdated Show resolved Hide resolved

chappjc reviewed Aug 13, 2022

View reviewed changes

client/cmd/mmbot/main.go Show resolved Hide resolved

chappjc reviewed Aug 13, 2022

View reviewed changes

client/cmd/mmbot/main.go Outdated Show resolved Hide resolved

chappjc reviewed Aug 13, 2022

View reviewed changes

client/core/core.go Show resolved Hide resolved

buck54321 commented Aug 13, 2022

View reviewed changes

buck54321 force-pushed the mmbot branch 4 times, most recently from 5cfb98e to 4bb7527 Compare August 15, 2022 02:21

buck54321 force-pushed the mmbot branch from 4bb7527 to d91890a Compare August 31, 2022 19:37

buck54321 force-pushed the mmbot branch from caf5ce0 to 7d8aa8b Compare September 9, 2022 16:09

chappjc requested changes Sep 14, 2022

View reviewed changes

JoeGruffins approved these changes Sep 16, 2022

View reviewed changes

buck54321 force-pushed the mmbot branch 2 times, most recently from 1dc4ad5 to 903dcca Compare September 22, 2022 14:56

buck54321 force-pushed the mmbot branch from 903dcca to 79b03ce Compare October 12, 2022 15:35

chappjc self-requested a review October 12, 2022 15:37

martonp reviewed Oct 16, 2022

View reviewed changes

buck54321 force-pushed the mmbot branch from 79b03ce to f3dc62e Compare October 19, 2022 00:30

chappjc requested a review from martonp October 24, 2022 14:34

buck54321 force-pushed the mmbot branch from f3dc62e to 60ecbf3 Compare October 26, 2022 00:04

martonp approved these changes Oct 30, 2022

View reviewed changes

chappjc approved these changes Nov 1, 2022

View reviewed changes

buck54321 added 8 commits November 2, 2022 05:55

early reviews

ecbaea6

check for changes before closing forms

f77a2cf

add drain protection

5118c2a

chappjc js fixes

ddb87e6

review followup. delete simharness

7d0e2de

martonp review

82065d1

add experimental flag

32e3cf1

buck54321 force-pushed the mmbot branch from 60ecbf3 to 32e3cf1 Compare November 2, 2022 11:06

martonp review touchups

ae2ecc0

chappjc merged commit 7d3e6c0 into decred:master Nov 2, 2022

chappjc added the bot label Mar 31, 2023

		@@ -1563,15 +1577,9 @@ func (btc baseWallet) PreSwap(req asset.PreSwapForm) (*asset.PreSwap, error) {
		}

multi: implement a market-maker bot #1738

multi: implement a market-maker bot #1738

Uh oh!

Conversation

buck54321 commented Jul 27, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

buck54321 commented Jul 27, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

buck54321 commented Aug 13, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chappjc commented Aug 13, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

martonp commented Aug 14, 2022

Uh oh!

buck54321 commented Aug 15, 2022

Uh oh!

martonp commented Aug 15, 2022

Uh oh!

chappjc left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chappjc Sep 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

JoeGruffins left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

buck54321 commented Sep 22, 2022

Uh oh!

buck54321 commented Sep 22, 2022

Uh oh!

chappjc commented Sep 22, 2022

Uh oh!

martonp left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

buck54321 commented Jul 27, 2022 •

edited

Loading

buck54321 commented Jul 27, 2022 •

edited

Loading

chappjc left a comment •

edited

Loading

chappjc Sep 14, 2022 •

edited

Loading

buck54321 Oct 18, 2022 •

edited

Loading

buck54321 Oct 18, 2022 •

edited

Loading