No work has been done to optimize general.el. Generally, this works out okay since most keybindings can be in a with-eval-after-load, and general will automatically delay them if the keymap does not exist. At some point, I should profile things (definitely will if I ever rewrite or split general). For now, minimizing the work done in general-define-key before the delay should help.

As @hlissner points out here, general can impact startup time significantly if using a lot prefix keywords. This could be done in general--define-key instead, so that it is deferred until the keymap(s) exist,. This would reduce the initial time general-define-key takes when it is possible to delay keybindings (and spreading out the total time when the user is autoloading packages).

@hlissner
If you remember, were most of these calls to general that were slowing things down making the keybinding immediately (making the prefix concatenation itself the issue) or were they defining keys for keymaps before the keymaps were available (making the fact that prefix concatenation is done before the delay the issue)?