Recently GitHub drastically tightened rate limits around these links,
causing our link checking process to fail.

We thought that we had addressed the problem in previous commits [1],
[2], but those solutions did not result in a practically-useful
configuration.

[1]: e8387c0
[2]: 8f5a818

Allows annotation of the individual config pieces

Modifying template1 has weird, unexpected effects on subsequent tests
in other modules. I sunk more time than I want to admit to figuring
out what was going on.

New benchmark just recipes for the vectorizer:

- `just pgai benchmark-mem`: runs the vectorizer on a queue of 500
  items, generates a memray profile, and a memory usage flamegraph.

- `just pgai benchmark-cpu`: runs the vectorizer on a queue of 500
  items, generates a CPU usage flamegraph using py-spy.

- `just pgai benchmark-cpu-top`: runs the vectorizer on a queue of 500
  items, displays a top like interface of CPU usage.

- `just pgai benchmark-queue-count`: shows the queue count of the
  running benchmark. Should be executed in a separate terminal. The
  count is updated in an interval.

The benchmark DB uses the wiki.dump stored in the repository, creates an
openAI vectorizer, and runs the benchmark tool storing the results in
`projects/pgai/benchmark/results`.

There's a new command for the cli `pgai vectorizer worker-benchmark`.
It's the same as the regular worker, but it's wrapped in a vcr client
that replays openAI requests. This command is used by the CPU benchmarks
in order to have constant request/response times to the API, and have
more deterministic time results. The cassette file is tracked using
git-lfs.

Memory benchmarks don't use the vcr wrapped command because they pollute
VCR will use considerable memory to handle the API calls.

* fix: openai per batch limit

* chore: trigger ci

* chore: leave a comment on why using estimated tokens

…rs (#683)" (#735)

This reverts commit 2a77389.

Release-As: 0.10.3