-
-
Notifications
You must be signed in to change notification settings - Fork 6.4k
perf: avoid unnecessary count on getManyAndCount #11524
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
perf: avoid unnecessary count on getManyAndCount #11524
Conversation
Skip count query when it can be deduced from the number of returned rows. This will avoid one round trip and could be very helpful on pagination when the limit is not reached.
commit: |
WalkthroughThe changes introduce a lazy count optimization in the Changes
Sequence Diagram(s)sequenceDiagram
participant TestSuite as Test Suite
participant QueryBuilder as SelectQueryBuilder
participant DB as Database
participant Subscriber as AfterQuerySubscriber
TestSuite->>QueryBuilder: getManyAndCount({ take/limit, skip/offset })
QueryBuilder->>DB: SELECT entities
DB-->>QueryBuilder: entities[]
alt conditions allow lazy count
QueryBuilder-->>TestSuite: [entities[], inferredCount]
else
QueryBuilder->>DB: SELECT COUNT(*)
DB-->>QueryBuilder: count
QueryBuilder-->>TestSuite: [entities[], count]
end
QueryBuilder-->>Subscriber: afterQuery event(s)
Poem
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 ESLint
npm error Exit handler never called! 📜 Recent review detailsConfiguration used: CodeRabbit UI 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
✨ Finishing Touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 0
🧹 Nitpick comments (6)
test/other-issues/lazy-count/subscribers/AfterQuerySubscriber.ts (1)
8-29
: Strengthen typing & encapsulation ofcalledQueries
.
calledQueries
stores raw SQL strings, yet it’s typed asany[]
and left public.
Tighten the contract and avoid accidental external mutation:- calledQueries: any[] = [] + private calledQueries: string[] = []No downstream code relies on direct field access (tests use the helper methods), so this should be non-breaking while improving safety.
src/query-builder/SelectQueryBuilder.ts (1)
1892-1899
: Minor issues in cache-id handling & typo
- Typo:
retreive
→retrieve
.this.expressionMap.cacheId = cacheId ? \
${cacheId}-count` : cacheId‑ When
cacheIdis
undefined` the assignment is a no-op; consider skipping it entirely to avoid the extra write.- The original
cacheId
is not restored after the count query; if the same builder instance is reused later it will carry the suffixed id.- // Creates a new cacheId for the count query, or it will retreive the above query results + // Creates a new cacheId for the count query, or it will retrieve the previous query results ... - this.expressionMap.cacheId = cacheId ? `${cacheId}-count` : cacheId + if (cacheId) { + this.expressionMap.cacheId = `${cacheId}-count` + }Restore
cacheId
afterexecuteCountQuery
if the builder is intended to be reusable.test/other-issues/lazy-count/lazy-count.ts (4)
29-34
: Factor out test-data seeding to reduce repetitionThe four
for
-loops that seed 2 – 5Post
rows are identical boilerplate. Extracting a small helper (e.g.await seedPosts(connection, 5)
) keeps each test focused on the behaviour under scrutiny and makes future changes to the seed logic (different entity, additional fields, etc.) one-liner edits.Also applies to: 58-63, 87-92, 116-121
36-38
: Avoid brittle index-based access to the subscriber
connection.subscribers[0]
silently breaks if another subscriber is registered first.
Prefer a type/instance lookup:-const afterQuery = connection.subscribers[0] as AfterQuerySubscriber +const afterQuery = connection.subscribers.find( + (s): s is AfterQuerySubscriber => s instanceof AfterQuerySubscriber, +)!This is safer and self-documenting.
50-52
: Regex assertion may miss driver-specificCOUNT
syntaxThe check
/.?count/i
assumes the raw SQL contains the literal wordCOUNT
. Some drivers alias the aggregate (e.g."COUNT"("*)"
→"cnt"
) or wrap it in comments, causing false negatives/positives.
A stricter approach is to expose a boolean fromAfterQuerySubscriber
likewasCountExecuted
instead of string-matching the SQL text.Also applies to: 79-81, 108-110, 138-140
46-49
: Inconsistent Chai styles (should
vsexpect
)Each assertion mixes the
should
interface (count.should.be.equal(...)
) with theexpect
interface. While both work, sticking to one style per file avoids cognitive overhead and linter noise.Also applies to: 75-78, 104-107, 134-137
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (4)
src/query-builder/SelectQueryBuilder.ts
(1 hunks)test/other-issues/lazy-count/entity/Post.ts
(1 hunks)test/other-issues/lazy-count/lazy-count.ts
(1 hunks)test/other-issues/lazy-count/subscribers/AfterQuerySubscriber.ts
(1 hunks)
🧰 Additional context used
🧬 Code Graph Analysis (2)
test/other-issues/lazy-count/subscribers/AfterQuerySubscriber.ts (1)
src/subscriber/event/QueryEvent.ts (1)
AfterQueryEvent
(39-59)
src/query-builder/SelectQueryBuilder.ts (1)
src/entity-manager/EntityManager.ts (1)
count
(1005-1017)
🔇 Additional comments (3)
test/other-issues/lazy-count/entity/Post.ts (1)
3-10
: Looks good – minimal, portable entity definition.The entity is concise and follows TypeORM conventions; no blocking issues spotted.
src/query-builder/SelectQueryBuilder.ts (1)
1880-1891
: Edge-case: deduplication via joins can yieldentities.length < take
while more rows still existThe shortcut assumes that if
entities.length < take|limit
(and noskip/offset
) then the full result-set was fetched.
With joins TypeORM usesDISTINCT
sub-queries to get entity IDs, but the finalentities.length
is calculated after hydration & duplicate elimination.Example:
take = 10 DISTINCT sub-query picks 10 ids JOIN duplicates inflate the result-set → 25 raw rows Hydrator collapses them to, say, 8 unique entities
entities.length
becomes 8 < take, the count query is skipped and8
is returned, while at least 10 entities actually exist.Please double-check
entitiesAndRaw
always contains exactlytake|limit
unique entities when more are available (incl. eager relations / complex joins).
If not, fallback to the explicit count query whenever joins are present or after comparing against the size of the DISTINCT id list instead of the hydrated entity list.test/other-issues/lazy-count/lazy-count.ts (1)
40-45
: Cross-dialect caution:.limit()
is MySQL-only
QueryBuilder.limit()
is translated only by MySQL drivers; for Postgres/SQLite/etc. the call is ignored. If the test-suite ever runs on those dialects, this test will misleadingly behave like the.take()
tests. Consider sticking to.take()
for portability or gating the test to MySQL connections.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 0
🧹 Nitpick comments (1)
test/other-issues/lazy-count/entity/Post.ts (1)
17-21
: Initialize or assert thecomments
collection
comments
is declared without a definite-assignment assertion (!
) or an initial value.
WithstrictPropertyInitialization
enabled, this triggers TS2740; at runtime it may also beundefined
, forcing callers to null-check. Two quick fixes:- comments: Comment[] + comments!: Comment[] // 💡 simplest – assert it will be set + + // Or give it a safe default: + // comments: Comment[] = []Either option keeps the intent clear and avoids unnecessary undefined checks.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (5)
src/query-builder/SelectQueryBuilder.ts
(2 hunks)test/other-issues/lazy-count/entity/Comment.ts
(1 hunks)test/other-issues/lazy-count/entity/Post.ts
(1 hunks)test/other-issues/lazy-count/lazy-count.ts
(1 hunks)test/other-issues/lazy-count/subscribers/AfterQuerySubscriber.ts
(1 hunks)
✅ Files skipped from review due to trivial changes (1)
- test/other-issues/lazy-count/entity/Comment.ts
🚧 Files skipped from review as they are similar to previous changes (3)
- test/other-issues/lazy-count/subscribers/AfterQuerySubscriber.ts
- src/query-builder/SelectQueryBuilder.ts
- test/other-issues/lazy-count/lazy-count.ts
🧰 Additional context used
🧬 Code Graph Analysis (1)
test/other-issues/lazy-count/entity/Post.ts (4)
test/other-issues/lazy-count/entity/Comment.ts (1)
Entity
(9-23)src/decorator/columns/PrimaryGeneratedColumn.ts (1)
PrimaryGeneratedColumn
(55-119)src/decorator/columns/Column.ts (1)
Column
(134-220)src/decorator/relations/OneToMany.ts (1)
OneToMany
(11-47)
🔇 Additional comments (1)
test/other-issues/lazy-count/entity/Post.ts (1)
9-16
: Entity skeleton looks goodThe basic entity wiring (primary key + simple column) is correct and consistent with the conventions used elsewhere in the test-suite.
Thanks for your amazing help @EQuincerot |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice fix, thank you @EQuincerot
Just a few code improvements that could be made.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One more tweak, otherwise looks good 👍
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for your contribution @EQuincerot 💜
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great improvement @EQuincerot thanks
Description of change
When
getManyAndCount
is called, stop execute thecount
query if it can be deduced from thenumber of returned rows. This will increase performance by avoiding one round trip to the database. It would be especially useful when using pagination libraries such as https://github.com/ppetzold/nestjs-paginate and even more when the max results per page is rarely reached.
Example
Before this change, it triggers two queries:
But if the first query returns only 5 results, we know that the count will also be 5.
After this change, if the first query returns less rows than the limit, we don't trigger the count:
ℹ️ Note that this optimization also works when some filtering is done, as the filtering will be the same for the
getMany
as for thecount
.Pull-Request Checklist
master
branchFixes #00000
Summary by CodeRabbit
Summary by CodeRabbit