🐛 Bug Report: Out of Memory Error when Collating TechDocs for Search #23047
Description
📜 Description
Currently when we enable TechDocs search in the Backstage Demo site we run into an Out of Memory error which crashes the container. This happens shortly after start up and causes a crashes loop.
Some important details:
- This seems to only happen with the new backend system (have not had time to test it on the legacy backend to confirm)
- We deploy to Heroku and only have 1gb of RAM and have this set in our Dockerfile:
ENV NODE_OPTIONS "--max-old-space-size=1000"(locally this seems to break between 1250 and 1500) - We are hosting the generated TechDocs on a GCS bucket
- The file it's attempting to read is roughly 20mb: search_index.json
- As this is a Demo we are using SQLite as our DB which is in-memory and Lunr for search indexing which is also in-memory
👍 Expected behavior
We should not hit an out of memory error when collating TechDocs for Search.
👎 Actual Behavior with Screenshots
First we see "Collating documents for techdocs via DefaultTechDocsCollatorFactory" in the logs, then it tries to get the search_index.json for one of the entities, in this case for: https://demo.backstage.io/catalog/default/component/backstage. There is a short delay and then we get this error and the app crashes:
<--- Last few GCs --->
[1:0x17938870] 16712 ms: Mark-sweep (reduce) 1240.7 (1271.2) -> 1239.8 (1271.5) MB, 432.3 / 0.0 ms (average mu = 0.145, current mu = 0.010) allocation failure; scavenge might not succeed
[1:0x17938870] 17326 ms: Mark-sweep (reduce) 1240.9 (1271.5) -> 1240.2 (1272.0) MB, 609.2 / 0.0 ms (average mu = 0.067, current mu = 0.007) allocation failure; scavenge might not succeed
<--- JS stacktrace --->
FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
1: 0xb8097c node::Abort() [node]
2: 0xa98f08 [node]
3: 0xd420c0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
4: 0xd42290 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
5: 0xf20844 [node]
6: 0xf327dc v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
7: 0xf0e960 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
8: 0xf0f938 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
9: 0xef1aec v8::internal::Factory::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [node]
10: 0xee879c v8::internal::FactoryBase<v8::internal::Factory>::AllocateRawArray(int, v8::internal::AllocationType) [node]
11: 0xee8958 v8::internal::FactoryBase<v8::internal::Factory>::NewFixedArrayWithFiller(v8::internal::Handle<v8::internal::Map>, int, v8::internal::Handle<v8::internal::Oddball>, v8::internal::AllocationType) [node]
12: 0x10c6e58 [node]
13: 0x11416d0 [node]
14: 0x1142c14 v8::internal::KeyAccumulator::GetKeys(v8::internal::Handle<v8::internal::JSReceiver>, v8::internal::KeyCollectionMode, v8::internal::PropertyFilter, v8::internal::GetKeysConversion, bool, bool) [node]
15: 0x12a06cc v8::internal::Runtime_ObjectKeys(int, unsigned long*, v8::internal::Isolate*) [node]
16: 0x167d76c [node]
👟 Reproduction steps
This is pretty tricky, I have a branch that can be used at a starting point here: https://github.com/awanlin/backstage/tree/debug/techdocs-search
To us this branch to test with do the following:
- Configure proper access to a GCC bucket and generate the TechDocs, this is the workflow that does this for the demo site: https://github.com/backstage/demo/blob/master/.github/workflows/techdocs.yml
- Run
yarn install,yarn tscandyarn build:backend --config ../../app-config.yaml --config ../../app-config.local.yaml - Run
docker image build . -t backstage --progress=plain - Then run
docker run -it -p 7007:7007 backstage
Notice: after a minute or so it will crash with the out of memory previously shared
📃 Provide the context for the Bug.
As this causes a crash loop we aren't able to enable TechDocs search in the Demo site. This would be nice to showcase there for those looking at Adopting Backstage!
🖥️ Your Environment
I think the critical environment specific details are: the size of the index file, using GCS bucket, the new backend system, and the limited amount of RAM.
OS: Darwin 23.3.0 - darwin/arm64
node: v18.18.2
yarn: 3.5.0
cli: 0.25.2-next.3 (installed)
backstage: 1.23.0-next.3
Dependencies:
@backstage/app-defaults 1.5.0-next.3
@backstage/backend-app-api 0.5.10, 0.5.11-next.3
@backstage/backend-common 0.19.9, 0.20.1, 0.21.0-next.3
@backstage/backend-defaults 0.2.10-next.3
@backstage/backend-dev-utils 0.1.3, 0.1.4-next.0
@backstage/backend-openapi-utils 0.1.3-next.3
@backstage/backend-plugin-api 0.6.10-next.3, 0.6.9
@backstage/backend-tasks 0.5.14, 0.5.15-next.3
@backstage/catalog-client 1.5.2, 1.6.0-next.1
@backstage/catalog-model 1.4.3, 1.4.4-next.0
@backstage/cli-common 0.1.13
@backstage/cli-node 0.2.2, 0.2.3-next.0
@backstage/cli 0.25.2-next.3
@backstage/config-loader 1.6.1, 1.6.2-next.0
@backstage/config 1.1.1
@backstage/core-app-api 1.12.0-next.1
@backstage/core-compat-api 0.2.0-next.3
@backstage/core-components 0.14.0-next.2
@backstage/core-plugin-api 1.9.0-next.1
@backstage/e2e-test-utils 0.1.1-next.0
@backstage/errors 1.2.3
@backstage/eslint-plugin 0.1.5-next.0
@backstage/frontend-plugin-api 0.6.0-next.3
@backstage/integration-aws-node 0.1.8, 0.1.9-next.0
@backstage/integration-react 1.1.24-next.2
@backstage/integration 1.8.0, 1.9.0-next.1
@backstage/plugin-api-docs 0.11.0-next.3
@backstage/plugin-app-backend 0.3.58-next.3
@backstage/plugin-app-node 0.1.10-next.3
@backstage/plugin-auth-backend-module-atlassian-provider 0.1.2-next.3
@backstage/plugin-auth-backend-module-aws-alb-provider 0.1.0-next.2
@backstage/plugin-auth-backend-module-gcp-iap-provider 0.2.4-next.3
@backstage/plugin-auth-backend-module-github-provider 0.1.7-next.3
@backstage/plugin-auth-backend-module-gitlab-provider 0.1.7-next.3
@backstage/plugin-auth-backend-module-google-provider 0.1.7-next.3
@backstage/plugin-auth-backend-module-microsoft-provider 0.1.5-next.3
@backstage/plugin-auth-backend-module-oauth2-provider 0.1.7-next.3
@backstage/plugin-auth-backend-module-oauth2-proxy-provider 0.1.2-next.3
@backstage/plugin-auth-backend-module-oidc-provider 0.1.0-next.3
@backstage/plugin-auth-backend-module-okta-provider 0.0.3-next.3
@backstage/plugin-auth-backend 0.21.0-next.3
@backstage/plugin-auth-node 0.4.3, 0.4.4-next.3
@backstage/plugin-badges-backend 0.3.7-next.3
@backstage/plugin-badges 0.2.54-next.3
@backstage/plugin-catalog-backend-module-scaffolder-entity-model 0.1.7-next.3
@backstage/plugin-catalog-backend 1.17.0-next.3
@backstage/plugin-catalog-common 1.0.20, 1.0.21-next.0
@backstage/plugin-catalog-graph 0.4.0-next.3
@backstage/plugin-catalog-node 1.6.1, 1.6.2-next.3
@backstage/plugin-catalog-react 1.10.0-next.3
@backstage/plugin-catalog 1.17.0-next.3
@backstage/plugin-cost-insights-common 0.1.2
@backstage/plugin-cost-insights 0.12.19-next.3
@backstage/plugin-events-node 0.2.19-next.3
@backstage/plugin-explore-backend 0.0.20-next.3
@backstage/plugin-explore-common 0.0.2
@backstage/plugin-explore-react 0.0.36-next.1
@backstage/plugin-explore 0.4.16-next.3
@backstage/plugin-github-actions 0.6.11-next.3
@backstage/plugin-graphiql 0.3.3-next.3
@backstage/plugin-home-react 0.1.8-next.3
@backstage/plugin-home 0.6.2-next.3
@backstage/plugin-kubernetes-backend 0.15.0-next.3
@backstage/plugin-kubernetes-common 0.7.4-next.2
@backstage/plugin-kubernetes-node 0.1.4-next.3
@backstage/plugin-kubernetes-react 0.3.0-next.3
@backstage/plugin-kubernetes 0.11.5-next.3
@backstage/plugin-org 0.6.20-next.3
@backstage/plugin-permission-common 0.7.12
@backstage/plugin-permission-node 0.7.20, 0.7.21-next.3
@backstage/plugin-permission-react 0.4.20-next.1
@backstage/plugin-proxy-backend 0.4.8-next.3
@backstage/plugin-scaffolder-backend-module-azure 0.1.2-next.3
@backstage/plugin-scaffolder-backend-module-bitbucket-cloud 0.1.0-next.1
@backstage/plugin-scaffolder-backend-module-bitbucket-server 0.1.0-next.1
@backstage/plugin-scaffolder-backend-module-bitbucket 0.2.0-next.3
@backstage/plugin-scaffolder-backend-module-gerrit 0.1.2-next.3
@backstage/plugin-scaffolder-backend-module-github 0.2.0-next.3
@backstage/plugin-scaffolder-backend-module-gitlab 0.2.13-next.3
@backstage/plugin-scaffolder-backend 1.21.0-next.3
@backstage/plugin-scaffolder-common 1.5.0-next.1
@backstage/plugin-scaffolder-node 0.3.0-next.3
@backstage/plugin-scaffolder-react 1.8.0-next.3
@backstage/plugin-scaffolder 1.18.0-next.3
@backstage/plugin-search-backend-module-catalog 0.1.14-next.3
@backstage/plugin-search-backend-module-explore 0.1.14-next.3
@backstage/plugin-search-backend-module-techdocs 0.1.14-next.3
@backstage/plugin-search-backend-node 1.2.14-next.3
@backstage/plugin-search-backend 1.5.0-next.3
@backstage/plugin-search-common 1.2.10
@backstage/plugin-search-react 1.7.6-next.3
@backstage/plugin-search 1.4.6-next.3
@backstage/plugin-tech-radar 0.6.13-next.3
@backstage/plugin-techdocs-backend 1.9.3-next.3
@backstage/plugin-techdocs-node 1.11.2-next.3
@backstage/plugin-techdocs-react 1.1.16-next.2
@backstage/plugin-techdocs 1.10.0-next.3
@backstage/plugin-todo-backend 0.3.8-next.3
@backstage/plugin-todo 0.2.34-next.3
@backstage/plugin-user-settings 0.8.1-next.3
@backstage/release-manifests 0.0.11
@backstage/repo-tools 0.6.0-next.3
@backstage/theme 0.5.1-next.1
@backstage/types 1.1.1
@backstage/version-bridge 1.0.7
👀 Have you spent some time to check if this bug has been raised before?
- I checked and didn't find similar issue
🏢 Have you read the Code of Conduct?
- I have read the Code of Conduct
Are you willing to submit PR?
No, but I'm happy to collaborate on a PR with someone else