Skip to content

Storage Leak in Fetcher #2821

@ChayanBansal

Description

@ChayanBansal

Fission/Kubernetes version

$ fission version
client:
  fission/core:
    BuildDate: "2023-05-18T12:33:09Z"
    GitCommit: f955d118
    Version: v1.19.0
server:
  fission/core:
    BuildDate: "2023-05-18T12:33:09Z"
    GitCommit: f955d118
    Version: v1.19.0

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.2", GitCommit:"7f6f68fdabc4df88cfea2dcf9a19b2b830f1e647", GitTreeState:"clean", BuildDate:"2023-05-17T14:13:27Z", GoVersion:"go1.20.4", Compiler:"gc", Platform:"darwin/arm64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"27+", GitVersion:"v1.27.3-eks-a5565ad", GitCommit:"78c8293d1c65e8a153bf3c03802ab9358c0e1a14", GitTreeState:"clean", BuildDate:"2023-06-16T17:32:40Z", GoVersion:"go1.20.5", Compiler:"gc", Platform:"linux/amd64"}

Kubernetes platform (e.g. Google Kubernetes Engine)
EKS (eks.3)
Describe the bug

I observed that env builder pod (that has builder and fetcher container) keeps getting evicted because of over utlization of ephemeral storage. On checking the container, I found that it still has files related to the packages it built previously which are piling up (since I was building more than 100 functions together simultaneously).

To Reproduce
Create an environment (version 3) and build a lot of packages.

Expected result

Ideally once the package is built, all its files should be removed from the builder pod

Actual result
Builder Pod is keeping all the files of previously generated packages

Screenshots/Dump file

$ fission support dump

Additional context

On digging the fetcher code, it was confirmed in the UploadHandler that even after a successful response from the ssClient.Upload func, the files are not deleted.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions