Skip to content

Allow graceful shutdown of the container #24

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Aug 7, 2022
Merged

Conversation

arikmaor
Copy link
Collaborator

@arikmaor arikmaor commented Aug 4, 2022

This should be reverted once k3d-io/k3d#1119 is merged

Still needs testing

This commit should be reverted once
k3d-io/k3d#1119 is merged
@arikmaor arikmaor force-pushed the graceful-shutdown branch from d455bb2 to 2557a4f Compare August 4, 2022 14:35
Copy link
Member

@yotamazriel yotamazriel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yotamazriel
Copy link
Member

Got the following logs from ubuntu:

tensorleap@tensorleap-OMEN-Laptop:~$ k3d cluster delete tensorleap
INFO[0000] No clusters found                            
tensorleap@tensorleap-OMEN-Laptop:~$ bash <(curl -s https://raw.githubusercontent.com/tensorleap/helm-charts/graceful-shutdown/install.sh)
Checking docker installation
Checking k3d installation
k3d version v5.4.4
k3s version v1.23.8-k3s1 (default)
Getting latest version...
1.0.50
Checking docker storage and memory limits...
Enter a path to be mounted and accessible by scripts (default: /home/tensorleap/tensorleap/data):

[sudo] password for tensorleap: 
Creating tensorleap k3d cluster...
WARN[0000] No node filter specified                     
WARN[0000] No node filter specified                     
WARN[0000] No node filter specified                     
WARN[0000] No node filter specified                     
INFO[0000] portmapping '4589:80' targets the loadbalancer: defaulting to [servers::proxy agents::proxy] 
INFO[0000] Prep: Network                                
INFO[0000] Created network 'k3d-tensorleap'             
INFO[0000] Created image volume k3d-tensorleap-images   
INFO[0000] Starting new tools node...                   
INFO[0000] Starting Node 'k3d-tensorleap-tools'         
INFO[0001] Creating node 'k3d-tensorleap-server-0'      
INFO[0001] Creating LoadBalancer 'k3d-tensorleap-serverlb' 
INFO[0001] Using the k3d-tools node to gather environment information 
INFO[0001] HostIP: using network gateway 172.25.0.1 address 
INFO[0001] Starting cluster 'tensorleap'                
INFO[0001] Starting servers...                          
ERRO[0001] Node k3d-tensorleap-server-0: Failed executing preStartAction 'WriteFileAction': [WriteFileAction] Writing 451 bytes to /bin/k3d-entrypoint.sh (mode -rwxr--r--): Write custom k3d entrypoint script (that powers the magic fixes): Failed to copy content to container '6ce9cb380c4e3d389631b770a411cb6c7e7b9d8981e986bcada6b0ddba9ecffd': Error response from daemon: Error processing tar file(exit status 1): unlinkat /bin/k3d-entrypoint.sh: device or resource busy 
INFO[0001] Starting Node 'k3d-tensorleap-server-0'      
INFO[0005] All agents already running.                  
INFO[0005] Starting helpers...                          
INFO[0005] Starting Node 'k3d-tensorleap-serverlb'      
INFO[0012] Injecting records for hostAliases (incl. host.k3d.internal) and for 2 network members into CoreDNS configmap... 
INFO[0014] Cluster 'tensorleap' created successfully!   
INFO[0014] You can now use it like this:                
kubectl cluster-info
Waiting for images to download and install... (This can take up to 15 minutes depends on network speed)


error: timed out waiting for the condition on jobs/helm-install-tensorleap
Timeout! Images may still be downloading, wait a few minutes and see if Tensorleap is available on http://127.0.0.1:4589 If it's not, contact support

@arikmaor
Copy link
Collaborator Author

arikmaor commented Aug 6, 2022

And what happens in the cluster?

@yotamazriel yotamazriel merged commit ce32d2a into master Aug 7, 2022
@yotamazriel yotamazriel deleted the graceful-shutdown branch August 7, 2022 09:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants