Sync clash with failed to diff ... git diff ... when using ManifestGeneratePaths and syncedRevision proceeds targetRevision #21014
Description
Checklist:
- I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
- I've included steps to reproduce the bug.
- I've pasted the output of
argocd version.
Describe the bug
- ArgoCD sync clashed by follwoing error.
- commit sha
481fbe3e687c8f339c0ecca000ee7ee7666a7aabproceeds85dd68870f5c09225ae05c2b81e9136af7bce397.
- commit sha
ComparisonError: Failed to load target state: failed to compare revisions for source 1 of 1: rpc error: code = Internal desc = unable to get changed files for repo https://github.com/xxx/xxx.git with revision 85dd68870f5c09225ae05c2b81e9136af7bce397: failed to diff 481fbe3e687c8f339c0ecca000ee7ee7666a7aab..85dd68870f5c09225ae05c2b81e9136af7bce397: git diff --name-only 481fbe3e687c8f339c0ecca000ee7ee7666a7aab..85dd68870f5c09225ae05c2b81e9136af7bce397` failed exit status 128: fatal: Invalid revision range 481fbe3e687c8f339c0ecca000ee7ee7666a7aab..85dd68870f5c09225ae05c2b81e9136af7bce397 (retried 3 times).`- We use manifestGeneratePaths with large monorepo and have multiple repo-server.
Root cause
- As is clear from the error message, the root cause is feat(controller): use manifest generate path during comparison (#14242) #15636
argo-cd/reposerver/repository/repository.go
Line 2778 in 27d1e64
- Indirectly, feat(reposerver): Skip calling git fetch if commit to checkout exists locally #18657 cause this error.
- The issue may occurs following situation:
- Someone commit s main branch with commit shaA.
- Someone commits main branch with commit shaB, where changed files are not included in manifestGeneratePaths.
- [Reconcilation Process]
UpdateRevisionForPathsupdate cache key with commit shaB and change syncRevision to shaB. - Sync is queued with targetRevision shaA.
- [SyncProcess]
UpdateRevisionForPathscomparesgit diff shaB..shaAby syncOperation.- The local repository may not have commit
sha B, so the error is thrown. - Note that
git fetch originis skipped in some cases by feat(reposerver): Skip calling git fetch if commit to checkout exists locally #18657
- The local repository may not have commit
*To Reproduce
Described in root cause. Note that we need more than two repo-servers to reproduce this issue.
argocd-repo-server-856db549f5-pgh49 repo-server time="2024-12-01T16:20:41Z" level=info msg="git cat-file -t db63a656360ed3a57c2bcd76f72ef38d793f5733" dir=/tmp/_argocd-repo/74473fb6-60e9-44dd-8fd0-25f9fb516214 execID=dde19
argocd-repo-server-856db549f5-pgh49 repo-server time="2024-12-01T16:20:41Z" level=info msg=Trace args="[git cat-file -t db63a656360ed3a57c2bcd76f72ef38d793f5733]" dir=/tmp/_argocd-repo/74473fb6-60e9-44dd-8fd0-25f9fb516214 operation_name="exec git" time_ms=1.7880369999999999
argocd-repo-server-856db549f5-pgh49 repo-server time="2024-12-01T16:20:41Z" level=info msg="git checkout --force db63a656360ed3a57c2bcd76f72ef38d793f5733" dir=/tmp/_argocd-repo/74473fb6-60e9-44dd-8fd0-25f9fb516214 execID=eae87
argocd-repo-server-856db549f5-pgh49 repo-server time="2024-12-01T16:20:41Z" level=info msg=Trace args="[git checkout --force db63a656360ed3a57c2bcd76f72ef38d793f5733]" dir=/tmp/_argocd-repo/74473fb6-60e9-44dd-8fd0-25f9fb516214 operation_name="exec git" time_ms=2.4994940000000003
argocd-repo-server-856db549f5-pgh49 repo-server time="2024-12-01T16:20:41Z" level=info msg="git clean -ffdx" dir=/tmp/_argocd-repo/74473fb6-60e9-44dd-8fd0-25f9fb516214 execID=5ae04
argocd-repo-server-856db549f5-pgh49 repo-server time="2024-12-01T16:20:41Z" level=info msg=Trace args="[git clean -ffdx]" dir=/tmp/_argocd-repo/74473fb6-60e9-44dd-8fd0-25f9fb516214 operation_name="exec git" time_ms=1.080456
argocd-repo-server-856db549f5-pgh49 repo-server time="2024-12-01T16:20:41Z" level=info msg="git diff --name-only c39784b87ce7df09f2c22651cbcb95536c938deb..db63a656360ed3a57c2bcd76f72ef38d793f5733" dir=/tmp/_argocd-repo/74473fb6-60e9-44dd-8fd0-25f9fb516214 execID=8e750
argocd-repo-server-856db549f5-pgh49 repo-server time="2024-12-01T16:20:41Z" level=error msg="`git diff --name-only c39784b87ce7df09f2c22651cbcb95536c938deb..db63a656360ed3a57c2bcd76f72ef38d793f5733` failed exit status 128: fatal: Invalid revision range c39784b87ce7df09f2c22651cbcb95536c938deb..db63a656360ed3a57c2bcd76f72ef38d793f5733" execID=8e750
argocd-repo-server-856db549f5-pgh49 repo-server time="2024-12-01T16:20:41Z" level=info msg=Trace args="[git diff --name-only c39784b87ce7df09f2c22651cbcb95536c938deb..db63a656360ed3a57c2bcd76f72ef38d793f5733]" dir=/tmp/_argocd-repo/74473fb6-60e9-44dd-8fd0-25f9fb516214 operation_name="exec git" time_ms=1.462371
argocd-repo-server-856db549f5-pgh49 repo-server time="2024-12-01T16:20:41Z" level=error msg="finished unary call with code Internal" error="rpc error: code = Internal desc = unable to get changed files for repo https://github.com/toyamagu-2021/argo-cd-sandbox.git with revision db63a656360ed3a57c2bcd76f72ef38d793f5733: failed to diff c39784b87ce7df09f2c22651cbcb95536c938deb..db63a656360ed3a57c2bcd76f72ef38d793f5733: `git diff --name-only c39784b87ce7df09f2c22651cbcb95536c938deb..db63a656360ed3a57c2bcd76f72ef38d793f5733` failed exit status 128: fatal: Invalid revision range c39784b87ce7df09f2c22651cbcb95536c938deb..db63a656360ed3a57c2bcd76f72ef38d793f5733" grpc.code=Internal grpc.method=UpdateRevisionForPaths grpc.service=repository.RepoServerService grpc.start_time="2024-12-01T16:20:41Z" grpc.time_ms=7.459 span.kind=server system=grpc
Expected behavior
Screenshots
Version
ArgoCD v2.13.1
Logs
Paste any relevant application logs here.