ISSUE REPORT: Race Condition in Concurrent Tag Creation Causes Silent Tag Update Failure
Description
In tag.Controller.Ensure, Harbor checks if a tag exists. If it doesn't, it attempts to create it inside a transaction. However, the conflict error handling logic has a critical flaw that can cause silent tag update failures during concurrent pushes.
Root Cause Analysis
If two processes concurrently push different artifacts to the same new tag (e.g., latest), both see the tag doesn't exist and attempt to create it. Process A succeeds. Process B encounters a unique constraint violation on (repository_id, name).
tagID := int64(0)
if err = orm.WithTransaction(func(ctx context.Context) error {
tag := &Tag{}
// ...
tagID, err = c.Create(ctx, tag)
return err
})(orm.SetTransactionOpNameToContext(ctx, "tx-tag-ensure")); err != nil && !errors.IsConflictErr(err) {
return 0, err
}
return tagID, nil
Because errors.IsConflictErr(err) is true for Process B, it swallows the error and falls through to return tagID, nil. Because tagID was initialized to 0 and Create failed, Process B silently returns 0, nil.
Process B NEVER retries the fetch or updates the tag to point to its own artifact. The API responds with an HTTP 201 Created, but the tag latest silently remains attached to Process A's artifact.
Impact
- Silent Data Corruption: The registry database becomes inconsistent with the client's view of reality. A user's
docker push succeeds, but pulling that tag returns a completely different, concurrent image.
- Broken Proxy Caching: In
src/controller/proxy/controller.go, EnsureTag receives tagID=0 and immediately attempts to update the PullTime for id=0. This triggers a NotFoundError (tag 0 not found), causing proxy pulls of uncached tags to fail entirely under concurrency.
- CI/CD Impact: Real-world container pipelines frequently push standard tags (e.g.,
latest, main, nightly) concurrently. The fundamental "last push wins" expectation is silently violated, causing deployments to pull stale or incorrect code into production.
Proposed Fix
If a conflict error is detected, it means the tag was just created by a concurrent request. We should handle the conflict by re-executing the Ensure logic (via a recursive call) to fetch the existing tag and properly update its ArtifactID.
if err != nil {
if !errors.IsConflictErr(err) {
return 0, err
}
// The tag was created concurrently, retry Ensure to fetch and update it
return c.Ensure(ctx, repositoryID, artifactID, name)
}
return tagID, nil
Reproduction Steps
- Configure two CI jobs to build and push two different images to the exact same repository and tag (
my-repo:latest).
- Ensure the tag
latest does not exist in my-repo beforehand.
- Start the two
docker push operations concurrently so they hit Harbor's API simultaneously.
- Process A inserts the tag record for Image 1. Process B hits a DB conflict and silently drops the tag update.
- Both
docker push commands report success.
- Run
docker pull my-repo:latest. The pulled image will reliably be Image 1.
ISSUE REPORT: Race Condition in Concurrent Tag Creation Causes Silent Tag Update Failure
Description
In
tag.Controller.Ensure, Harbor checks if a tag exists. If it doesn't, it attempts to create it inside a transaction. However, the conflict error handling logic has a critical flaw that can cause silent tag update failures during concurrent pushes.Root Cause Analysis
If two processes concurrently push different artifacts to the same new tag (e.g.,
latest), both see the tag doesn't exist and attempt to create it. Process A succeeds. Process B encounters a unique constraint violation on(repository_id, name).Because
errors.IsConflictErr(err)is true for Process B, it swallows the error and falls through toreturn tagID, nil. BecausetagIDwas initialized to0andCreatefailed, Process B silently returns0, nil.Process B NEVER retries the fetch or updates the tag to point to its own artifact. The API responds with an HTTP 201 Created, but the tag
latestsilently remains attached to Process A's artifact.Impact
docker pushsucceeds, but pulling that tag returns a completely different, concurrent image.src/controller/proxy/controller.go,EnsureTagreceivestagID=0and immediately attempts to update thePullTimeforid=0. This triggers aNotFoundError(tag 0 not found), causing proxy pulls of uncached tags to fail entirely under concurrency.latest,main,nightly) concurrently. The fundamental "last push wins" expectation is silently violated, causing deployments to pull stale or incorrect code into production.Proposed Fix
If a conflict error is detected, it means the tag was just created by a concurrent request. We should handle the conflict by re-executing the
Ensurelogic (via a recursive call) to fetch the existing tag and properly update itsArtifactID.Reproduction Steps
my-repo:latest).latestdoes not exist inmy-repobeforehand.docker pushoperations concurrently so they hit Harbor's API simultaneously.docker pushcommands report success.docker pull my-repo:latest. The pulled image will reliably be Image 1.