Skip to content

Race Condition in Concurrent Tag Creation Causes Silent Tag Update Failure #23118

Description

@pulkitvats2007-crypto

ISSUE REPORT: Race Condition in Concurrent Tag Creation Causes Silent Tag Update Failure

Description

In tag.Controller.Ensure, Harbor checks if a tag exists. If it doesn't, it attempts to create it inside a transaction. However, the conflict error handling logic has a critical flaw that can cause silent tag update failures during concurrent pushes.

Root Cause Analysis

If two processes concurrently push different artifacts to the same new tag (e.g., latest), both see the tag doesn't exist and attempt to create it. Process A succeeds. Process B encounters a unique constraint violation on (repository_id, name).

tagID := int64(0)
if err = orm.WithTransaction(func(ctx context.Context) error {
	tag := &Tag{}
	// ...
	tagID, err = c.Create(ctx, tag)
	return err
})(orm.SetTransactionOpNameToContext(ctx, "tx-tag-ensure")); err != nil && !errors.IsConflictErr(err) {
	return 0, err
}

return tagID, nil

Because errors.IsConflictErr(err) is true for Process B, it swallows the error and falls through to return tagID, nil. Because tagID was initialized to 0 and Create failed, Process B silently returns 0, nil.

Process B NEVER retries the fetch or updates the tag to point to its own artifact. The API responds with an HTTP 201 Created, but the tag latest silently remains attached to Process A's artifact.

Impact

  • Silent Data Corruption: The registry database becomes inconsistent with the client's view of reality. A user's docker push succeeds, but pulling that tag returns a completely different, concurrent image.
  • Broken Proxy Caching: In src/controller/proxy/controller.go, EnsureTag receives tagID=0 and immediately attempts to update the PullTime for id=0. This triggers a NotFoundError (tag 0 not found), causing proxy pulls of uncached tags to fail entirely under concurrency.
  • CI/CD Impact: Real-world container pipelines frequently push standard tags (e.g., latest, main, nightly) concurrently. The fundamental "last push wins" expectation is silently violated, causing deployments to pull stale or incorrect code into production.

Proposed Fix

If a conflict error is detected, it means the tag was just created by a concurrent request. We should handle the conflict by re-executing the Ensure logic (via a recursive call) to fetch the existing tag and properly update its ArtifactID.

	if err != nil {
		if !errors.IsConflictErr(err) {
			return 0, err
		}
		// The tag was created concurrently, retry Ensure to fetch and update it
		return c.Ensure(ctx, repositoryID, artifactID, name)
	}
	return tagID, nil

Reproduction Steps

  1. Configure two CI jobs to build and push two different images to the exact same repository and tag (my-repo:latest).
  2. Ensure the tag latest does not exist in my-repo beforehand.
  3. Start the two docker push operations concurrently so they hit Harbor's API simultaneously.
  4. Process A inserts the tag record for Image 1. Process B hits a DB conflict and silently drops the tag update.
  5. Both docker push commands report success.
  6. Run docker pull my-repo:latest. The pulled image will reliably be Image 1.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions