Skip to content

Fix ParallelLoopExecution abandoning in-flight iterations and miscounting MinimumIterations - #807

Merged
imadityaa merged 7 commits into
microsoft:mainfrom
ankitsharma-99:fix/parallel-loop-execution-duration
Sep 29, 2026
Merged

imadityaa merged 7 commits into
microsoft:mainfrom
ankitsharma-99:fix/parallel-loop-execution-duration

Conversation

@ankitsharma-99

@ankitsharma-99 Ankit Sharma (ankitsharma-99) commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Two defects in ParallelLoopExecution.ExecuteComponentLoopAsync:

  1. In-flight iterations are abandoned when Duration elapses. On timeout the loop breaks out of
    Task.WhenAny without cancelling or awaiting the child. The child keeps running on the profile token and
    is only stopped as a side effect of profile shutdown when the loop happens to be the last action. Nothing
    awaits its cleanup, so:
    • its logs and upload requests are written after FileUploadMonitor's final drain and are not uploaded in
      the same run;
    • its processes can outlive VirtualClient;
    • if the loop is not the last action, the abandoned iteration runs concurrently with subsequent work.
  2. currentIteration is incremented twice per pass (at the top of try and again in finally), so the
    guards evaluate 1, 3, 5, ... . MinimumIterations: 1 therefore guarantees no completed iteration, and
    MinimumIterations: N guarantees only floor(N/2).

Example: with children that run for 30 minutes and Duration: 00:40:00, one iteration completes, a second
starts with 10 minutes remaining and is abandoned about 10 minutes in. With Duration equal to the child
duration and MinimumIterations: 1, the only iteration is abandoned at the deadline.

Fix

Restores the documented contract (website/docs/guides/0011-profiles.md): run until Duration, and let
MinimumIterations guarantee complete iterations.

  • A single deadline CancellationTokenSource (CancelAfter(Duration)) replaces the timeoutTask field and is
    shared by all parallel loops.
  • Iterations required to satisfy MinimumIterations run with a token linked only to the caller's token, so
    they complete even if Duration elapses.
  • Later iterations run with a token linked to both the caller's token and the deadline. When the deadline
    elapses the iteration is cancelled and awaited; no child task is left running when the component returns.
  • The counter tracks completed, non-cancelled iterations only, incremented once.
  • MinimumIterations default remains 1, docs corrected to match.
  • Per-iteration telemetry context is cloned instead of mutating the context shared by the parallel loops.
  • XML docs for Duration and MinimumIterations corrected.

Behavior

Children that run for 30 minutes:

Duration MinimumIterations Before After
40m 1 1 complete + 1 abandoned 1 complete + 1 cancelled and awaited at 40m
30m 1 only iteration abandoned at 30m 1 complete
40m 2 1 complete + 1 abandoned 2 complete
30m 0 abandoned at 30m cancelled and awaited at 30m
none any loops until cancelled unchanged

Unchanged: children still run in parallel, unsupported components are skipped, caller cancellation stops all
loops, and child exceptions propagate as before.

Testing

  • New tests:
    • ParallelLoopExecution_CancelsAndAwaitsTheInFlightIterationWhenTheDurationElapses
    • ParallelLoopExecution_CompletesTheMinimumIterationEvenWhenItExceedsTheDuration
    • ParallelLoopExecution_CancelsAndAwaitsAnIterationBeyondTheMinimumWhenTheDurationElapses
    • ParallelLoopExecution_LoopsUntilCancelledWhenNoDurationIsDefined
  • Existing MinimumIterations tests now assert exact completed-iteration counts instead of start counts.
  • VirtualClient.Core.UnitTests: 968 passed, 0 failed, 1 skipped (pre-existing flaky test).

Out of scope

A cancelled iteration still emits a Succeeded scenario metric, because VirtualClientComponent.ExecuteAsync
treats OperationCanceledException as success. That behavior is in the base component, not this loop.

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

@ankitsharma-99
Ankit Sharma (ankitsharma-99) marked this pull request as draft September 28, 2026 05:20
Replace the iteration-duration heuristic with a shared deadline token. Iterations needed to satisfy MinimumIterations run to completion; later iterations are cancelled and awaited when Duration elapses. Restore the documented MinimumIterations default of 0.
@ankitsharma-99 Ankit Sharma (ankitsharma-99) changed the title Fix ParallelLoopExecution duration handling Fix ParallelLoopExecution abandoning in-flight iterations and miscounting MinimumIterations Sep 28, 2026
@ankitsharma-99
Ankit Sharma (ankitsharma-99) marked this pull request as ready for review September 28, 2026 07:45
Signed-off-by: Ankit Sharma <58849812+ankitsharma-99@users.noreply.github.com>
Signed-off-by: Ankit Sharma <58849812+ankitsharma-99@users.noreply.github.com>
@imadityaa
imadityaa merged commit bd7fb21 into microsoft:main Sep 29, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants