Skip to content

docs(ops): make site monitoring aware of log rotation - #3753

Merged
LauraGPT merged 1 commit into
mainfrom
codex/rotation-aware-site-monitoring-20261002
Oct 2, 2026
Merged

LauraGPT merged 1 commit into
mainfrom
codex/rotation-aware-site-monitoring-20261002

Conversation

@LauraGPT

@LauraGPT LauraGPT commented Oct 2, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Correct the product-site monitoring procedure so an empty current log file cannot be mistaken for a healthy service after log rotation.

The completed 24-hour follow-up found live Nginx descriptors still writing .log.1 while current .log paths were empty. Read-only diagnosis also found an empty configured PID file and a failed systemd unit while the manual master continued serving. The packaged rotate selector, run with --test only, returned no matching process; its init-script wrapper is written to return success unconditionally. This establishes the current control-path mismatch, not the event that originally emptied the PID file.

  • Replace blind tail and user-agent-filtered route counting with an explicit, numeric master-PID guard and read-only descriptor/inode inspection.
  • Require actual open logs plus rotated/compressed history, defined equal-duration windows, and aggregate-only reporting. Distinguish validation traffic from organic attribution.
  • Explain why legacy upstream 502/503/504 converted to 302 means zero visible 5xx is not proof of backend health.
  • Record both scheduled follow-ups as complete; scope rollback to cases where the static release is implicated.
  • Add three regressions in the existing release-script test module, including execution of the missing/invalid-PID guards and syntax checking of monitoring shell blocks.

Validation

  • Before documentation fix: 3 expected failures, 5 existing tests passed, zero collection errors/skips.
  • After fix: 8 passed, zero errors/failures/skips.
  • bash -n on monitoring blocks and git diff --check pass.
  • Independent source review found no P1/P2 issues. These tests do not identify a real master from an arbitrary numeric PID, simulate log rotation, or constitute a general shell security audit; the procedure still requires fresh operator verification.
  • Exact submitted head dce73ccb98a3a4c65cb08119ed745e488c5260a9: Product site run 36943928705 completed successfully. Downloaded job logs show 46 legacy-link tests, 524 build/validation tests, 315 additional selected tests, 2 browser-preparation tests, and 154 browser tests passing. Deselected cases are not counted as executed tests.

Integration

Merged as 14f12f1dc04418c3a2cd28e29623d1ad810c6d35 after fresh exact-head checks and the existing administrator merge exemption, without changing repository policy. Both merge parents and tree 23bd6e5b337ab00dbec69b607a4c49bc7833e8cb match the validated candidate.

Exact-merge Product site run 36951173195 and Update API Documentation run 36951173119 both completed successfully. Downloaded product-site job logs again show 46, 524, 315, 2, and 154 passing tests in the respective stages described above, with 3 and 6 explicitly deselected cases in the two selected-test stages. All returned head/merge check runs are terminal and successful. No full native site-suite execution is claimed beyond the eight focused native regressions.

Only the operations document and its existing test module change. No production PID-file/configuration write, service action, signal, restart, log reopen, deployment, rollback, model operation, or package release. The operational logging defect remains unresolved; this PR fixes the misleading acceptance procedure, not Nginx process ownership.

Signed-off-by: LauraGPT <18321252+LauraGPT@users.noreply.github.com>
@LauraGPT
LauraGPT merged commit 14f12f1 into main Oct 2, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant