Skip to content

test(map): add benchmarks for v1 and q10 map parsing and rendering - #970

Open
allenporter wants to merge 2 commits into
Python-roborock:mainfrom
allenporter:perf-map-benchmarks
Open

allenporter wants to merge 2 commits into
Python-roborock:mainfrom
allenporter:perf-map-benchmarks

Conversation

@allenporter

@allenporter allenporter commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Description

Adds a comprehensive benchmark and profiling suite for Roborock V1 and Q10 map parsing in tests/map/test_benchmarks.py.

Capabilities

  • Pytest & CodSpeed Integration: Runs as fast unit assertions during standard pytest, and automatically collects high-precision wall-time/callgraph benchmarks when run with pytest --codspeed.
  • Standalone CLI Runner & Profiler: Can be run directly via python -m tests.map.test_benchmarks or with --profile to print latency percentiles (min, median, mean, p95), ops/sec, and cProfile hotspot breakdowns.
  • Pixel & Integrity Validation: Strictly verifies image dimensions, format (PNG), color mode (RGBA), and non-empty bounding boxes on every rendered benchmark output outside the timed loop to guarantee benchmarked changes preserve visual output correctness.

Workloads & Permutations Benchmarked

  1. Scale Variations:
    • V1 S6 at scale 1 (411x336 px) vs scale 2 (822x672 px) vs scale 4 (1644x1344 px).
    • Q10 200x200 grid at scale 1 (200x200 px) vs scale 4 (800x800 px).
  2. Drawables Permutations:
    • V1 S6 with all drawables disabled (drawables=[]) vs default drawables vs all available drawables (list(Drawable)).
  3. Payload Workloads:
    • Real Roborock V1 S5 map with 2 rooms (s5_fw2008_with_segments.bin).
    • Real Roborock V1 S6 map with 6 rooms and active segment (s6_fw2652_with_active_segment_and_no_mop_zone.bin).
    • Q10 wire packet decompression & metadata unpack (b01_q10_map.bin).
    • Q10 small map packet decoding and PNG rendering (b01_q10_map.bin).
    • Q10 full-scale 200x200 multi-room composite layer render.

Baseline Benchmark Results

Benchmark Target Image Size Min (ms) Median (ms) Mean (ms) P95 (ms) Throughput
V1 S5 Map (Scale 4, Default) 1024x964 53.7 ms 54.4 ms 54.9 ms 59.4 ms 18.2 ops/s
V1 S6 Map (Scale 4, Default) 1644x1344 111.5 ms 113.5 ms 114.1 ms 117.9 ms 8.8 ops/s
V1 S6 Map (Scale 2) 822x672 53.4 ms 54.9 ms 55.6 ms 61.4 ms 18.0 ops/s
V1 S6 Map (Scale 1) 411x336 40.8 ms 42.1 ms 42.1 ms 43.8 ms 23.7 ops/s
V1 S6 Map (No Drawables) 1644x1344 91.4 ms 94.3 ms 96.2 ms 116.0 ms 10.4 ops/s
V1 S6 Map (All Drawables) 1644x1344 115.6 ms 117.2 ms 117.8 ms 121.6 ms 8.5 ops/s
Q10 Wire Packet (Unpack only) — 0.008 ms 0.009 ms 0.009 ms 0.012 ms 109,917 ops/s
Q10 Small Map (Scale 1) 32x24 0.195 ms 0.197 ms 0.202 ms 0.263 ms 4,947 ops/s
Q10 200x200 Map (Scale 1) 200x200 21.8 ms 22.2 ms 22.3 ms 23.6 ms 44.7 ops/s
Q10 200x200 Map (Scale 4) 800x800 36.7 ms 37.5 ms 38.1 ms 42.8 ms 26.3 ops/s

Copilot AI lite review requested due to automatic review settings September 28, 2026 03:04

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The documented CodSpeed workflow lacks its dependency, and invalid iteration counts can crash the CLI.

Review effort: Lite
Findings: 2 Medium severity

Open (2)
What changed in this PR

Adds benchmark and profiling coverage for V1 and Q10 map parsing and rendering.

Changes:

  • Adds five pytest/CodSpeed benchmark workloads.
  • Adds synthetic full-scale Q10 rendering data.
  • Adds standalone timing and cProfile reporting.
File Summary Findings
tests/​map/​test_benchmarks.py Benchmark tests and standalone profiling runner Add the CodSpeed development dependency and validate positive --iterations values.

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread tests/map/test_benchmarks.py Outdated
Comment thread tests/map/test_benchmarks.py
@Lash-L

Lash-L commented Sep 28, 2026 •

Copy link
Copy Markdown
Collaborator

I like where this is heading :)

Trying to think of some edge cases that are worth encoding here:

  • Different scaling values?
  • One with all drawables on
  • one with no drawables on

Should we be comparing the pixels of the images as well?

@allenporter

Copy link
Copy Markdown
Contributor Author

Thanks for the feedback @Lash-L! I've updated the benchmark suite to incorporate all of those permutations:

  1. Different Scaling Values:

    • V1 S6:
      • Scale 1 (411x336 px): 42.1 ms (23.7 ops/s)
      • Scale 2 (822x672 px): 54.9 ms (18.0 ops/s)
      • Scale 4 (1644x1344 px, default): 113.5 ms (8.8 ops/s)
    • Q10 (200x200 grid):
      • Scale 1 (200x200 px): 22.2 ms (44.7 ops/s)
      • Scale 4 (800x800 px): 37.5 ms (26.3 ops/s)
  2. Drawables Permutations (V1 S6):

    • No drawables: 94.3 ms (10.4 ops/s)
    • Default drawables (charger, path, vacuum position): 113.5 ms (8.8 ops/s)
    • All drawables (including walls, no-go, mop zones, obstacles): 117.2 ms (8.5 ops/s)
  3. Pixel & Output Integrity Validation:

    • Added _validate_image() which runs outside the timed loop to assert PNG format, RGBA mode, expected pixel dimensions, and non-empty pixel bounding boxes (image.getbbox() is not None) for every benchmark target to guarantee that performance optimizations preserve output fidelity.

@allenporter
allenporter marked this pull request as ready for review October 4, 2026 22:49
@allenporter
allenporter requested a review from Lash-L October 4, 2026 22:49

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants