Skip to content

[Bug]: Crusoe offers fail because quotas API returns counters as strings #4318

Description

@r4victor

Steps to reproduce

  1. Configure a crusoe backend.
  2. Run dstack offer or dstack apply for any configuration.

Actual behaviour

The Crusoe backend returns no offers, and the server logs a TypeError when comparing quota availability (see server logs).

Crusoe's GET /projects/{project_id}/quotas now returns max, used, and available as JSON strings, although the API docs still document them as integers:

{'programmatic_name': 'h100-80gb-sxm-ib', 'category': 'GPU Instance', 'max': '16', 'used': '0', 'available': '16', 'usage_tracking': 'QUOTA_USAGE_TRACKING_CENTRAL', ...}

_get_quota_map stores available as is, and get_all_offers_with_availability compares it with 0:

for offer in offers:
family = _get_instance_family(offer.instance.name)
availability = InstanceAvailability.UNKNOWN
for prog_name, available in quota_map.items():
if family.startswith(prog_name) or prog_name.startswith(family):
availability = (
InstanceAvailability.AVAILABLE
if available > 0
else InstanceAvailability.NO_QUOTA
)
break
result.append(offer.with_availability(availability=availability))
return result

def _get_quota_map(self) -> dict[str, int]:
try:
quotas = self._client.list_quotas()
except Exception:
logger.warning("Failed to fetch Crusoe quotas, availability will be UNKNOWN")
return {}
result = {}
for q in quotas:
prog_name = q.get("programmatic_name", "")
available = q.get("available", 0)
category = q.get("category", "")
if "Instance" in category:
result[prog_name] = available
return result

This code hasn't changed since the backend was added, so the change is on Crusoe's side. GET /capacities, used by gpuhunt, still returns quantity as an int, which is why the gpuhunt Crusoe catalog job keeps working.

Expected behaviour

Crusoe offers are returned with quota-based availability. If the quotas response can't be parsed, offers should fall back to UNKNOWN availability, as they already do when the quotas request fails, instead of dropping all Crusoe offers.

Possible fix: parse the counters with int(...) so both forms are accepted, and move the parsing inside the existing try.

dstack version

master (229f535)

Server logs

ERROR    dstack._internal.server.services.backends:504 Got exception when requesting offers from backend BackendType.CRUSOE
Traceback (most recent call last):
  ...
  File "src/dstack/_internal/core/backends/base/compute.py", line 285, in _get_all_offers_with_availability_cached
    return self.get_all_offers_with_availability(unallocated_resources)
  File "src/dstack/_internal/core/backends/crusoe/compute.py", line 164, in get_all_offers_with_availability
    if available > 0
TypeError: '>' not supported between instances of 'str' and 'int'

Additional information

No response

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions