Steps to reproduce
- Configure a
crusoe backend.
- Run
dstack offer or dstack apply for any configuration.
Actual behaviour
The Crusoe backend returns no offers, and the server logs a TypeError when comparing quota availability (see server logs).
Crusoe's GET /projects/{project_id}/quotas now returns max, used, and available as JSON strings, although the API docs still document them as integers:
{'programmatic_name': 'h100-80gb-sxm-ib', 'category': 'GPU Instance', 'max': '16', 'used': '0', 'available': '16', 'usage_tracking': 'QUOTA_USAGE_TRACKING_CENTRAL', ...}
_get_quota_map stores available as is, and get_all_offers_with_availability compares it with 0:
|
for offer in offers: |
|
family = _get_instance_family(offer.instance.name) |
|
availability = InstanceAvailability.UNKNOWN |
|
for prog_name, available in quota_map.items(): |
|
if family.startswith(prog_name) or prog_name.startswith(family): |
|
availability = ( |
|
InstanceAvailability.AVAILABLE |
|
if available > 0 |
|
else InstanceAvailability.NO_QUOTA |
|
) |
|
break |
|
result.append(offer.with_availability(availability=availability)) |
|
return result |
|
def _get_quota_map(self) -> dict[str, int]: |
|
try: |
|
quotas = self._client.list_quotas() |
|
except Exception: |
|
logger.warning("Failed to fetch Crusoe quotas, availability will be UNKNOWN") |
|
return {} |
|
result = {} |
|
for q in quotas: |
|
prog_name = q.get("programmatic_name", "") |
|
available = q.get("available", 0) |
|
category = q.get("category", "") |
|
if "Instance" in category: |
|
result[prog_name] = available |
|
return result |
This code hasn't changed since the backend was added, so the change is on Crusoe's side. GET /capacities, used by gpuhunt, still returns quantity as an int, which is why the gpuhunt Crusoe catalog job keeps working.
Expected behaviour
Crusoe offers are returned with quota-based availability. If the quotas response can't be parsed, offers should fall back to UNKNOWN availability, as they already do when the quotas request fails, instead of dropping all Crusoe offers.
Possible fix: parse the counters with int(...) so both forms are accepted, and move the parsing inside the existing try.
dstack version
master (229f535)
Server logs
ERROR dstack._internal.server.services.backends:504 Got exception when requesting offers from backend BackendType.CRUSOE
Traceback (most recent call last):
...
File "src/dstack/_internal/core/backends/base/compute.py", line 285, in _get_all_offers_with_availability_cached
return self.get_all_offers_with_availability(unallocated_resources)
File "src/dstack/_internal/core/backends/crusoe/compute.py", line 164, in get_all_offers_with_availability
if available > 0
TypeError: '>' not supported between instances of 'str' and 'int'
Additional information
No response
Steps to reproduce
crusoebackend.dstack offerordstack applyfor any configuration.Actual behaviour
The Crusoe backend returns no offers, and the server logs a
TypeErrorwhen comparing quota availability (see server logs).Crusoe's
GET /projects/{project_id}/quotasnow returnsmax,used, andavailableas JSON strings, although the API docs still document them as integers:_get_quota_mapstoresavailableas is, andget_all_offers_with_availabilitycompares it with0:dstack/src/dstack/_internal/core/backends/crusoe/compute.py
Lines 156 to 168 in 229f535
dstack/src/dstack/_internal/core/backends/crusoe/compute.py
Lines 170 to 183 in 229f535
This code hasn't changed since the backend was added, so the change is on Crusoe's side.
GET /capacities, used by gpuhunt, still returnsquantityas an int, which is why the gpuhunt Crusoe catalog job keeps working.Expected behaviour
Crusoe offers are returned with quota-based availability. If the quotas response can't be parsed, offers should fall back to
UNKNOWNavailability, as they already do when the quotas request fails, instead of dropping all Crusoe offers.Possible fix: parse the counters with
int(...)so both forms are accepted, and move the parsing inside the existingtry.dstack version
master (229f535)
Server logs
Additional information
No response