Xet is out !
Over the past few weeks, Hugging Face’s Xet Team took a major step forward by migrating the first Model and Dataset repositories off LFS and to Xet storage.
See more information on the HF blog: https://huggingface.co/blog/xet-on-the-hub
You can already enable Xet on Hugging Face account to benefit from faster downloads and uploads :)
We finalized an official integration with the huggingface_hub library that means you get the benefits of Xet without any significant changes to your current workflow.
Previous versions of datasets
For older versions of datasets you might see this warning in push_to_hub():
Uploading files as bytes or binary IO objects is not supported by Xet Storage.
This means the huggingface_hub + Xet integration isn't enabled for your version of datasets.
You can fix this by updating to datasets>=3.6.0 and huggingface_hub>=0.31.0
pip install -U datasets huggingface_hub
The future
Stay tuned for more Xet optimizations, especially on Xet-optimized Parquet
Xet is out !
Over the past few weeks, Hugging Face’s Xet Team took a major step forward by migrating the first Model and Dataset repositories off LFS and to Xet storage.
See more information on the HF blog: https://huggingface.co/blog/xet-on-the-hub
You can already enable Xet on Hugging Face account to benefit from faster downloads and uploads :)
We finalized an official integration with the
huggingface_hublibrary that means you get the benefits of Xet without any significant changes to your current workflow.Previous versions of
datasetsFor older versions of
datasetsyou might see this warning inpush_to_hub():This means the
huggingface_hub+ Xet integration isn't enabled for your version ofdatasets.You can fix this by updating to
datasets>=3.6.0andhuggingface_hub>=0.31.0The future
Stay tuned for more Xet optimizations, especially on Xet-optimized Parquet