Migrating Git LFS Repositories to Xet: A Technical Guide

Context and Background

The deployment of Hugging Face’s Xet Team storage backend in January has marked a pivotal shift in the management of large-scale data within the Generative AI Models & Applications sector. Initially, the transition saw approximately 6% of Hub downloads utilizing this new infrastructure. This early integration has since evolved to encompass 500,000 repositories, aggregating over 20 petabytes of data, signifying a crucial evolution as the Hub outpaces Git LFS. The migration to Xet is designed to accommodate the increasingly demanding storage needs of AI developers, facilitating a seamless transition for users.

As of now, Xet serves over one million users on the Hub, having become the default storage option for new users in May. This migration has proceeded with minimal disruption, supported by robust infrastructural elements, including the Git LFS Bridge and continuous background migrations, which have ensured that the transition is both efficient and user-friendly.

Main Goal and Methodology

The primary objective of this migration is to provide a scalable and efficient storage solution that enhances the user experience while supporting the growing demands of AI workloads. This goal is achieved through a design philosophy that emphasizes backward compatibility and operational continuity. Critical decisions made during the migration planning included the avoidance of a “hard cut-over” from Git LFS, allowing repositories to contain both Xet and LFS files without requiring immediate user intervention. This thoughtful approach has mitigated potential disruptions and facilitated a smooth transition for all users.

Advantages of Migrating to Xet

• Scalability: Xet’s architecture is capable of scaling to meet the demands of AI workloads, significantly improving data handling capacity compared to Git LFS.
• Seamless User Experience: The migration allows users to maintain their existing workflows without needing to adopt new protocols or tools immediately, thereby minimizing disruption.
• Efficient Background Migration: The use of an orchestrator ensures efficient file migrations from LFS to Xet without affecting ongoing operations, allowing for continuous usage during the transition.
• Performance Optimization: The introduction of chunk-based uploads and downloads optimizes transfer speeds and reduces the load on the system, enhancing overall performance.
• Community-Centric Design: Engaging with power users during the initial rollout provided valuable feedback that has been instrumental in refining the infrastructure and processes.

Limitations and Caveats

While the migration to Xet presents numerous advantages, it is essential to recognize certain limitations. The initial transition phase may introduce minor discrepancies for users operating on older versions of the huggingface_hub or huggingface.js, which do not support the new chunk-based transfer methodology. Additionally, while the system has demonstrated robust throughput capabilities, ongoing adjustments may be necessary to handle peak loads effectively, as evidenced by challenges encountered during large-scale migrations.

Future Implications for AI Development

The implications of transitioning to Xet extend beyond immediate operational improvements, as it lays the groundwork for future advancements in the management of AI-related datasets. By open-sourcing the Xet protocol and the underlying infrastructure, the Hugging Face team aims to foster a collaborative environment that encourages innovation in data storage and transfer methodologies. As AI models continue to grow in complexity and size, solutions like Xet will be critical in ensuring that developers are equipped to handle these changes efficiently. The move towards a unified storage system will not only streamline workflows but also enhance the ability to manage and leverage large datasets effectively, driving further advancements in the Generative AI domain.