EECS Department, University of California, Berkeley

Technical Report No. UCB/

May 1, 2024

http://www2.eecs.berkeley.edu/Pubs/TechRpts/Hold/5791847d977b2449e52f4ca9cb7efddf.pdf

Modern machine learning applications increasingly rely on large volumes of contextual data, such as pre-computed features and embeddings, to make accurate predictions. However, maintaining the freshness of this derived data across geographically distributed cloud environments presents significant challenges. This thesis explores novel approaches to ensure data freshness for model serving in multi-cloud and multi-region settings, focusing on both feature maintenance and efficient data transfer.

First, we present RALF, a system that optimizes feature updates by leveraging feedback from downstream models to prioritize updates that have the greatest impact on prediction accuracy. RALF introduces the concept of "feature store regret" to quantify the impact of stale features on model performance, enabling more efficient use of computational resources while maintaining high prediction quality.

Building on this foundation, we then address the challenge of efficient data transfer across cloud regions and providers. Skyplane introduces a cloud-aware overlay network that optimizes both cost and throughput for large-scale data transfers. This approach enables faster and more cost-effective replication of feature data across distributed environments.

Finally, we present Cloudcast, which extends the ideas from Skyplane to the multicast setting. Cloudcast leverages cloud pricing models and ephemeral waypoints to minimize the cost of bulk data replication across multiple destinations, further improving the efficiency of maintaining fresh data copies across distributed model serving infrastructure.

Together, these systems form a comprehensive approach to maintaining fresh, derived data for machine learning applications in multi-cloud environments. By addressing both the computational aspects of feature maintenance and the networking challenges of data transfer, this thesis provides a foundation for building more efficient and accurate distributed model serving systems.

---

BibTeX citation:

@phdthesis{31523,
    Editor= {Gonzalez, Joseph and Stoica, Ion},
    Title= {Ensuring Data Freshness Across Clouds for Model Serving},
    School= {EECS Department, University of California, Berkeley},
    Year= {2024},
    Number= {UCB/},
    Abstract= {Modern machine learning applications increasingly rely on large volumes of contextual data, such as pre-computed features and embeddings, to make accurate predictions. However, maintaining the freshness of this derived data across geographically distributed cloud environments presents significant challenges. This thesis explores novel approaches to ensure data freshness for model serving in multi-cloud and multi-region settings, focusing on both feature maintenance and efficient data transfer.

First, we present RALF, a system that optimizes feature updates by leveraging feedback from downstream models to prioritize updates that have the greatest impact on prediction accuracy. RALF introduces the concept of "feature store regret" to quantify the impact of stale features on model performance, enabling more efficient use of computational resources while maintaining high prediction quality.

Building on this foundation, we then address the challenge of efficient data transfer across cloud regions and providers. Skyplane introduces a cloud-aware overlay network that optimizes both cost and throughput for large-scale data transfers. This approach enables faster and more cost-effective replication of feature data across distributed environments.

Finally, we present Cloudcast, which extends the ideas from Skyplane to the multicast setting. Cloudcast leverages cloud pricing models and ephemeral waypoints to minimize the cost of bulk data replication across multiple destinations, further improving the efficiency of maintaining fresh data copies across distributed model serving infrastructure.

Together, these systems form a comprehensive approach to maintaining fresh, derived data for machine learning applications in multi-cloud environments. By addressing both the computational aspects of feature maintenance and the networking challenges of data transfer, this thesis provides a foundation for building more efficient and accurate distributed model serving systems.},
}

EndNote citation:

%0 Thesis
%E Gonzalez, Joseph 
%E Stoica, Ion 
%T Ensuring Data Freshness Across Clouds for Model Serving
%I EECS Department, University of California, Berkeley
%D 2024
%8 May 1
%@ UCB/
%F 31523