DeveloperWeek 2021 DeveloperWeek 2021
Join this event to build your agenda.

PRO WORKSHOP: Fugue: A New Way to Think and Solve Distributed Computing Problems

DeveloperWeek PRO Stage C

Han Wang
Lyft, Staff Engineer, Tech Lead

Han Wang is the tech lead of Lyft Machine Learning Platform, focusing on distributed computing solutions. Before joining Lyft, he worked at Microsoft, Hudson River Trading, Amazon and Quantlab. Han is the founder of the Fugue project, aiming at democratizing distributed computing and machine learning.

Through my career, I have seen and solved so many different computing problems. I find the challenge is not we can't solve large scale problems, the challenge is not we are lack of great tools. The real challenge is lack of consistency: solving different scale problems requires different programming models and frameworks, solving ETL and machine learning problems requires different skillsets, mindsets and even different languages.

In this session, I am going to introduce the Fugue project, whose purpose is to:
* unify the key concepts of distributed computing and machine learning problems
* unify the key interfaces for end to end ETL and machine learning pipelines
* decouple user's code and mindset from the underlying compute engines such as Spark and Ray and Fugue itself

I am also going to go through a couple of real examples how Fugue can shape your mind and optimize your code.

Fugue is already open sourced at: