Wednesday, February 17, 2021
Through my career, I have seen and solved so many different computing problems. I find the challenge is not we can't solve large scale problems, the challenge is not we are lack of great tools. The real challenge is lack of consistency: solving different scale problems requires different programming models and frameworks, solving ETL and machine learning problems requires different skillsets, mindsets and even different languages.
In this session, I am going to introduce the Fugue project, whose purpose is to:
* unify the key concepts of distributed computing and machine learning problems
* unify the key interfaces for end to end ETL and machine learning pipelines
* decouple user's code and mindset from the underlying compute engines such as Spark and Ray and Fugue itself
I am also going to go through a couple of real examples how Fugue can shape your mind and optimize your code.
Fugue is already open sourced at: https://github.com/fugue-project/fugue
Download these images to your phone and post using the Instagram app.