GIỚI THIỆU VỀ TẬP NÀY
Data engineering is constantly evolving and staying ahead means mastering tools like Apache Airflow. In this episode, we explore the world of data engineering with Alexandre Magno Lima Martins, Senior Data Engineer at Teya. Alexandre talks about optimizing data workflows and the smart solutions they've created at Teya to make data processing easier and more efficient.
Key Takeaways:
(02:01) Alexandre explains his role at Teya and the responsibilities of a data platform engineer.
(02:40) The primary use cases of Airflow at Teya, especially with dbt and machine learning projects.
(04:14) How Teya creates self-service DAGs for dbt models.
(05:58) Automating DAG creation with CI/CD pipelines.
(09:04) Switching to a multi-file method for better Airflow performance.
(12:48) Challenges faced with Kubernetes Executor vs. Celery Executor.
(16:13) Using Celery Executor to handle fast tasks efficiently.
(17:02) Implementing KEDA autoscaler for better scaling of Celery workers.
(19:05) Reasons for not using Cosmos for DAG generation and cross-DAG dependencies.
(21:16) Alexandre's wish list for future Airflow features, focusing on multi-tenancy.
Resources Mentioned:
Alexandre Magno Lima Martins -
https://www.linkedin.com/in/alex-magno/
Teya -
https://www.linkedin.com/company/teya-global/
Apache Airflow -
https://airflow.apache.org/
dbt -
https://www.getdbt.com/
Kubernetes -
https://kubernetes.io/
KEDA -
https://keda.sh/
Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow #MachineLearning
Key Takeaways:
(02:01) Alexandre explains his role at Teya and the responsibilities of a data platform engineer.
(02:40) The primary use cases of Airflow at Teya, especially with dbt and machine learning projects.
(04:14) How Teya creates self-service DAGs for dbt models.
(05:58) Automating DAG creation with CI/CD pipelines.
(09:04) Switching to a multi-file method for better Airflow performance.
(12:48) Challenges faced with Kubernetes Executor vs. Celery Executor.
(16:13) Using Celery Executor to handle fast tasks efficiently.
(17:02) Implementing KEDA autoscaler for better scaling of Celery workers.
(19:05) Reasons for not using Cosmos for DAG generation and cross-DAG dependencies.
(21:16) Alexandre's wish list for future Airflow features, focusing on multi-tenancy.
Resources Mentioned:
Alexandre Magno Lima Martins -
https://www.linkedin.com/in/alex-magno/
Teya -
https://www.linkedin.com/company/teya-global/
Apache Airflow -
https://airflow.apache.org/
dbt -
https://www.getdbt.com/
Kubernetes -
https://kubernetes.io/
KEDA -
https://keda.sh/
Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow #MachineLearning
Tiếng Anh
Hoa Kỳ
BẢN CHÉP ÂM THANH 🔗
WEBVTT
00:00:00.239 --> 00:00:14.000
If there is any way of like maybe separating secrets per teams in Airflow, of course, you would need to create the concept of teams or like domains inside Airflow to be able to create a secret per team.
00:00:14.160 --> 00:00:23.199
So those like minor things that actually if you want to use the Airflow in a multi-tenant way, right now you would need to deal with these things yourself.
00:00:23.280 --> 00:00:28.000
And uh if it comes with the next release in …
00:00:00.239 --> 00:00:14.000
If there is any way of like maybe separating secrets per teams in Airflow, of course, you would need to create the concept of teams or like domains inside Airflow to be able to create a secret per team.
00:00:14.160 --> 00:00:23.199
So those like minor things that actually if you want to use the Airflow in a multi-tenant way, right now you would need to deal with these things yourself.
00:00:23.280 --> 00:00:28.000
And uh if it comes with the next release in …
TÌM KIẾM CÁC TẬP TRƯỚC
Tìm kiếm các tập trước của The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI.
Tuyên bố miễn trừ trách nhiệm: Podcast và tác phẩm nghệ thuật được nhúng trên trang này là của Astronomer, là tài sản của chủ sở hữu và không được liên kết hoặc xác nhận bởi Listen Notes, Inc.
CHỈNH SỬA
Cảm ơn bạn đã giúp cơ sở dữ liệu podcast luôn cập nhật.