data-engineer-handbook

# The Data Engineering Handbook
<a href="https://trendshift.io/repositories/8755" target="_blank"><img src="https://trendshift.io/api/badge/repositories/8755" alt="DataExpert-io%2Fdata-engineer-handbook | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>

This repo has all the resources you need to become an amazing data engineer!

### Join the free Databricks AI boot camp on August 3rd [here](https://learn.dataexpert.io/program/the-one-week-beginners-databricks-boot-camp-7129) and signup for Databricks Free Edition [here](https://signup.databricks.com/?provider=DB_FREE_TIERutm_source=github&utm_medium=video&utm_campaign=DataExpert)

## Getting started

If you are new to data engineering, start by following this [2024 breaking into data engineering roadmap](https://blog.dataengineer.io/p/the-2024-breaking-into-data-engineering)

If you are here for the [4-week free beginner boot camp](https://learn.dataexpert.io/program/the-absolute-beginner-data-engineering-boot-camp-starting-august-7th-6453/details) you can check out:
- [introduction](beginner-bootcamp/introduction.md)
- [software needed](beginner-bootcamp/software.md)

If you are here for the [6-week free intermediate boot camp](https://learn.dataexpert.io/program/free-community-boot-camp/details) you can check out
- [introduction](intermediate-bootcamp/introduction.md)
- [software needed](intermediate-bootcamp/software.md)


For more applied learning:
- Check out the [projects](projects.md) section for more hands-on examples!
- Check out the [interviews](interviews.md) section for more advice on how to pass data engineering interviews!
- Check out the [books](books.md) section for a list of high quality data engineering books
- Check out the [communities](communities.md) section for a list of high quality data engineering communities to join
- Check out the [newsletter](newsletters.md) section to learn via email 


## Resources

### Great [list of over 25 books](books.md)

Top 3 must read books are:
- [Fundamentals of Data Engineering](https://www.amazon.com/Fundamentals-Data-Engineering-Robust-Systems/dp/1098108302/)
- [Designing Data-Intensive Applications](https://www.amazon.com/Designing-Data-Intensive-Applications-Reliable-Maintainable/dp/1449373321/)
- [Designing Machine Learning Systems](https://www.amazon.com/Designing-Machine-Learning-Systems-Production-Ready/dp/1098107969)

### Great [list of over 10 communities to join](communities.md):

Top must-join communities for DE:
- [DataExpert.io Community Discord](https://discord.gg/JGumAXncAK)
- [Data Talks Club Slack](https://datatalks.club/slack)
- [Data Engineer Things Community](https://www.dataengineerthings.org/)

Top must-join communities for ML:
- [AdalFlow Discord](https://discord.com/invite/ezzszrRZvT)
- [Chip Huyen MLOps Discord](https://discord.gg/dzh728c5t3)

### Companies:

- Orchestration  
  - [Mage](https://www.mage.ai)
  - [Astronomer](https://www.astronomer.io)
  - [Prefect](https://www.prefe