What you’ll learn
- Set Up Data Ingestion In Fabric using different methods Pipelines, Dataflows Gen 2 and Pyspark Notebooks
- Clean and Transform Data : Set up data flows in Data Factory to clean and transform the raw data. Apply filters and remove duplicates as necessary.
- Aggregate and Process Data: Create additional data flows or Stored Proc to aggregate and process the cleaned data. Store the processed data in the GoldenDWH.
- Prepare Data for Reporting: Prepare the final dataset for the Power BI report, optimizing it for performance. Use the Gold Layer data to create the BI reports
- Automate the Workflow: Schedule the data pipelines to run daily. Set up triggers and alerts in Data Factory to notify stakeholders via email about the pipeline
- Configure Access Control Implement role-based access control (RBAC) to manage access to each layer of the medallion architecture.
I am a Microsoft big data engineer with 5 years of experience in Azure and data engineering. I’m passionate about developing and implementing scalable data solutions that help businesses unlock the full potential of their data
My expertise includes Azure services like Azure Data Factory, Azure Databricks, and Azure Stream Analytics, as well as big data technologies such as Hadoop, Spark, and Kafka. I love sharing my knowledge and helping others succeed in the field of data engineering. When I’m not working with data, I enjoy exploring new technologies and staying up-to-date on the latest developments in the industry.