DATA ENGINEER BLOG

1] Post: Microsoft Fabrics


  Understand what is Microsoft Fabrics in Simple Way...


Written Notes:::>>>>  Link2Pdf

Reference :>>> Youtube Video Link2video


________________________________________________________________________________

2] Post : Celebal Technology Link

 Celebal Tracker:

 Tracker :> Link

Sample :> Link

Colab :> Link

Curiculum :> Link

Youtube Links 

https://www.w3schools.com/sql/sql_ref_sqlserver.asp

Naval Sir Channel

https://www.youtube.com/@thedatamaster

https://youtu.be/tZw4Nrv5X9g

Company Initernals

 Celebal Company Internal : Drive



Naval Yemul Sirs Link:

Databricks: https://learn.microsoft.com/en-gb/azure/databricks/

Delta Lake:   https://delta.io/

Pyspark:>>> https://sparkbyexamples.com/

Azure Linked Service:>> https://youtube.com/playlist?list=PL7S7dD8r4QdUcHTTF-EYS3aYzuZiisuax


-------------------------------------------------------------------------------------------------

30 June :>>>>>>>>

DP 203:  Link  

Azure Data Factory: Youtube

Architecture:> Architecture

Image : img

----------------------------------------------------------------------------------------------------------

27 June :>>>

Naval Sir NoteBook Link: Navalsir27-Link

Datasets Naval Sirs Drive: > Drive

----------------------------------------------------------------------------------------------------------------

23rd June:>


Youtube 

1) Big DATA PySpark :>  https://www.youtube.com/playlist?list=PL7S7dD8r4QdWUXL4zOC6NYnxDD8OmIVLr

2) Delta Lake > 40,41
https://www.youtube.com/playlist?list=PL7S7dD8r4QdVzOYRzIG2UJdCaCasqBv1F
All about Delta Lake:> https://youtu.be/agtUI25LxuA

Naval Yemul:> 9689777700


3) Spark Eg> To Learn Spark
https://sparkbyexamples.com/pyspark-tutorial/?expand_article=1


4) Do These Channel You Will Be Master in SQL:>
Do Hands On....
 Ankit Bansal:> https://youtu.be/dOLBRfwzYcU
Kudvenkat:> https://youtu.be/7GVFYt6_ZFM


SIMPLE FUNDA:>
<<<<<<<<<<<<<<<SQL------PYHTON------PYSPARK>>>>>>>>>>

------------------------------------------------------------------------------------------------------------------------

 

DE Course (SQL) Curiculum :> Link

SQL W3 Schools :---->>> Link  

You Tube :---->>>  Youtube

Joins :------->> Link

Import data From CSV to SQL Server :> Youtube

Big Data: PySpark ~ Naval Sir : Youtube

Azure Cloud | Make Data your Career ~ Naval Sir  : Youtube

Start New > Try Databricks free  : Link

Data Bricks Login : Link 

https://spark.apache.org/docs/latest/api/python/reference/index.html
spark

notebook

https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/4223192473721802/3413217916972752/2212737435575810/latest.html

https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/4223192473721802/3413217916972752/2212737435575810/latest.html


 

Databricks: https://learn.microsoft.com/en-gb/azure/databricks/

Delta Lake:   https://delta.io/

Pyspark:>>> https://sparkbyexamples.com/

Azure Linked Service:>> https://youtube.com/playlist?list=PL7S7dD8r4QdUcHTTF-EYS3aYzuZiisuax


-------------------------------------------------------------------------------------------------

30 June :>>>>>>>>

DP 203:  Link  

Azure Data Factory: Youtube

Architecture:> Architecture

Image : img

----------------------------------------------------------------------------------------------------------

27 June :>>>

Naval Sir NoteBook Link: Navalsir27-Link

Datasets Naval Sirs Drive: > Drive

----------------------------------------------------------------------------------------------------------------

23rd June:>


Youtube 

1) Big DATA PySpark :>  https://www.youtube.com/playlist?list=PL7S7dD8r4QdWUXL4zOC6NYnxDD8OmIVLr

2) Delta Lake > 40,41
https://www.youtube.com/playlist?list=PL7S7dD8r4QdVzOYRzIG2UJdCaCasqBv1F
All about Delta Lake:> https://youtu.be/agtUI25LxuA

Naval Yemul:> 9689777700


3) Spark Eg> To Learn Spark
https://sparkbyexamples.com/pyspark-tutorial/?expand_article=1


4) Do These Channel You Will Be Master in SQL:>
Do Hands On....
 Ankit Bansal:> https://youtu.be/dOLBRfwzYcU
Kudvenkat:> https://youtu.be/7GVFYt6_ZFM


SIMPLE FUNDA:>
<<<<<<<<<<<<<<<SQL------PYHTON------PYSPARK>>>>>>>>>>

------------------------------------------------------------------------------------------------------------------------

 

DE Course (SQL) Curiculum :> Link

SQL W3 Schools :---->>> Link  

You Tube :---->>>  Youtube

Joins :------->> Link

Import data From CSV to SQL Server :> Youtube

Big Data: PySpark ~ Naval Sir : Youtube

Azure Cloud | Make Data your Career ~ Naval Sir  : Youtube

Start New > Try Databricks free  : Link

Data Bricks Login : Link 

https://spark.apache.org/docs/latest/api/python/reference/index.html
spark

notebook

https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/4223192473721802/3413217916972752/2212737435575810/latest.html

https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/4223192473721802/3413217916972752/2212737435575810/latest.html


3] Post :  Ideas Related to career development

 Ideas related to career development

Ideas

  • Do 
  • ALL NOTES DRIVE : save there PDF=> Sunday:16 June
  • Add Language bar in resume as Data Engineer so SQL
  • All python, sql base one blog study and revise it
  • Fuel Internship project: Make there IPL
  • AMCAT REport drive upload
  • Zensar Certificate Drive and Linked in Certification section Edit
  • Celebal Internship Experience BLOG
  • Linked in Exp Section: Fuel Intstitute
  • Big Data  Book : fuel intern pages PDF  and  upload to drive tear attached pages  
__________________________________________________________________________________

Post 4] Data Engineer Links:

How would i learn DE from start :

             


    RP-2 How would i learn DE from start






 

Road-Map-Link

            How would I learn DE : Link

          Roadmap Darshil : Link

--------------------------------------------------------------------------------------------------------

  1. Computer Science Fundamentals (If you don’t have a CS background)

Watch this if you don’t have a computer science background, as a Data Engineer having good knowledge of CS fundamentals is important to understand big systems and how they work

Watching these videos will give you a basic understanding of CS fundamentals  

6527::You can watch the first 7 lectures from this playlist

  1. CS50 2022
  2. Book - Grokking Algorithms: An illustrated guide

  1. Programming Language

                Do any courses, your main goal here is to understand how to write basic Python

                Code and how to work with different datasets!

  1. Darshil - Python for Data Engineering  (Recommended)
  2. DataCamp - Data Engineering With Python 
  3. Coursera - Python for Everybody Specialization (Do this if you don’t know anything about python)
  4. Udemy - Python Bootcamps: Learn Python Programming and Code Training 
  5. freeCodeCamp - Learn Python - Full Course for Beginners

Practice Projects:

  • Scrape Data Using BeautifulSoup Library eg. Amazon, Covid, Wikipedia, or any website you like
  • Build A Calculator Using Python

  1. SQL (Structured Query Language)

Learn about the basics of SQL and how to write queries, once you complete the course make sure you do hands-on practice on Hackerrank or any website you like!

  1. Udemy - The Complete SQL Bootcamp for the Manipulation and Analysis of Data (Recommended)
  2. Coursera - SQL for Data Science
  3. DataCamp - Intro To SQL DataCamp

        Practice SQL here

  1. Basics Of Linux

Why Linux? Because you will be working with many remote machines, doing SSH to access them, and performing operations so it’s important to learn them.

You don’t have to remember all the commands but just understand what they do and how to write them

  1. Udemy - Linux for Beginners: Linux Basics
  2. Coursera - Linux Fundamentals
  3. freeCodeCamp - Top 50 Most Popular Linux Commands  (Recommended)

        

Do Hands-On Project

  1. Big Data Fundamentals

This section is theoretical and you need to understand how big data system works and their history of them

  1. Coursera - Big Data Specialization (Recommended)
  2. Udemy - Learn Big Data: The Hadoop Ecosystem Masterclass (Do this if you want to learn about legacy systems)

  1. Data Warehouse Fundamentals + Tool

        Learn Fundamentals and then learn one tool, Snowflake, BigQuery, Redshift, etc… Just learn one and you are good!

  1. Fundamentals
  1. Coursera - Data Warehousing for Business Intelligence Specialization (recommended for deep dive)
  2. Udemy - Data Warehouse Fundamentals for Beginners (recommended for quick learning)
  1. Tools
  1. Snowflake - Snowflake – The Complete Masterclass
  2. Snowflake Doc - https://www.snowflake.com/certifications/ 

  1. Learn Batch Processing + Tool
  1. Spark Fundamentals
  1. DataCamp - Big Data Fundamentals with PySpark (recommended)
  2. Udemy - Spark and Python for Big Data with PySpark
  1. Databricks
  1. Udemy - Azure Databricks & Spark Core
  2. Udemy - Databricks Certified Data Engineer Associate
  3. Coursera - Databricks for Data Engineering

  1. Learn RealTime Streaming
  1. Realtime Streaming (Kafka)
  1. Udemy - Apache Kafka Course for Beginners: Learn Kafka Online (check this)
  2. edX - Building ETL and Data Pipelines with Bash, Airflow, and Kafka

        

Do Hands-On Project - Stock Market Real-Time Streaming Pipeline

  1. Data Orchestration (AirFlow)
  1. Udemy - The Complete Hands-On Introduction to Apache Airflow
  2. DataCamp - Airflow

Do Hands-On Project - Twitter Data Pipeline using Airflow

  1. Cloud Computing

        Advance section, do courses, and then do the certification to add value in your

        Resume, If you are new then start with AWS but if you know about

        other clouds then you can do that too!

  1. AWS (Amazon Web Services)
  1. Udemy - Ultimate AWS Certified Cloud Practitioner
  2. Udemy - Ultimate AWS Certified Solutions Architect Associate (SAA)
  3. Coursera - AWS Solution Architect Associate
  1. GCP (Google Cloud Platform)
  1. Coursera - Cloud Data Engineer Professional Certificate
  1. Microsoft Azure
  1. Coursera - Microsoft Azure Data Engineering Associate
  2. Udemy - AZ-900: Microsoft Azure Fundamentals
  3. Udemy - Azure Data Engineer Certified:8 COURSE BUNDLE

Do Hands-On Project

  1. Build ETL Pipeline Using AWS Cloud
  2. Covid Data Analysis Project
  3. YouTube Data Analysis (End-To-End Data Engineering Project)

  1. Learn Modern Data Stack
  1. Learn Basics - https://analyticsindiamag.com/modern-data-stack-and-what-we-know-about-it/
  2. Dbt - https://www.getdbt.com/dbt-learn/
  3. Airbyte - https://airbyte.com/
  4. Fivetran - ​​https://www.fivetran.com/

  1. DataOps
  1. Docker Guide - https://www.coursera.org/projects/docker-for-absolute-beginners
  2. Udemy - Docker & Kubernetes: The Practical Guide 

Recommended Books

  1. Designing Data-Intensive Applications
  2. Fundamentals of Data Engineering
  3. The Data Warehouse Toolkit

Read Real-World Case Studies

  1. Netflix - https://netflixtechblog.medium.com/
  2. AWS - https://aws.amazon.com/solutions/case-studies/
  3. GCP - https://cloud.google.com/customers
  4. Azure - https://azure.microsoft.com/en-us/resources/customer-stories/ 

Follow Me Here:

  1. Twitter - https://twitter.com/parmardarshil07
  2. Linkedin - https://www.linkedin.com/in/darshil-parmar/
  3. YouTube - https://www.youtube.com/c/DarshilParmar 

All the best <3

_______________________________________________________________________

POST 5] Information of Data Engineer Job role

 


WIPRO DATA ENGINEER  4+ years EXPERIENCE PERSON KNOWLEDGE....



----------------------------------------------------------------------------------------------------------------

MLAI DIGITAL PVT LTD

Azure Databricks Engineer











Azure Open AI














People Tech Group 
Cloud Engineer











Prolifics
Azure Developers







TVS NXT 
Senior Azure DE










#QUADRA SYSTEM
AZURE DE


AZURE Technical Support Engineer


Win Wire
> AZURE DE













>> DATA Analyst



WIPRO 
>AZURE DE 







POST 6] MY PROJECTS

Revise these projects :

  1.  Uber -->  Link to video : Link
  2. Ipl
  3. Spotify

Revise, modify, understand, add resume section

 Project Ideas:>

  •  PYTHON :   PREPARE SCANNER (code scanner ok) 
  • note: why thought came because i want my photo inside and from side the QR code will be there 


UBER DATA ENGINEERING PROJECT:
  • Uber Data Engineering Project : Video
  • Self made Data Engineer : Video
  • How would I learn DE : Link
        Roadmap Darshil : Link
        Meeting link
 
____________________________________________________________________________

POST 7] DE Map

 Data Engineer


FUNDAMENTAL TRUTH: 

  1. Get Knowledge of 30+ 
  2. Get package of 30+ LPA

June Target
  • REVISE 
  • COURSE
  • DE
  • CE
  • Apply



July Target
  • OWN DE [Project]




August Target
  • DE MASTER [30+ LPA Knowledge]




Last Updated: 1 june saturday 6:30 Evening:

Next Move : Plan this on sunday ---> JUNE| JULY



 REVISE/ Projects :

  • Projects: uber,ipl,spotify pipeline [ Revise all and make notes of those project]


STUDY/Course:

  • Course Name: Apache Spark With DATA BRICKS for Data Engineering [Daily 2 Hour] 



NEW KNOWLEDGE:

  • videos youtube : Darshil bhau  [ Alternately Watch his all video when get time]
  • apply  (daily to know your level)
  • newsletter  [ WEEKLY]
  • linked in   [DAILY ]
  • DE Blogs Substack [DAILY]
  • General Podcast Youtube [Weekly] 




HOBBY:

  • Snap intro Daily Evening [Upload to youtube]
  •     6:30-6:45




PRACTICE: 

  • Python  [Daily 2 hour]   [M+E]
  • SQL  [Daily 1 Hour]
  • DataBricks  [Going on in Course]


RESUME

  •     Edit Continuosly


LEARN This
  •  Git hub
  • Power BI Tools
  • Co pilot

Comments

Post a Comment

Popular posts from this blog

[ STRUCTURED ] Unity Catalog Concept in Databricks

Image
>>>  Topics in UC   Definition History Before UC After UC Features and Limitations Eg > >>Additional Terms   Catalog Binding Delta Sharing   Lakehouse Fedaration   _______________________________________________________________________ >>  Definition   Unified Governance solution + Data Security >> History Databricks Was lacking in Fundamental data governance centralization over databricks >> B4 Vs After Previous Approach:Without UC[Separate] Lack governance and Centralization Previously We have to do governance on each workspace individually Seperate User Management, Metastore We Used Data Cloning Activity which increased the data duplication and also cost Current Approach: With UC[One] Unity Catalog Centralized governance model sit at top of workspaces There is single entity from there you can govern each and every entity in your workspace from one place Main Challenge : 1. User Group: We dont had centralized pl...
Read more