<aside> ☝

List of notes for this specialization + Lecture notes & Repository & Quizzes + Home page on Coursera. Read this note alongside the lecture notes—some points aren't mentioned here as they're already covered in the lecture notes.

</aside>

Plan for Course 2

• This course focuses on 1st 2 steps of the DE life cycle — data generation and source systems and data ingestion from those source system.

  

  DE life cycle.

• People misunderstand that 80% are for modeling and 20% are for ingestion but it’s not true, it’s the opposite!

• Lot of the time is actually spent thinking about the data → for all of AI workloads.

• In this course, we not only work with structured data, but also with text, image data and so forth. The volume of unstructured data in the world is much greater than the volume of structured data.

The plan:

• Week 1 - Common source systems
  • Databases, object storage, and streaming sources
  • Working with source systems on AWS
• Week 2 - Setting up ingestion from source systems
• Week 3 - DataOps undercurrent
  • Automating some of your pipeline tasks
  • Monitoring data quality
• Week 4 - Orchestration, monitoring, and automating data pipelines
  • Setting up directed acyclic graphs using Airflow.
  • Working with infrastructure as code

Intro to Source Systems

Different types of source systems

• 3 types of data:

  • Structured Data: data organized as tables of rows and columns. Eg: excel, SQL, csv,…
  • Semi-Structured data: Data that is not in tabular form but still has some structure. Eg: JSON format.
  • Unstructured data: Data that does not have any predefined structure. Eg: text, video, audio, images…

• 3 types of source system to ingest (they don’t need to be corresponding 1-to-1 with 3 types of data): Databases, Files and Streaming Systems.

  

  In this figure: CRUD, DBMS, relational databases, non-relational (NoSQL) databases.

  

  any types of files. They’re one of the most common source systems you’ll work with as a DE.

  

As a data engineer, you'll extract raw data from various sources like databases, files, and streaming systems. This data can be structured, semi-structured, or unstructured.

Relational Databases

• The most common type of source system you’ll interact with is relational database.
• Where? → many web and mobile apps use relational database as backend. corporate systems like customer relationship, human resourec, …. Online Transaction Processing (OLTP) = execute a high volume of transactions concurrently like banking and bookings.
• Why? → if we use one big table (OBT) for everything → informations may be duplicated in multiple rows.