Big Data Engineer

Basics of Business Intelligence (approx. 2 days)

Fields of application, dimensions of a BI architecture

Basics of business intelligence, OLAP, OLTP, tasks of data engineers

Data Warehousing (DWH): handling and processing of structured, semi-structured and unstructured data

Requirements management (approx. 2 days)

Tasks, objectives and procedures in requirements analysis

Data modeling, introduction/modeling with ERM

Introduction/modeling in UML

- Class diagrams

- Use case analysis

- Activity diagrams

Artificial intelligence (AI) in the work process

Presentation of specific AI technologies

and possible applications in the professional environment

Databases (approx. 3 days)

Basics of database systems

Architecture of database management systems

Application of RDBMS

Implementation of data model in RDBMS, normal forms

Practical and theoretical introduction to SQL

Limits of relational databases, csv, json

Data Warehouse (approx. 4 days)

Star Schema

Data modeling

Creation of Star Schema in RDBMS

Snowflake Schema, basics, data modeling

Creation of Snowflake Schema in RDBMS

Galaxy Schema: Basics, data modeling

Slowly Changing Dimension Tables Type 1 to 5 - Restating, Stacking, Reorganizing, mini Dimension and Type 5

Introduction to normal, causal, mini and monster, heterogeneous and sub dimensions

Comparison of state and transaction oriented

Fact tables, density and storage from DWH

ETL (approx. 4 days)

Data Cleansing

- Null Values

- Preparation of data

- Harmonization of data

- Application of regular expressions

Data Understanding

- Data validation

- Statistical data analysis

Data protection, data security

Practical structure of ETL routes

Data Vault 2.0, basics, hubs, links, satellites, hash key, hash diff.

Data Vault data modeling

Practical structure of a Data Vault model - Raw Vault, practical implementation of hash procedures

Project work (approx. 5 days)

To consolidate the content learned

Presentation of the project results

What is Big Data? (approx. 1 day)

Volume, Velocity, Variety, Value, Veracity

Opportunities and risks of large amounts of data

Differentiation: business intelligence, data analytics, data science

Introduction to data mining

Role of AI and data-driven systems in the big data environment

Introduction to big data frameworks (approx. 2 days)

Big data solutions in the cloud (overview of AWS, Azure, GCP)

Data access patterns

Data storage

Introduction to data lakes and data warehouses

Overview of Apache Hadoop and Spark

Distributed data processing with Spark (approx. 3 days)

Basics of distributed systems

Apache Spark (Core and SQL)

Comparison of different approaches to data processing

Processing large amounts of data

Introduction to simple ML workflows with Spark

Data pipelines and data integration (approx. 2 days)

ETL and ELT processes

Batch vs. streaming processing

Basics of data pipelines

Introduction to orchestration (e.g. Airflow overview)

Data quality and preparation

Components (approx. 2 days)

Brief presentation of various tools

Data transfer

Overview of resource management in big data systems

Hadoop ecosystem

Apache Spark deepening

Introduction to streaming technologies

NoSQL and data storage (approx. 2 days)

CAP theorem

ACID and BASE

Types of databases

HBase

Introduction to document-oriented databases

Introduction to storage formats

Overview of data lakehouse approaches

Big Data Visualization (approx. 2 days)

Theories of visualization

Diagram selection

New types of diagrams

Tools for data visualization

Introduction to BI tools (e.g. Power BI, Tableau)

Basics of data-driven decision making

Data governance and data protection (approx. 1 day)

Basics of the GDPR in the data context

Data ethics and responsible handling of data

Data quality and governance concepts

Access controls and security

Fundamentals of responsible AI use

Project work (approx. 5 days)

To consolidate the content learned

Presentation of the project results