Implementing best practices for PySpark ETL jobs and applications.
-
Updated
Jan 1, 2023 - Python
#
etl-job
Here are 110 public repositories matching this topic...
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
python airflow spark apache-spark scheduler s3 data-engineering data-lake warehouse redshift data-migration livy etl-framework apache-airflow emr-cluster etl-pipeline etl-job data-engineering-pipeline airflow-dag goodreads-data-pipeline
-
Updated
Mar 9, 2020 - Python
A comprehensive guide to building a modern data warehouse with SQL Server, including ETL processes, data modeling, and analytics.
data-science sql sql-server etl data-warehouse sql-query datascience data-engineering data-analytics datawarehousing data-analysis sqlserver datawarehouse data-cleaning datalake data-warehousing etl-pipeline etl-job data-lakehouse medallion-architecture
-
Updated
Apr 23, 2025 - TSQL
Mass processing data with a complete ETL for .net developers
csv etl dotnet extract sftp entity-framework business-intelligence dotnet-core load transform dotnet-standard csv-parser csv-reader csv-writer etl-job
-
Updated
Sep 24, 2026 - C#
Provides guidance for fast ETL jobs, an IDataReader implementation for SqlBulkCopy (or the MySql or Oracle equivalents) that wraps an IEnumerable, and libraries for mapping entites to table columns.
-
Updated
Nov 21, 2025 - C#
-
Updated
Oct 28, 2024 - HTML
A Python PySpark Projet with Poetry
python template data-science spark etl poetry jupyter-notebook project python3 pytest pyspark data-engineering black spark-sql isort etl-pipeline etl-job pyspark-boilerplate-template
-
Updated
May 2, 2026 - Jupyter Notebook
This code creates a Kinesis Firehose in AWS to send CloudWatch log data to S3.
big-data analytics terraform kinesis-firehose cloudwatch-logs parquet terraform-provider etl-job terraform-aws big-data-processing
-
Updated
Aug 4, 2021 - HCL
An end-to-end Twitter Data Pipeline that extracts data from Twitter and loads it into AWS S3.
python airflow twitter etl twitter-api scheduler s3 data-engineering tweepy boto3 data-pipeline apache-airflow etl-pipeline etl-job data-engineering-pipeline airflow-dags twitter-data-pipeline
-
Updated
Aug 26, 2023 - Python
Built a Data Pipeline for a Retail store using AWS services that collects data from its transactional database (OLTP) in Snowflake and transforms the raw data (ETL process) using Apache spark to meet business requirements and also enables Data Analyst create Data Visualization using Superset. Airflow is used to orchestrate the pipeline
-
Updated
May 25, 2023 - Python
A declarative, SQL-like DSL for data integration tasks.
-
Updated
Jul 4, 2018 - Go
Extract transform load CLI tool for extracting small and middle data volume from sources (databases, csv files, xls files, gspreadsheets) to target (databases, csv files, xls files, gspreadsheets) in free combination.
database etl extractor google-sheets data-engineering data-lake business-intelligence dwh elt datapipeline data-pipeline extract-transform-load usql excel-to-sql etl-job etl-process sharepoint-integration sharepoint-sync sync-google-sheet
-
Updated
Aug 2, 2026 - Python
Airflow POC demo : 1) env set up 2) airflow DAG 3) Spark/ML pipeline | #DE
python shell docker infrastructure instagram data-science machine-learning airflow environment spark etl data-engineering travis daas elt dag etl-pipeline etl-job etl-jobs astro-airflow
-
Updated
Dec 19, 2022 - Python
PHP ETL engine with pluggable steps: extractors, transformers, loaders
csv etl transformation etl-framework etl-pipeline etl-job etl-jobs etl-automation etl-process etl-processes
-
Updated
Aug 2, 2026 - PHP
Introduction to the data pipeline management with Airflow. Airflow schedule and maintain numerous ETL processes running on a large scale Enterprise Data Warehouse.
-
Updated
Nov 26, 2018 - Python
Live and historical European air traffic: a VPS collector streams ADS-B, OpenSky movements, schedules and weather through Kafka into a Bronze/Silver lake, a dbt warehouse and a read-only serving copy behind a public dashboard. Budget-aware polling and atomic publishes keep it correct while unattended.
data-science data database apache-spark etl data-visualization datascience data-structures data-engineering dataset data-analytics data-analysis apache-kafka etl-framework apache-airflow etl-pipeline etl-job etl-automation etl-pipelines huggingface-spaces
-
Updated
Sep 25, 2026 - Python
A data pipeline from source to data warehouse using Taipei Metro Hourly Traffic data
python airflow s3 data-warehouse data-engineering redshift taipei-metro-stations etl-pipeline etl-job data-engineering-pipeline taipei-metro
-
Updated
Jul 3, 2023 - Jupyter Notebook
Add this topic to your repo
To associate your repository with the etl-job topic, visit your repo's landing page and select "manage topics."