Data
Databricks Lakeflow (replaces Airflow)
Databricks LakeFlow is built on top of Databricks Workflows and Delta Live Tables. It is an implementation of Apache Airflow built into the Databricks eco syst…
Read More »
DataAutomating S3 to Redshift with Glue
A straightfoward method to automate data ingestion from S3 buckets (data lake) to a Redshift (data warehouse) cluster; by using Glue. Create a Redshift cluster…
Read More »
AWS TechnologyAWS and Data Pipeline Ingestion
[Data engineering lifecycle from “Fundamentals of Data Engineering” by Matt Housley] Data Ingestion Challenges Data ingestion can be complicated. There are usu…
Read More »
AWS TechnologyAWS Glue introduction
AWS Glue is a meta data catalogue service with Extract-Transform-Load logic. The Glue catalogue is based on Hive and is a MySQL DB and a Java front end. Glue &…
Read More »
DataData Lake Design and Change Data Capture
Data flowing into the Data Lake obviously changes. Data table changes are captured by CDC or change data capture. Changes in the source database are delivered …
Read More »
DataRedshift vs Redshift Spectrum
Amazon Redshift is a petabyte scalable columnar data warehouse that is very efficient in storing raw data and collecting data from various sources. Redshift su…
Read More »
Cloud Design GoalsData products – a summary
Data products are the end result of file or data movements to the cloud; ETL; processing; de-duplication; curation and storage in a consumable layer. There is …
Read More »
AWS TechnologyStandard Technology Stack for an AWS based Data Lake
A typical Technology Stack for a Data Lake. S3 as the Golden Source. Snowflake as a corporate Data Share with SQL use cases. If AWS-S3 and Redshift are not pro…
Read More »
DataImproving Redshift Data Warehouse performance
(ETL engine in the above could be AWS Glue) There are various ways to define performance and what that means. A simple way to be consistent with management is …
Read More »