Databricks Lakeflow (replaces Airflow)
Databricks LakeFlow is built on top of Databricks Workflows and Delta Live Tables. It is an implementation of Apache Airflow built into the Databricks eco syst…
Read More »
Data Partitioning
Data files or tables are parsed into smaller units. This is also called ‘partitioning’. A partition is usually performed against a primary attribut…
Read More »
Parquet file format for Data Lakes
Parquet is a file format standard used in many enterprises. It allows the standardisation of files and provides a common framework for queries and storage. Par…
Read More »
Databricks and Snowflake: Summary
Databricks and Snowflake overlap in many areas. Firms deploying both need to clearly demarcate the epics and use case journeys to be supported by the technolog…
Read More »
Automating S3 to Redshift with Glue
A straightfoward method to automate data ingestion from S3 buckets (data lake) to a Redshift (data warehouse) cluster; by using Glue. Create a Redshift cluster…
Read More »
AWS and Data Pipeline Ingestion
[Data engineering lifecycle from “Fundamentals of Data Engineering” by Matt Housley] Data Ingestion Challenges Data ingestion can be complicated. There are usu…
Read More »
AWS Glue introduction
AWS Glue is a meta data catalogue service with Extract-Transform-Load logic. The Glue catalogue is based on Hive and is a MySQL DB and a Java front end. Glue &…
Read More »
Data Lake Design and Change Data Capture
Data flowing into the Data Lake obviously changes. Data table changes are captured by CDC or change data capture. Changes in the source database are delivered …
Read More »
Redshift vs Redshift Spectrum
Amazon Redshift is a petabyte scalable columnar data warehouse that is very efficient in storing raw data and collecting data from various sources. Redshift su…
Read More »
Data products – a summary
Data products are the end result of file or data movements to the cloud; ETL; processing; de-duplication; curation and storage in a consumable layer. There is …
Read More »
Understanding Data Lakes and Data Warehouses
In simple terms we can identify the differences between Data Lakes and Data Warehouses. Data Lake: A data lake is a centralized repository, usually a platform,…
Read More »
Digital and Data Transformation (Why)
Digital Transformation Digital transformation not a magic solution nor a buffet of word salads. DT is roughly defined as the integration of digital technologie…
Read More »