Agile
Data Ops, a summary
In essence Data Operations is based on DevSecOps or DevOps and applies these same ideas to the life cycle of data management. DataOps advocates for agility, na…
Read More »
AgileDataLake and its Challenges
DataLake The entire concept of a Data Operations Platform rests on top of a Data Lake. There is no simple definition of a Data Lake, but based on the author’s …
Read More »
AgileData Operations + Agile
Data Operations ‘DataOps’ has been inspired by the Agile-premised ‘Development Operations’ model. The ‘DevOps’ model which usually includes security (DevSecOps…
Read More »
Cloud ModelsApache Iceberg: an overview
The icebergth is hereth. Apache Iceberg is an open-source table format for large-scale data systems, designed to provide efficient and reliable management of s…
Read More »
DataDatabricks Lakeflow (replaces Airflow)
Databricks LakeFlow is built on top of Databricks Workflows and Delta Live Tables. It is an implementation of Apache Airflow built into the Databricks eco syst…
Read More »
AWS TechnologyData Partitioning
Data files or tables are parsed into smaller units. This is also called ‘partitioning’. A partition is usually performed against a primary attribut…
Read More »
Cloud Design GoalsParquet file format for Data Lakes
Parquet is a file format standard used in many enterprises. It allows the standardisation of files and provides a common framework for queries and storage. Par…
Read More »
Cloud ModelsDatabricks and Snowflake: Summary
Databricks and Snowflake overlap in many areas. Firms deploying both need to clearly demarcate the epics and use case journeys to be supported by the technolog…
Read More »
DataAutomating S3 to Redshift with Glue
A straightfoward method to automate data ingestion from S3 buckets (data lake) to a Redshift (data warehouse) cluster; by using Glue. Create a Redshift cluster…
Read More »
AWS TechnologyAWS and Data Pipeline Ingestion
[Data engineering lifecycle from “Fundamentals of Data Engineering” by Matt Housley] Data Ingestion Challenges Data ingestion can be complicated. There are usu…
Read More »
AWS TechnologyAWS Glue introduction
AWS Glue is a meta data catalogue service with Extract-Transform-Load logic. The Glue catalogue is based on Hive and is a MySQL DB and a Java front end. Glue &…
Read More »
DataData Lake Design and Change Data Capture
Data flowing into the Data Lake obviously changes. Data table changes are captured by CDC or change data capture. Changes in the source database are delivered …
Read More »