AWS Technology
AWS S3 Iceberg Tables
Amazon S3 Iceberg Tables introduced fully managed Apache Iceberg table support to S3, optimizing the storage and querying of tabular data for analytics. By cre…
Read More »
AWS TechnologyRelational, Document, Key Value Databases: summary, use cases
There are a wide variety of databases. With cloud and hybrid architectures often see the following: Relational DB A structured format with rows and columns, re…
Read More »
AWS TechnologyBatching and AWS S3 on prem to S3 cloud
A useful architecture to move data from on-premises to AWS is to consider using AWS S3 outputs and move data directly over a Direct Connect to S3 in AWS. This …
Read More »
DataAWS DataBase Migration Service Overview
AWS Database Migration Service or DMS is a mature process to move on premises data to the AWS cloud, including to a S3 Data Lake. It is not recommended that fi…
Read More »
DataData Platform Automation
The problems with Data Pipelines and the hydration of a Data Lake include: Data teams often end with technical debt surrounding CI/CD, IaS, observability, and …
Read More »
AgileData Ops, a summary
In essence Data Operations is based on DevSecOps or DevOps and applies these same ideas to the life cycle of data management. DataOps advocates for agility, na…
Read More »
AgileDataLake and its Challenges
DataLake The entire concept of a Data Operations Platform rests on top of a Data Lake. There is no simple definition of a Data Lake, but based on the author’s …
Read More »
AgileData Operations + Agile
Data Operations ‘DataOps’ has been inspired by the Agile-premised ‘Development Operations’ model. The ‘DevOps’ model which usually includes security (DevSecOps…
Read More »
Cloud ModelsApache Iceberg: an overview
The icebergth is hereth. Apache Iceberg is an open-source table format for large-scale data systems, designed to provide efficient and reliable management of s…
Read More »
DataDatabricks Lakeflow (replaces Airflow)
Databricks LakeFlow is built on top of Databricks Workflows and Delta Live Tables. It is an implementation of Apache Airflow built into the Databricks eco syst…
Read More »
AWS TechnologyData Partitioning
Data files or tables are parsed into smaller units. This is also called ‘partitioning’. A partition is usually performed against a primary attribut…
Read More »
Cloud Design GoalsParquet file format for Data Lakes
Parquet is a file format standard used in many enterprises. It allows the standardisation of files and provides a common framework for queries and storage. Par…
Read More »