AWS Technology
AWS RDS summary
Amazon RDS (Relational Database Service) RDS is a managed relational database service that supports various database engines such as MySQL, PostgreSQL, Oracle,…
Read More »
AWS TechnologyAWS S3 Iceberg Tables
Amazon S3 Iceberg Tables introduced fully managed Apache Iceberg table support to S3, optimizing the storage and querying of tabular data for analytics. By cre…
Read More »
AWS TechnologyRelational, Document, Key Value Databases: summary, use cases
There are a wide variety of databases. With cloud and hybrid architectures often see the following: Relational DB A structured format with rows and columns, re…
Read More »
AWS TechnologyBatching and AWS S3 on prem to S3 cloud
A useful architecture to move data from on-premises to AWS is to consider using AWS S3 outputs and move data directly over a Direct Connect to S3 in AWS. This …
Read More »
AgileData Ops, a summary
In essence Data Operations is based on DevSecOps or DevOps and applies these same ideas to the life cycle of data management. DataOps advocates for agility, na…
Read More »
AWS TechnologyData Partitioning
Data files or tables are parsed into smaller units. This is also called ‘partitioning’. A partition is usually performed against a primary attribut…
Read More »
AWS TechnologyAWS and Data Pipeline Ingestion
[Data engineering lifecycle from “Fundamentals of Data Engineering” by Matt Housley] Data Ingestion Challenges Data ingestion can be complicated. There are usu…
Read More »
AWS TechnologyAWS Glue introduction
AWS Glue is a meta data catalogue service with Extract-Transform-Load logic. The Glue catalogue is based on Hive and is a MySQL DB and a Java front end. Glue &…
Read More »
AWS TechnologyStandard Technology Stack for an AWS based Data Lake
A typical Technology Stack for a Data Lake. S3 as the Golden Source. Snowflake as a corporate Data Share with SQL use cases. If AWS-S3 and Redshift are not pro…
Read More »