AWS Technology
AWS Glue Use Case examples
AWS Glue can serve a wide array of data engineering use cases. Loading Data To The Data Warehouse: One of Glue’s earliest and most popular use cases is loading…
Read More »- AWS Technology
AWS, ETL with Glue
AWS Glue was designed based on these principles: Provide customers the ability to solve problems when the system can not satisfy their needs. Examples include …
Read More » 
AWS TechnologyMigrating ‘Big Data’ processes to AWS
Migrating ‘ Big Data’ pipelines and data processes to AWS will allow organizations to: Scale Seamlessly: AWS services like EMR (to replace Hadoop),…
Read More »
AWS TechnologyAWS Data Lake, Streaming Architecture summary
Example project deployment at a Financial Institution. Creation of a Data Lake with a streaming/real time data ingestion requirement. AWS Data Architecture Des…
Read More »
AWS TechnologyAWS RDS summary
Amazon RDS (Relational Database Service) RDS is a managed relational database service that supports various database engines such as MySQL, PostgreSQL, Oracle,…
Read More »
DataAWS Redshift, a summary
Amazon Redshift Redshift is a fully managed data warehousing service designed for online analytical processing (OLAP) workloads. It is optimized for high-perfo…
Read More »
Cloud Design GoalsImproving data queries, pipelines
On real world projects and deployments, you hear the lament that a datawarehouse or data engine ‘does not work’. Query response times are slow, it …
Read More »
AWS TechnologyAWS S3 Iceberg Tables
Amazon S3 Iceberg Tables introduced fully managed Apache Iceberg table support to S3, optimizing the storage and querying of tabular data for analytics. By cre…
Read More »
AWS TechnologyRelational, Document, Key Value Databases: summary, use cases
There are a wide variety of databases. With cloud and hybrid architectures often see the following: Relational DB A structured format with rows and columns, re…
Read More »
AWS TechnologyBatching and AWS S3 on prem to S3 cloud
A useful architecture to move data from on-premises to AWS is to consider using AWS S3 outputs and move data directly over a Direct Connect to S3 in AWS. This …
Read More »
DataAWS DataBase Migration Service Overview
AWS Database Migration Service or DMS is a mature process to move on premises data to the AWS cloud, including to a S3 Data Lake. It is not recommended that fi…
Read More »
DataData Platform Automation
The problems with Data Pipelines and the hydration of a Data Lake include: Data teams often end with technical debt surrounding CI/CD, IaS, observability, and …
Read More »