Data Ops, a summary
In essence Data Operations is based on DevSecOps or DevOps and applies these same ideas to the life cycle of data management. DataOps advocates for agility, na…
Read More »
Data Partitioning
Data files or tables are parsed into smaller units. This is also called ‘partitioning’. A partition is usually performed against a primary attribut…
Read More »
AWS and Data Pipeline Ingestion
[Data engineering lifecycle from “Fundamentals of Data Engineering” by Matt Housley] Data Ingestion Challenges Data ingestion can be complicated. There are usu…
Read More »
AWS Glue introduction
AWS Glue is a meta data catalogue service with Extract-Transform-Load logic. The Glue catalogue is based on Hive and is a MySQL DB and a Java front end. Glue &…
Read More »
Standard Technology Stack for an AWS based Data Lake
A typical Technology Stack for a Data Lake. S3 as the Golden Source. Snowflake as a corporate Data Share with SQL use cases. If AWS-S3 and Redshift are not pro…
Read More »
Data Lake Basics and AWS
A data lake is a centralized repository that allows a firm to store structured and unstructured data at any scale. You can store your data as-is, without havin…
Read More »
AWS, Snowpipe, Snowflake
Moving data from S3 to Snowflake to satisfy use cases around analysis, corporate reporting, or cross-domain information collaboration is best achieved through …
Read More »
Data Glossary, Dictionary, Catalogue
The Data Glossary focuses on business terminology and definitions, bridging the gap between business and IT. A data glossary is a document that defines and des…
Read More »
AWS Sagemaker vs Databricks
A comparison of AWS Sage Maker and Databricks. Both satisify different use cases. A key aspect is the principle of ‘cloud native’, meaning that if …
Read More »