Generative AI has given rise to Agentic AI. While ChatGPT is primarily a chatbot that can generate text responses, AI agents can execute complex tasks autonomously, e.g., make a sale, plan a trip, make a flight booking, book a contractor to do a house job, order a meal. Agentic AI is another area of future AI growth.
Agentic AI can be applied to two core data management processes: data cataloging and data engineering (warehousing) — outlining the task-specific AI agents relevant for both scenarios. There is a reference architecture of an agentic AI platform where there is an orchestration of agents (for data management) in a self-sustaining fashion in the face of changing business and data landscapes. Tasks include:
- automating data pipelines (ingestion, modeling, transformation),
- operationalizing governance & compliance with AI-driven policy enforcement;
- enabling insights & predictions for real-time business decision-making
- Supervisor agent: scans enterprise source systems for new and relevant data — assigning and scheduling tasks to agents.
- Data discovery agent: performs autonomous extraction of entities to detect relationships and apply metadata enrichment.
- Data integration agent: provides seamless integration with ERP, CRM, etc. enterprise systems enabling real-time catalog updates.
- Metadata validation agent: performs metadata consistency checks, detecting duplicates, ensuring relationship mapping accuracy.
- Data observability agent: continuously tracks data lineage, applies security and access control policies, and ensures compliance
For Data Engineering we can deploy the same process;
- Supervisor agent: schedules batch & real-time jobs, automating ingestion from batch and streaming sources.
- ETL agents provide end-to-end automation of data pipelines, comprising data ingestion. modeling, and transformation.
- Data quality agent: performs data quality, integrity and consistency checks, deduplicates records, etc.
- Data modeling and tuning agent: dynamically adjusts schemas & indexing based on schema drift detection and user query trends — automatically adapting table structures.
- Data observability agent: continuously monitors data warehouse performance, auto-tuning data pipelines for speed & cost efficiency.
