Migrating On prem CRM, SFA to Azure

Azure deployment, CRM and a SaaS front end application for end users. Migration of data to Azure and the Azure Data Lake.

Necessary Data Migrations

1-On Premises systems to Dynamics/CRM and SaaS front end application

• Historical, Operational data

 

2-HubSpot to Dynamics: Lead management data stored in HubSpot will be migrated.

 

Migration to Azure and Azure Data Lake

Data Extraction

• Extract data from the source systems in a suitable format. This could involve:
  • Database queries
  • API calls
  • Data exports
• For documents, extract them from the current storage solution (e.g., Pandora).

Data Cleansing and Transformation

1. • Cleanse and transform the extracted data to ensure it meets the requirements of the target systems (Dynamics, SaaS, Azure Data Lake).
   • This may involve:
     • Data standardization
     • Data validation
     • Data mapping (source to target fields)
     • Data transformation (e.g., format conversions, calculations)
   • Azure Data Factory or Azure Data Lake Analytics for scalable data transformation.

Data Loading

1. • Load the transformed data into the target systems:
     • Dynamics: Use Dynamics APIs or data import tools to load member data, lead management data, and other relevant information.
     • INSTANDA: Use the SaaS APIs or data import mechanisms to load membership product data, rating data, and other relevant information.
     • Azure Data Lake:
       • Store the data in the appropriate format (e.g., Parquet, Delta) within the Azure Data Lake.
       • Organize the data into a logical folder structure for efficient access and querying.
       • Azure Data Factory to orchestrate the data loading process.

Document Migration

1. • Migrate the extracted documents to the new document storage solution (Azure Blob Storage).
   • This may involve:
     • Uploading documents to Azure Blob Storage
     • Metadata tagging for efficient retrieval
     • Integrating document storage with Dynamics and the SaaS

Historical Data Migration

1. • The documents mention that the level of granularity migrated for older business will be reduced.
   • This implies a need to define a strategy for handling historical data, which may involve:
     • Data aggregation or summarization
     • Data archiving
     • Defining retention policies

Key Considerations for Azure Migration

• Azure Services: Leverage Azure services for efficient and scalable data migration:
  • Azure Data Factory: For data orchestration, transformation, and loading.
  • Azure Blob Storage: For storing documents and unstructured data.
  • Azure Data Lake Storage: For storing large volumes of data in various formats.
  • Azure SQL Database: For storing structured data.
• Data Security: Implement appropriate security measures to protect data during migration:
  • Encryption in transit and at rest
  • Access control and authorization
  • Data masking or anonymization (if necessary)
• Migration Approach: Choose an appropriate migration approach:
  • Big bang: Migrate all data at once (higher risk, shorter downtime).
  • Phased migration: Migrate data in stages (lower risk, longer duration).
• Testing and Validation: Thoroughly test and validate the migrated data to ensure accuracy and completeness:
  • Data reconciliation
  • Data quality checks
  • User acceptance testing
• Cutover Planning: Plan the cutover to the new systems carefully to minimize disruption to business operations:
  • Downtime planning
  • Rollback strategy