Role Overview We are partnering with an established investment firm undertaking a core data migration from eFront Invest FIA into an Azure Databricks Lakehouse. We are seeking a hands-on Senior Data QA / Data Quality Engineer to lead automated migration testing, source-to-target reconciliation, and pipeline validation. You will be responsible for ensuring that all data flowing from eFront lands in Databricks with 100% mathematical and schema integrity. Key Responsibilities Source-to-Target Reconciliation: Design, build, and execute automated validation queries to compare legacy eFront extracts against target Databricks Delta Lake tables (Bronze, Silver, Gold). Business Logic and Metrics Testing: Verify complex private equity calculation logic (cash flows, commitments, valuations, NAV, and return metrics) across data pipelines. Automated Data Quality Frameworks: Develop automated test suites using Python (PyTest), PySpark, and advanced SQL to catch discrepancies, nulls, and duplicate records. CI/CD Quality Gates: Integrate automated testing and data validation checks into Azure DevOps (ADO) deployment pipelines. Schema Drift and Anomaly Detection: Monitor and resolve schema evolution, edge-case discrepancies, and late-arriving financial records. Required Experience and Skills Experience testing data extracted from or integrated with eFront (eFront Invest / FIA / FrontScript). Proven track record as a Data QA Engineer, Data Quality Engineer, or Data Test Automation
..... full job details .....