Data Architect

Photon - Louisville, KY

Full-time - Senior

Louisville, KY

Professional, Scientific, and Technical Services

About the position

We are seeking an experienced Data Architect with hands-on expertise in Azure Databricks, Azure Data Lakehouse, and Medallion Architecture. The ideal candidate will have strong implementation experience, including the integration of Delta Lake tables, and a background in working with Customer Data Platforms (CDP). This role requires a deep understanding of cloud-native data solutions and the ability to architect and deliver scalable, high-performing data platforms that can support advanced analytics, machine learning, and business intelligence solutions.

Responsibilities

• Lead the design and development of scalable, cloud-based data architectures leveraging Azure Data Lakehouse and Medallion Architecture principles.
• Architect and implement data pipelines and ETL processes using Azure Databricks, ensuring seamless integration with Delta Lake to enable ACID transactions, time travel, and optimized data storage.
• Implement the Medallion Architecture pattern, building Bronze, Silver, and Gold layers to ensure efficient data processing, aggregation, and enrichment for analytics and reporting.
• Design and manage the data architecture to effectively handle customer data, ensuring the Customer Data Platform (CDP) is integrated with the Azure data ecosystem, enabling personalized and customer-centric analytics.
• Lead the implementation and optimization of Azure Data Lake Storage (ADLS) to support structured, semi-structured, and unstructured data storage, ensuring efficient querying and data retrieval.
• Define and implement robust data governance, security, and compliance practices using tools like Azure Data Catalog, Azure Purview, and Azure Key Vault.
• Optimize data pipelines for performance, scalability, and cost-efficiency, ensuring that data is processed efficiently and meets SLAs for downstream analytics and reporting systems.
• Work closely with data engineers, business analysts, and data scientists to define data requirements, ensuring that data pipelines are designed to meet business needs.
• Architect real-time data ingestion pipelines, integrating with various data sources (APIs, event hubs, etc.), and streamlining real-time analytics using Azure Databricks and Delta Lake.
• Establish best practices for data architecture, provide documentation for data pipelines and architecture, and ensure knowledge sharing across the organization.

Requirements

• 5+ years of experience in data architecture and engineering, with expertise in Azure Databricks, Azure Data Lakehouse, and Delta Lake.
• Hands-on experience implementing Medallion Architecture, building and managing Bronze, Silver, and Gold data layers.
• Strong experience with Azure Data Lake Storage (ADLS) and integration with Delta Lake for efficient data storage and querying.
• Customer Data Platform (CDP) experience, with the ability to design and integrate customer-centric data architectures that drive personalized analytics.
• Experience in data governance, security, and compliance using Azure tools like Azure Purview, Data Catalog, and Key Vault.
• Proven experience in ETL/ELT development and real-time data ingestion pipelines, working with large-scale datasets.
• Strong knowledge of SQL, Python, and Spark for data processing, analysis, and pipeline development.
• Familiarity with data modeling techniques, big data technologies, and data warehousing in a cloud environment.
• Excellent communication and collaboration skills, with the ability to work cross-functionally and lead data-driven initiatives.

Nice-to-haves

• Experience with machine learning and advanced analytics workflows using Azure Databricks.
• Familiarity with API integration and real-time data streaming solutions like Azure Event Hubs or Azure Stream Analytics.
• Knowledge of DevOps for data, including automated CI/CD pipelines for data deployments.