Senior Data Engineer (Azure)

Sial Technology Partners - Nashville, TN

Full-time - Senior

Nashville, TN

11-50 employees

Professional, Scientific, and Technical Services

About the position

The Senior Data Engineer will play a crucial role in building and operating the firm's data warehouse, data marts, transaction hubs, and operational data store. This position focuses on modernizing the current ecosystem by migrating legacy systems to Azure, specifically working on a green-field project that will define the future state of the firm's data needs. The role involves designing and developing data ingestion frameworks, ensuring data integrity, and implementing cloud-based data services.

Responsibilities

• Design and develop data ingestion pipeline framework using Python running in Kubernetes
• Design and develop event-driven ingestion and regression framework to consume and publish data
• Design and develop Airflow DAGs for scheduling the ingestion and extract pipelines
• Build container images using Docker and deploy to Kubernetes, particularly in Azure Kubernetes Service
• Maintain Linux systems for containers built from Docker images
• Design and develop streaming ingestion solutions to process live data from various vendors
• Design and develop Continuous Integration and Continuous Deployment strategies
• Set up and deploy cloud-based data services such as blob services, databases, and analytics
• Secure the platform and the stored data, ensuring only necessary users can access the data
• Ensure business continuity using techniques for high availability and disaster recovery
• Monitor systems for proper operation and cost-effectiveness
• Define the data to be extracted and the transformation operations required
• Test ETL/ELT jobs in development or test environments before migrating to production
• Set up proactive and reactive monitoring systems for ETL operations
• Document and articulate technology choices and approaches to the team

Requirements

• 12+ years of experience in coding and building software
• 8+ years of experience in programming using SQL Server, Oracle, or Sybase
• 5+ years of experience in using Python
• 3+ years of in-depth working knowledge and hands-on experience in Airflow
• 2+ years of experience with CI/CD processes using Azure DevOps
• Proficient with Azure concepts like Resource group, Key Vault, Blob Storage, Table Storage, and Data Lakes
• Exposure to container technologies such as Kubernetes and Docker
• Exposure to Azure Data Factory and Data Bricks
• Bachelor's degree in computer science or related fields; Master's degree preferred

Nice-to-haves

• Finance knowledge
• Strong communication skills (oral and written)
• Strong analytical skills to map data elements and develop transformation logic
• Excellent interpersonal skills to interface with multiple groups
• Proven track record to own the entire project lifecycle
• Excellent documentation skills