About the position
As a Senior Data Engineer, you will be responsible for building data pipelines that assemble large, complex sets of data to meet both non-functional and functional business requirements. You will develop ETL solutions using various technologies such as Python, Powershell, SQL, and SSIS to load and automate complex datasets in formats including PDF, Excel, flat files, JSON, XML, and EDI. You will take full ownership of end-to-end data processes on Azure Cloud environments and work closely with data architects, SMEs, and other technology partners to develop and execute data architecture and product roadmaps. Additionally, you will collaborate with backend developers to understand legacy applications and implement features in a new system. Your role will also involve troubleshooting issues and operational bottlenecks to support continuous data delivery for various applications, making improvements, and addressing technical debt. You will implement complex warehouse views, make database design decisions to support UI needs, and optimize scripts for refreshing large volume datasets. Furthermore, you will perform code reviews, coach team members, develop reports on data dictionaries and server metadata, and implement reporting tools as needed. Keeping current on big data and data visualization technology trends, you will evaluate and work on proof-of concepts and make recommendations on cloud technologies.
Responsibilities
- Build data pipelines to assemble large, complex sets of data that meet non-functional and functional business requirements
- Develop ETL solutions using Python, Powershell, SQL, SSIS etc. to load and automate complex datasets in PDF, Excel, flat files, JSON, XML, EDI etc.
- Take full ownership of end-to-end data processes on Azure Cloud environments
- Work closely with data architect, SMEs and other technology partners to develop & execute data architecture and product roadmap
- Collaborate with backend developers to understand legacy applications and implement features in a new system
- Troubleshoot issues and operational bottlenecks to support continuous data delivery for various applications
- Take initiatives to make changes and improvements, work on technical debt, new and complex challenges
- Implement complex warehouse views, make database design decisions to support UI needs, optimize scripts to periodically refresh large volume datasets
- Perform code reviews and coach team members
- Develop reports on data dictionary, server metadata, data files and implement reporting tools as needed
- Implement best practices for data updates and development, troubleshoot performance and other data related issues on multiple product applications
- Keep current on big data and data visualization technology trends, evaluate, work on proof-of-concept and make recommendations on cloud technologies.
Requirements
- 7+ years of data engineering experience working in partnership with large data sets and cloud architecture
- Deep experience in building data pipelines using ETL tools and loading data to and from RDBMS such as Postgres, SQL Server, Oracle or similar
- Proficient in cloud services technologies such as Microsoft Fabric, Azure Data Factory, Data Lake, and other related technologies
- Proficient in using SSDT tools for building SQL server relational databases, databases in Azure SQL, Analysis Services data models, Integration Services packages and Reporting Services reports
- Solid experience building data solutions with programming languages such as Python, Powershell
- Advanced T-SQL and ETL automation experience
- Experience working with orchestration tools such as Airflow and building complex dependency workflows
- Self-motivated with the ability to work and learn new technology independently
- Great problem-solving capabilities, troubleshooting data issues and experience in stabilizing big data systems
- Excellent communication and presentation skills
Nice-to-haves
- Hands-on deep experience with cloud data migration, and experience working with analytic platforms like Fabric, Databricks on the cloud
- Certification in one of the cloud platforms (AWS/GCP/Azure)
- Experience with real-time data streaming tools like Kafka, Kinesis or any similar tools
- Experience with the US health care reimbursement-related terminology and data is a plus
