Data Engineer

Public Partnerships - Harrisburg, PA

Full-time

Harrisburg, PA

Securities, Commodity Contracts, and Other Financial Investments and Related Activities

About the position

The Data Engineer at Public Partnerships LLC will be responsible for designing, building, and managing data pipelines primarily using Azure Data Factory and Databricks. This role focuses on both batch and near real-time data ingestion, ensuring data quality, security, and optimization of storage and costs. The Data Engineer will work with various data formats and sources, implementing automation frameworks and monitoring pipeline performance to meet service level agreements (SLAs).

Responsibilities

• Design, build, and manage Azure Data Factory pipelines for data ingestion.
• Handle both batch and near real-time data ingestion using Azure Service Bus.
• Integrate structured, semi-structured, and unstructured data sources.
• Utilize formats such as Parquet, Avro, JSON, etc., and manage loading into the Bronze layer.
• Build and manage data engineering pipelines using Databricks (PySpark, SparkSQL).
• Implement Spark-based data transformation jobs from Bronze to Silver and Gold layers.
• Establish and manage Data Quality Framework using Databricks and Data Factory.
• Set up data security using RBAC and ACLs to manage permissions on sensitive datasets.
• Implement automation frameworks for pipeline execution using Databricks Workflows.
• Monitor pipelines using Azure Monitor, set up alerts and ensure SLAs are met.
• Optimize storage in ADLS Gen2 and use Azure Key Vault to secure credentials.
• Optimize costs using Azure Cost Management.

Requirements

• Experience with Azure Data Factory for ETL/ELT operations.
• Proficiency in Databricks (PySpark, SparkSQL) for data transformations.
• Knowledge of Azure Data Lake Storage (ADLS Gen2).
• Experience with SQL Server, PostgreSQL, Cosmos DB, CRM, and ERP systems.
• Understanding of Data Governance, RBAC, and ACLs for managing permissions.
• Familiarity with CI/CD tools: Azure DevOps, GitHub Actions, Terraform.
• Experience with Azure Monitoring, Alerting, and Logging tools.