Data Engineer

$124,800 - $135,200/Yr

The Judge Group - Philadelphia, PA

Full-time - Mid Level

Remote - Philadelphia, PA

Administrative and Support Services

About the position

As a Senior Data Engineer, you will be an integral part of our software development team, focusing on the creation and implementation of strategies and solutions for ingesting, storing, and distributing big data. This role requires a deep understanding of Big Data technologies, including but not limited to Hadoop, PySpark, Hive, JSON, and SQL. You will be responsible for developing products, tools, and software features that leverage these technologies to meet the needs of our clients and enhance our data capabilities. In this position, you will innovate and drive the development of data strategies and roadmaps that align with our technology organization’s project portfolio and business strategy. You will also be tasked with establishing enterprise standards for data domains and solutions, ensuring simplified integration and streamlined operational and analytical uses. Your role will involve leveraging innovative technologies to renovate and transform existing core data assets, which include SQL-based, NoSQL-based, and Cloud-based data platforms. You will define high-level migration plans to bridge the gaps between current and future states, typically in alignment with budgeting and capital planning processes. Additionally, you will lead the analysis of the technology environment to identify critical deficiencies and recommend solutions for improvement. Mentoring team members in data principles, patterns, processes, and practices will also be a key aspect of your responsibilities. You will promote the reuse of data assets and manage the data catalog for reference, ensuring that our data resources are utilized effectively. Your role will also involve drafting and reviewing architectural diagrams, interface specifications, and other design documents. You will proactively lead activities that create deliverables guiding the direction, development, and delivery of technological responses to targeted business outcomes. This includes providing facilitation, analysis, and design tasks necessary for developing an enterprise's data and information architecture, with a focus on treating data as a valuable asset for the organization. You will develop target-state guidance, including reusable standards, design patterns, and guidelines, to evolve the technical infrastructure related to data and information across the enterprise.

Responsibilities

• Innovate, develop, and drive the development and communication of data strategy and roadmaps across the technology organization to support project portfolio and business strategy.
• Drive the development and communication of enterprise standards for data domains and data solutions, focusing on simplified integration and streamlined operational and analytical uses.
• Drive digital innovation by leveraging innovative new technologies and approaches to renovate, extend, and transform the existing core data assets, including SQL-based, NoSQL-based, and Cloud-based data platforms.
• Define high-level migration plans to address the gaps between the current and future state, typically in sync with the budgeting or other capital planning processes.
• Lead the analysis of the technology environment to detect critical deficiencies and recommend solutions for improvement.
• Mentor team members in data principles, patterns, processes and practices.
• Promote the reuse of data assets, including the management of the data catalog for reference.
• Draft and review architectural diagrams, interface specifications and other design documents.
• Proactively and holistically lead activities that create deliverables to guide the direction, development, and delivery of technological responses to targeted business outcomes.
• Provide facilitation, analysis, and design tasks required for the development of an enterprise's data and information architecture, focusing on data as an asset for the enterprise.
• Develop target-state guidance (i.e., reusable standards, design patterns, guidelines, individual parts and configurations) to evolve the technical infrastructure related to data and information across the enterprise.

Requirements

• 5+ years of experience as a Data Engineer, with a focus on PySpark and big data technologies.
• Strong proficiency in Python and experience with PySpark for building and optimizing data pipelines.
• Proficiency in SQL for querying and manipulating large datasets.
• Experience with cloud platforms like AWS, Azure, or GCP, including services such as EMR, Databricks, or BigQuery.
• Strong problem-solving skills and the ability to work with complex datasets.
• Excellent communication skills and the ability to collaborate effectively with cross-functional teams.