Senior Big Data Engineer - Python/PySpark

Nr Consulting - Hanover, NJ

Full-time - Senior

Hanover, NJ

Administrative and Support Services

About the position

The Senior Big Data Engineer - Python/PySpark role involves building and optimizing PySpark applications for batch and streaming data processing. The position requires extensive experience with Hadoop and NoSQL databases, as well as strong backend development skills in Python. The engineer will participate in the software development lifecycle, collaborating with cross-functional teams to deliver high-quality solutions and address complex problems, particularly in a financial context.

Responsibilities

• Build PySpark based applications for batch and streaming requirements.
• Optimize performance of Spark applications in Hadoop using configurations around Spark Context, Spark-SQL, Data Frame, and Pair RDD's.
• Develop, test, and maintain high-quality software using Python.
• Participate in the entire software development lifecycle, from building to delivering solutions.
• Collaborate with cross-functional teams to identify and solve complex problems.
• Design and implement web applications using Python frameworks like Streamlit.
• Create visualizations and user-friendly web interfaces for complex data processing tasks.
• Integrate APIs and databases with Streamlit applications.
• Work with large datasets and incomplete information as required.
• Engage with users and stakeholders to understand customer needs and develop solutions independently.
• Rewrite and decompose legacy controls using Python frameworks.

Requirements

• 8+ years of experience in Python, PySpark, and Hadoop Big Data.
• Strong backend development experience in Python.
• In-depth knowledge of Hadoop and NoSQL databases.
• Experience with Python libraries such as Pandas, Polars, NumPy, Matplotlib, Plotly, Seaborn, and Altair.
• Conceptually strong in Python and analytics.
• Experience in developing web applications using Python frameworks like Streamlit.
• Familiarity with financial concepts, especially related to Trading hubs.

Nice-to-haves

• Experience working in a financial organization.
• Awareness of financial concepts related to Trading hubs.