About the position
A production engineer is a developer who thinks deeply about systems and how they behave in the wild. Whether it be networking, or the Linux kernel, or even a specific interest in scaling, algorithms, or distributed systems. You are a systems engineer who aims to code themselves out of a job by automating all the things, learning to leverage great development practices like Test-Driven-Development or continuous integration (to start with).
Responsibilities
- Radiate knowledge about the service's infrastructure and reliability to the rest of the development team.
- Identify parts of the system that do not scale, provide immediate palliative measures and drive long term resolution of these incidents.
- Plan the growth of Benzinga's infrastructure.
- Document every action so your learnings turn into repeatable actions and then into automation.
- Improve the deployment process to make it as boring as possible.
- Manage our infrastructure with Kubernetes and Terraform.
- Proactively identify and reduce security risks.
- Develop security training and guidance to internal development teams.
- Ability to discover and patch SQLi, XSS, CSRF, SSRF, authentication and authorization flaws, and other web-based security vulnerabilities (OWASP Top 10 and beyond).
- Knowledge of common authentication technologies including JWT, OAuth, SAML, CAs, OTP/TOTP.
- Design, build and maintain core infrastructure pieces that allow Benzinga to scale to support hundreds of thousands of concurrent users.
- Be on an on-call rotation to respond to benzinga.com availability incidents and provide support for service engineers with customer incidents.
- Debug production issues across services and levels of the stack.
- Make monitoring and alerting alert on symptoms and not on outages.
- Manage day-to-day maintenance and evolution of Benzinga's Prometheus monitoring and alerting infrastructure.
- Bundle Prometheus monitoring as an out-of-the-box monitoring solution for Benzinga products.
- Configure dashboards and alerts using Grafana.
- Build and maintain the benzinga.com public monitoring gateway.
- Improve coverage of Benzinga performance monitoring.
- Create automated alerts to notify team members of regressions.
Requirements
- You can reason about software, algorithms, and performance from a high level.
- You have experience thinking about systems - edge cases, failure modes, behaviors, and specific implementations.
- You have worked with distributed systems and have a solid understanding of how modern web stacks are built, and why.
- You know your way around Linux and the Unix Shell.
- Strong communication skills.
- Experience with Docker, Nginx, Go, Kubernetes, Kong, Prometheus.
- Experience with managing large amounts of telemetry.
- Oddly enjoys updating services at 1am on a Sunday, hanging out with other devs while everyone else is at the bar sleeping.
- First to jump in when fighting fires.
- Prepares us to fight fires as little as possible.
- Self-motivated with strong organizational skills.
