Networking Deployment Engineer
Are you a software engineer with a disposition toward networked systems design or networking engineer with a disposition toward systems programming and software development? We are seeking engineers to join our networking team with a focus on modeling and verification of Oxide deployments, developing testbed technologies to emulate both Oxide systems and broader deployment environments, and work with our customer support team to prepare for new types of deployment environments.
As a cloud infrastructure company, our success is defined by our customers' success in deploying Oxide Cloud Computers that host critical business functions. There are many different types of environments our customers deploy Oxide systems into today and the heterogeneity and scale is growing rapidly. We take being prepared for any and all environments that Oxide Cloud Computers will deploy to very seriously. This role is all about that: being prepared. We view validation and verification of cloud infrastructure as an engineering discipline in its own right and are looking for engineers who share this viewpoint.
We are taking a very ambitious approach with respect to networking: not only have we developed our own compute sled and cabled backplane, we have also developed our own switch, and then the entire stack of software from the lowest levels of firmware to end-user experience — and everything in between! To understand why we’ve taken this steeper path (and the advantages that it affords), listen to the team in its own voice in our Oxide and Friends discussion on rack-scale networking.
As a networking deployment engineer, you will:
Collaborate with other engaged, friendly systems engineers to understand customer use cases and ensure the Oxide platform is ready for a wide variety of deployment environments.
Develop tools to emulate and analyze complex networks, including those deployed in production at customer sites. Logging, tracing, and metrics are critical pieces of distributed systems, and you’ll get the chance to dig into them all.
Write code in Rust, P4 and more. The code you write will be open source (e.g., Falcon, a Rust-based framework for modeling and deploying networked systems with emulated Oxide racks), and across many different codebases.
Operate across multiple layers of the stack to model and verify how true rack-scales system interact their deployment environment, including our integrated switch, a custom compute sled, and all software connecting and orchestrating them.
You will thrive in this role if you:
Have experience deploying and operating large-scale mission-critical network and compute infrastructure.
Have previously worked with Rust or another low-level systems language such as C.
Have implemented networking protocols in environments that were constrained by performance and correctness.
Have experience designing, implementing, deploying and operating data center networks using dynamic routing protocols and overlay networks.
Have experience with BGP implementation or deployment.
Enjoy the idea of owning the inter-rack networking end-to-end (host OS + NIC + switch + switch OS) – and also of integrating that networking with a customer’s larger network.
Are energized by the thought of jumping between implementing a new protocol (or a new implementation of an old one!), analyzing customer networking pathologies, and bringing kernel drivers to life.
Produce good documentation and/or written descriptions of your ideas.
Believe that the closed nature of low-level networking is a barrier to progress and want to work with a team breaking down that barrier.
Before applying for this role, you should:
Browse our public Requests for Discussion to get a flavor for how we work.
Listen to Hiring Processes with Gergely Orosz to familiarize yourself with the Oxide hiring process.
Watch Building a Rack Scale Computer with P4 at the Core and DTrace + P4.
Check out OPTE — and watch Kyle Simpson’s dtrace.conf(24) talk
-
Listen to some of our network-centric episodes of Oxide and Friends. A few recommendations:
Rack-scale Networking to understand our approach to networking and some of the details of our implementation
The Network Behind the Network on the specific challenges of our service processor network
Hell is other networks to learn about our disposition to customer problems
Transparency in Hardware/Software Interfaces to hear our perspective on what software needs from networking silicon
Performance Engineer
Are you an engineer who loves understanding and improving the performance of systems? We are seeking versatile engineers to work on the performance of Oxide systems wherever suboptimality is to be found: from the depths of hardware, through the operating system kernel and hypervisor and into the guest operating system — and to the highest levels of the application stack.
As an engineer focused on systems performance, you will:
Work rigorously to understand existing limiters to performance, wherever those limiters may lie.
Prototype improvements to the system, be they small fixes, larger scale re-writes, or entirely de novo subsystems.
Work on systems of a variety of ages, spanning from decades-old to entirely new — and everything in between.
Work on systems primarily written in Rust and C.
Work with a wide variety of our systems software, including (but not limited to!) our host operating system (Helios), our hypervisor (Propolis), our block storage service (Crucible), our embedded operating system (Hubris), and our control plane (Omicron).
Work with a variety of hardware as needed to understand and model the performance ramifications of different architectural or component decisions.
Develop infrastructure and tooling to better understand systems performance.
You will thrive in this role if you:
Believe that every instruction is sacred, every instruction is great.
Love to hunt slow, broken code — and replace it with a vastly improved alternative.
Are deeply analytical and data-intensive.
Have used whatever tooling at your disposal to understand systems behavior (e.g., DTrace/eBPF, snoop/tcpdump, truss/strace).
Have implemented your own tools where the right tool didn’t exist (or otherwise needed to be extended).
Have experience shipping software written in Rust, C, or another systems-oriented language.
Before applying for this role, you should:
Browse our public Requests for Discussion to get a flavor for how we work
Listen to Hiring Processes with Gergely Orosz to familiarize yourself with the Oxide hiring process.
-
Listen to some of our episodes of Oxide and Friends. A few recommendations:
When Async Attacks! on a particularly pathological performance problem and the tooling we developed to understand it
Mr. Nagle’s Wild Ride on a timeless performance tale, re-told anew
Crucible: The Oxide Storage Service on our storage service and our approach to improving its performance
Heterogeneous Computing with Raja Koduri on how hardware comprises the ultimate limiter of performance — and why different approaches are called for by different problems
Operating System Networking Engineer
Do you love both networking and operating systems implementation, digging into the gritty interface where software meets the wire? We are seeking a software engineer to work on our illumos-based host operating system, Helios with a specific focus on its networking aspects. We are taking a very ambitious approach with respect to networking: not only have we developed our own compute sled and cabled backplane, we have also developed our own switch, and then the entire stack of software from the lowest levels of firmware to end-user experience — and everything in between! To understand why we’ve taken this steeper path (and the advantages that it affords), listen to the team in its own voice in our Oxide and Friends discussion on rack-scale networking. We are seeking an operating system software engineer who is energized by this end-to-end approach, and excited about joining our team to develop new kinds of functionality only made possible by true hardware/software co-design.
As an engineer working on operating system networking, you will:
Work with a helpful and experienced team to design, build, and bring up host system software, with a particular focus on its networking aspects
Enhance/write drivers several different kinds of networking devices: merchant silicon-based NICs, switching silicon, FPGA-based NICs, etc.
Read and write code in C (for the operating system writ large), in Rust (for the Oxide Packet Transformation Engine and perhaps Propolis), and potentially some P4. All the code you write will be open source.
Help to improve the performance of the lowest layers of the networking stack, with much in scope: improved use of hardware offloads, software rearchitecture, implementation improvements, etc.
You will thrive in this role if you:
Are familiar with operating systems networking concepts and are comfortable in C and/or Rust
Have dealt with networks at the hardware layer — and operating systems at the network layer
Are excited by the idea of writing operating system kernel code that enables networking hardware features
Love to make networks perfom at their line rate, knowing that it requires rigorous analysis, potentially novel architecture, and careful implementation
Have previously worked in the guts of operating systems
Appreciate the hard-won thrill of debugging a knotty problem to root cause
Believe in fully documenting your ideas for both current colleagues and your own future self
Enjoy reading excellent documentation produced by others
Make the tools you wish you had
Communicate effectively with engineers across a range of disciplines to exchange knowledge and expertise
Before applying for this role, you should:
Browse our public Requests for Discussion to get a flavor for how we work
Listen to Hiring Processes with Gergely Orosz to familiarize yourself with the Oxide hiring process.
Prowl around the code in Helios
Read IPD 45 Flow trees in the MAC datapath to get a flavor for the kind of work this role entails
-
Listen to some of our episodes of Oxide and Friends. A few recommendations:
Helios on the genesis of our illumos derivative as discussed in RFD 26
Rack-scale Networking to understand our approach to networking and some of the
Holistic Engineering on our integrated hardware/software approach
Hell is other networks to learn about our disposition to customer problems
Transparency in Hardware/Software Interfaces to hear our perspective on what software needs from networking silicon
Networking Software Engineer
Are you a software engineer with a disposition towards networking software who loves working up and down the OSI stack? We are seeking a software engineer to join our networking team. We are taking a very ambitious approach with respect to networking: not only have we developed our own compute sled and cabled backplane, we have also developed our own switch, and then the entire stack of software from the lowest levels of firmware to end-user experience — and everything in between! To understand why we’ve taken this steeper path (and the advantages that it affords), listen to the team in its own voice in our Oxide and Friends discussion on rack-scale networking. We are seeking a networking-focused software engineer who is energized by this approach, and excited about joining our team to develop new kinds of functionality only made possible by true hardware/software co-design.
As a software engineer focused on networking, you will:
Collaborate with other engaged, friendly systems engineers to understand customer use cases and implement the core of the Oxide platform.
Operate across multiple layers of the stack to build a true rack-scale system, including our integrated switch, a custom compute sled, and all software connecting and orchestrating them.
Develop tools to simulate and analyze complex networks, including those deployed in production at customer sites. Logging, tracing, and metrics are critical pieces of distributed systems, and you’ll get the chance to dig into them all.
Write code in Rust, C, and some P4. The code you write will be open source (e.g., the Oxide Packet Transformation Engine, an in-kernel Rust module for packet processing), and across many different codebases.
You will thrive in this role if you:
Have previously worked with Rust or another low-level systems language such as C.
Have a deep understanding of load balancing.
Have experience operating and developing IPsec-based VPN technologies.
Have implemented networking protocols in environments that were constrained by performance and correctness.
Have experience designing, implementing, deploying and operating data center networks using dynamic routing protocols and overlay networks.
Have experience with BGP implementation or deployment.
Enjoy the idea of owning the inter-rack networking end-to-end (host OS + NIC + switch + switch OS) – and also of integrating that networking with a customer’s larger network.
Are energized by the thought of jumping between implementing a new protocol (or a new implementation of an old one!), analyzing customer networking pathologies, and bringing kernel drivers to life.
Produce good documentation and/or written descriptions of your ideas.
Believe that the closed nature of low-level networking is a barrier to progress and want to work with a team breaking down that barrier.
Before applying for this role, you should:
Browse our public Requests for Discussion to get a flavor for how we work.
Listen to Hiring Processes with Gergely Orosz to familiarize yourself with the Oxide hiring process.
Watch Building a Rack Scale Computer with P4 at the Core and DTrace + P4.
-
Listen to some of our network-centric episodes of Oxide and Friends. A few recommendations:
Rack-scale Networking to understand our approach to networking and some of the details of our implementation
The Network Behind the Network on the specific challenges of our service processor network
Hell is other networks to learn about our disposition to customer problems
Transparency in Hardware/Software Interfaces to hear our perspective on what software needs from networking silicon
Control Plane Engineer
Are you an experienced Rust programmer who loves building, deploying, and debugging distributed systems? We are seeking software engineers to work on the Oxide control plane, the spinal column of our software that is responsible for managing and provisioning virtual machines, storage and networking.
As a software engineer working on the control plane, you will:
Work on a robust distributed system, Omicron, that ships across an air gap to Oxide customers running racks in their own datacenters.
Write code mostly in Rust, read code primarily written in Rust with some C. The code you write will be open source, be it in Omicron itself or the many components with which it interacts, e.g. our block storage service (Crucible), our hypervisor (Propolis), our host operating system (Helios), our embedded operating system (Hubris), or another of our many software components.
Collaborate with other engaged, friendly, systems-oriented engineers to understand customer use cases and implement the core of the Oxide platform.
Work to understand and improve the performance of the services offered by the Oxide rack.
Develop tools to test and analyze complex systems, including those deployed in production at customer sites. Logging, tracing, and metrics are critical pieces of distributed systems, and you’ll get the chance to dig into them all.
You will thrive in this role if you:
Have experience shipping software written in Rust
Have experience working on distributed systems
Are energized by the thought of diving deep on neglected (but essential!) distributed systems topics like cluster attestation or live system software update
Believe in fully documenting your ideas for both current colleagues and your own future self
Enjoy reading excellent documentation produced by others
Make the tools you wish you had
Before applying for this role, you should:
Browse our public Requests for Discussion to get a flavor for how we work
Listen to Hiring Processes with Gergely Orosz to familiarize yourself with the Oxide hiring process.
Watch Update on Update to understand the challenge of systems software in the large.
Prowl around the code in Omicron
-
Listen to some of our episodes of Oxide and Friends. A few recommendations:
The Saga of Sagas on how we implement workflow management in the control plane
A Crate is Born on how we think about software modularity and reuse
The Frontend of the Computer on our API-driven approach in the control plane
Get You a State Machine for Great Good on how we use property-based testing
Querying Metrics with OxQL on why we developed our own query language for the control plane
Debugger-Driven Development on how we have developed our Omicron debugger (omdb) in parallel with Omicron itself
The full posting opens here — pay, setting and the full description, without leaving the list.