How we learnt a new language while finishing a rewrite in under 2 months

by Nisheeth Barthwal

Prelude

We, the team at Route Optimization, are primarily responsible for ensuring that the incoming requests are matched with our MOIA fleet to the best of our customers’ satisfactions. That is easier said than done, for customer satisfaction comes at a price. And for us it became evident this year how much we would be required to pay.

“The price one pays for maintaining their Software is transacted from one’s own sanity”

In our case, our Object allocations were just a tad bit too much, arising from our combinatorial problem space of assigning customer requests to vehicles in a slightly-more-sophisticated-than-naive approach.

Previously we were running our algorithm on <Insert-Inadequate-Stack-Here>, where plenitude of Object allocations provided a throughput that fell short of anything noteworthy to put on a Medium article. And with all things Software, it was about to get worse as we started theorycrafting MOAR features!™

As we started self reflecting on the potential performance hits the upcoming months would barrage upon our component, the need became evident to have a better tech stack to capitalize on our home grown reservoir of “Really Smart Optimization Ideas We’d Like To Try”. Once the team was done evaluating different stacks in an all too pubescent approach of creating a Pros & Cons list, we had a decision.

“The grass isn’t always greener on the other side. Sometimes there’s just no grass. Sometimes there’s only but Rust.”

The Pursuit of ̷H̷a̷p̷p̷y̷n̷e̷s̷s̷ Rewrite

The rationale was sound. We needed MOAR juice for MOAR features!™, a 1:1 match. The Rust language could handle our all too many Object allocations, continue to be performant enough for writing new algorithms and yet modern enough to not leave the developers with the familiar aftertaste of an *cough* Indonesian Coffee *cough* as they returned home after committing their code.

And thus it began. Once the loom of “Yet another rewrite” finally subsided, the team started drafting a Design Document. Therein were listed all the salient details regarding the Whys, the Whats, the Hows, the What-Ifs and the Holy-Molys. The document was presented to the Product office where it was discussed in-depth, especially the motivations around it. The timing came in handy due to the team’s current level of expertise and also in parts due to MOIA’s service break owing to the COVID-19 pandemic situation.

“When life gives you bats with quirky viruses, make build"

We were quickly given a green light to start on our endeavour, which can formally be formulated as:

    Optimize a rewrite with 4 Developers,
    within a self-imposed ambitious timeline of 3 Months, 
    such that the new system increases Developer velocity, 
    whilst avoiding knowledge silos.

The team was composed of a pair with algorithm and optimization expertise and the other pair with cloud-native and operations expertise. Of the 4, only a single member had had prior experience with Rust and the other 3 were tasked to pick it up in the due course of the rewrite.

The strategy?

1. Research
2. Program
3. Discuss
4. goto 1

Research

Almost everyone nose-dived into the Rust language books and any related learning material like it was High School, sans the awkward Cafeteria interactions. There was no dearth of learning materials — The Book, Rust by Example, Rustonomicon, etc. The language itself brought a new paradigm of ‘Borrowing and Ownership’ which was quite new to some but thereafter quickly understood. The team would often share resources with each other to facilitate learning, especially when it came to writing idiomatic Rust code and the package ecosystem.

Program

Our team in particular lays a high emphasis on Pair Programming, and as such, it was imperative to have two pairs of programmers working in parallel. We took the decision of splitting up the code into the core optimization part and the network part. The optimization part would deal with implementing the algorithms that impact our customers and MOIA's capability to serve them directly. The network part would, on the other hand, deal with all the heavy lifting that comes along with distributed systems, network I/O and high availability.

An additional approach we took was to have the person, who felt the least empowered to write code, write the code(!), whilst the other person in the pair would be the navigator. This ensured that the entire team became comfortable writing code in Rust and additionally sped up knowledge transfer within the pair.

Overtime as both pairs felt empowered within their own respective components, we threw in the proverbial wrench in the machinery. A pair shuffle(!). Now two people had to familiarize themselves with the other component. This ensured that knowledge was disseminated on yet another level to secure a T-shaped competency in the team.

“Divide and Conquer” is nice for organizing tasks. People, however, resonate better with “Unite and Conquer”.

The efforts bore fruit and soon the entire team was comfortable working with most parts of the system independently.

Discussions

An integral component to our “learn the language, write the darn code” rhythm were our “Dev Huddles”. Often someone would find something nice about idiomatic Rust code, a clever algorithm, a yet another young library, or just struggling with some part(s) of the code. After our daily stand-ups we would regularly have these often hour-long huddles to discuss these aforementioned topics, often sharing code snippets.

Our personal favorite was defining the idiomatic way (from over 5 different ways) to convert a string slice &str to a String object (the answer is left as an exercise to the reader).

These discussions served as yet another knowledge sharing and learning platform where noobness was more than welcome and the insane (Rust) was acknowledged.

“If you care about something, make sure to show it… as code.”

Eventually most of us also found ourselves on the many language and library related discord channels that the Rust community has found itself to cherish and build around.

Rinse, iter::repeat

Over the complete period of the rewrite, we persisted this way of working and even applied it ad hoc, if the situation demanded. The team highly enjoyed the sessions, since they were often informational and yet provided everyone the safe space to rant on their favorite misgivings of the language and/or the ecosystem.

The entire endeauvor took us around 2 months to be done with the complete rewrite, with feature complete parity, and in a new language.

And the results? Well, this time they were significant enough to be mentioned in a Medium article. We achieved almost 30 fold raw performance gain with the rewrite and the brilliant minds at Route Optimization are leveraging this performance improvement even further by devising smarter algorithms around the new system.

With all things STEM, often the final numbers take the pie and end up becoming the cause for people to go YEET! For us, however, the positive performance impact was simply an addition on top of something we managed to accomplish — rewriting a service in an unfamiliar language in just under 2 months whilst avoiding knowledge silos. And that’s something we’re immensely proud of. The process was a success, the product shall soon follow.

“Numbers and data are cool, but so are the people that generate them.”

The entire experience only made us more aware of our strengths as a team, the joy of collectively learning something new and not being afraid to take bold decisions when the need arises. And of course, document everything.

“The sole difference between Engineering and playing around? Documenting the results.”

Route Optimization Team — We are hiring!