"Hey, we have 30 new people joining next week." Not something you hear often in a company. At least, it wasn't something I heard at my past jobs.

Hyper growth almost sounds like something out of a science fiction movie or book. "Let's get the ship on hyper growth!" It is not fictional though. It is something very real. Remote is actually in the middle of it.

Hyper growth means the company is growing more than exponentially. In November 2020, Remote was around 35 people. At the time of publication for this article, we have more than 800.

That means 17 times more people in one year.

This averages out to about 50 new people joining per month.

And of course, it's not just the people, but the product, our revenue, and almost everything else that has been growing toward the upper-right at rapid velocity. We are building the world’s greatest employer of record services and international contractor management, which means we are scaling quickly in every way possible.

Navigating this phase of growth is exciting, scary, rewarding, and exhausting, all at the same time. Managing yourself is hard when things get blurry, not to mention leading and managing teams .

As I have experienced both sides — the slower one, as well as this fast one now with Remote — I am excited to share a few learnings that I have had in my time here.

Yes, growing is fun and in theory an economically good thing for your company. But growth doesn't come without challenges.

Here are a few things that the team and I are currently facing. All are things you should be aware of in general, but especially when you are on hyperdrive. My perspective is mostly focused on the product engineering side, so be aware that other areas might experience it slightly differently.

In theory, you can be in hyper growth and not hire a lot of people, but I guess that would be tough on anyone involved. As I stated earlier, Remote is growing. A lot. Hiring is a challenge, but it doesn't stop there. Anything that involves people creates more tasks and problems you need to tackle. Creating a great hiring pipeline, bringing all those people onboard and into the right teams. You might need to train some, and ultimately you also need leadership: people who can help, assist, and grow everyone.

Things are moving fast. Some might break, others might never get built. Hyper growth is challenging. You might have a bigger end goal, but getting there is tough. Making sure that all the people you are hiring are working on the right thing is another challenge.

It all comes down to how your teams and communication are set up. Does everyone know what their goal is? What should they focus on? And if not, how can they figure it out?

Maybe another team is struggling with a lot of important work, so you need to send a few engineers to that team to help out. Putting the overall company goals first is key in a moment like this.

Building the right thing means getting everyone rallied behind the same mission, in the same direction, while at the same time giving each team as much autonomy as they need to enable each individual to do their best work.

During my interviews at Remote and even in conversations with friends, I often get the question: "How are you keeping your culture?" It is a valid question. Introducing many new people, from different backgrounds, different cultures, creates a big potpourri of opinions, values, and thoughts.

This is a topic you should not ignore. You will not come back from cultural mishandling if you fail to address it. If you start out by not caring about values alignment in your team, you will pay for it quite early on. Make sure to have a set of values that influence everything you do, especially hiring.

This is what it ultimately comes down to. Change. Hyper growth means change. In every aspect of yourself, your teams, and your organization.

Acknowledging that, speaking openly about it, supporting your leaders and teams, is well worth the effort. There isn't much else you can do about this. You want change. It is part of the growth, so don't hate it, don't fight it — embrace it.

What is it that I, as an engineering manager of multiple teams, am focused on? Great question. Let me share a few of the topics I confront day in and day out.

Processes: To use or not, that is the question

We've been intentional about not introducing too many unnecessary processes. Things are moving fast, and we want to keep it that way. Still keeping in mind that we need to build the right thing for our customers.

I think with my next sentence I will likely make some friends and enemies at the same time, so bear with me. Ceremonies are cool and all, but I don't think they are always helpful. Especially not in a virtual, async, and fast-paced environment. I really appreciated that coming into the company. I was never a big fan of meetings just for the sake of having a ceremony.

You can very easily over-engineer a system. Sprint meetings, estimation meetings, daily standups, retro meetings, etc. Putting too many ceremonies and processes in place isn't the solution in our eyes. Instead, focus on the hyper growth moment (where your system is constantly going to be stretched) and embrace the fact that it will be challenging and even a bit chaotic.

Hiring plays a big role. Growth means increasing the size of something. It’s very likely this will include the number of people in your company. We have been hiring for a while here at Remote and are continuing to do so with great speed. You can see our job postings here!

We are not just throwing more people into solving complex problems, though. We actually do need more hands on deck with everything we have planned.

Lean on your people team, also known as HR in many companies. They know what they are doing and can support you and the new joiners in the best way possible. But make sure you align with the needs of your team and area.

Keep diversity and inclusivity in mind when writing your job descriptions and building your hiring pipeline. You want to catch biases and bad behavior as early as possible. As I stated above, disrupting your teams and their alignment early on is not a price you want to pay.

I also see it as part of my job to create an environment where people want to stay. Of course, this is not just my job. But hyper growth means change, and early hires still need a sense of stability.

After hiring, the next-most important step is onboarding. You need to bring in all the people you just hired. Sometimes we don't know the exact teams yet. Sometimes the teams are actually still forming — that’s what hyper growth is like. Take this into consideration and make sure to create a higher level plan, plus a tracking database, so you know who is joining when and into which team.

We also use a buddy system, which I have seen work well over my past years at remote companies. This helps the new joiner adapt to the culture and have someone to ask for help. It can sometimes be scary to ask an open question in a channel of over 800 people.

Our first week of onboarding is the same for everyone, regardless of their location and function. The advantage of being fully distributed allows us to give everyone the same experience.

After week one, we try to get our engineers into our codebase as early as it makes sense. There is no stress attached to it — just making sure that they are able to write some code in a timely fashion. Most of the time, week two of your journey is focused on engineering-related onboarding topics, getting you into our codebase.

Another thing that I wanted to highlight is the fact that in hyper growth mode, flexibility reigns. Teams shouldn't behave rigidly. Focuses and goals will shift. Engineers might need to help other teams to get unblocked.

You also will have a set of engineers, the old guard, who know your product in and out. But they are limited. Hyper growth means a lot of people coming into the organization in a short period of time. Shield the old guard engineers from being spread too thin, as their knowledge is a highly regarded resource.

Instilling this flexibility into my teams has helped see the bigger picture and be OK with changes.

Toolkit for virtual engineering leaders in times of hyper growth

Having seen both sides, slow and hyper growth, I noticed some similarities and some differences. Everything you learn or need while growing is going to be important nonetheless. Overall, being a kind, honest, and empathetic human being gets you a long way.

To help with a few more actionable inputs, here are seven skills that I deem the essentials. These are essentials in good virtual and human leadership that will also help you get through a hyper growth period without too much anxiety. Here we go.

This feels almost boring. But I can't state this enough. With communication, everything falls or rises.

Communicate often and transparently, even when it might feel a bit uncomfortable. Document everything. Record all hands and team meetings. Make it possible for engineers to get as much focus time as possible, and lean on async communication wherever you can. This is a practice you can read about in our public handbook.

If people don't know answers to certain topics, give them pointers on where they can go digging for more information themselves. Introduce them to people who can be helpful. Just don't hide anything — make it transparent and available.

If you don't work on becoming more resilient in a hyper growth company, you will have a hard time. As I mentioned above, change is happening all around you, all the time. Not being influenced by it will be important to stay sane.

Understand what helps you stay in a good mood. Learn more about yourself to create an environment that will help you strive while everything around you might be chaotic.

Keep your emotions in check. When something is hard, remember that is part of the deal, but just because work is hard does not mean your life has to be hard, too. Your team members and leaders should be present and ready to support you.

If you are doing things right, things will always be a bit broken. Tell all new joiners that a little bit of chaos is ok. We humans tend to act on things very differently, and as a leader you need to be aware of that.

Show empathy toward your engineers, both the newcomers and the original crew. It will be a hectic and stressful environment at times. Some engineers might need to step out of their comfort zone. Encourage them and give them support.

I am currently doing monthly skip level 1:1s with all the engineers in my teams, in addition to the normal 1:1s with my engineering team leaders to show support wherever I can. This helps me make sure no one feels alone and everyone has a channel to ask for help if needed.

As a leader, your team, the engineers, will look to you to follow whether you like something or not. That doesn't imply any hierarchy in itself, but it is natural for a leader to have people following their cues. You are an example to them, so lead like it.

For example, don't stay on long over your normal work hours. Prevent over-working and end your day normally. Take days or weeks off. If you don’t, they won't either. This goes for everything you do. Want your team to communicate out in the open, transparently? Then don't default to direct message communication all the time.

I am not hiding the fact that this phase can be a bit scary. Maybe even you as a manager struggle sometimes. It is human to do so. Ultimately, we all are humans working together. So there is no reason to hide things from your team.

This is an important skill, not just in a growing company but for everyone, all the time. Feedback helps everyone. When we are focused on the nitty-gritty details of our products or the code we are writing, we forget the bigger picture. Feedback is a tool that helps yourself and others to do just that. Step back.

You've probably seen a painter step back from their canvas? Why? Well, they can't see the whole picture, literally, when they are up close fixing something. They need the physical distance to see it. That is hard for you to do with virtual work and remote colleagues. We need feedback to help us pull out and look at the bigger picture.

Be courageous and curious. The word “feedback” has some negative connotations, but believe me that giving great, constructive feedback is a skill you want to have in your toolkit.

While growth can happen exponentially, your time won't. There are 24 hours in a day, and that is rock solid. The time you have at work is even less, ideally between 6-8 hours a day. Spending that time to the best of your capabilities will be a skill you learn very quickly.

I always consider my calendar like I would my physical office door. It needs to signal to everyone else, if I am available, if I am out of office, or on a quick errand. Even if they can just come in and say hi, or maybe not. Your calendar can do all this for you. Shielding your time is key. You don't want to end up with six calls in the afternoon and just 15-30 minutes slots to get actual work done.

Block time on your calendar. Add your lunch time, add your starting and stopping time. Have an appointment? Add it to the calendar. If you don't want to share details just add a "busy" event. You would do the same if you were in an office.

As a manager on a hyper growth team, this became a superpower for me. I had days with four interview slots, three1:1s, and two team calls. I wasn't able to get anything else done. Yes, this is part of my work, but getting deep work time is impossible in scenarios like this. I learned my lesson. Now, hiring interviews are limited to two per day and only on certain days. I have days with calls and focus work around certain areas, so I have to context switch as little as possible.

Adapt, scale yourself, and embrace the change. Learning quickly, especially outside of your own skill set, is going to be the best thing you can do. Accept the failure that might happen — that is ok. Get the feedback, share it with others.

Don't get hung up on a problem you can't fix yourself. Attack it, not alone but with your peers. Being part of a hyper growth team will grow you too. Only if you are receptive to it, though. Don't waste the opportunity.

Journal, take notes, reflect, and do anything that will help you to take the most out of a situation like this. Stay open minded and be excited to jump on new opportunities.

So yeah, buckle up, friend. It is going to be a ride. It will be exciting and fun, but it will also be challenging. I am not hiding that fact. Embrace it, be open, and learn.

I don't like the metaphor of a rollercoaster that much, but it works. There will be ups, there will be downs. You will be anxious when you see a hill or a fall. But in the end, when you get out of the car, you'll smile and be glad you were on it. And maybe even get a picture as a reminder.

Navigating hyper growth is exciting, scary, rewarding, and exhausting, all at the same time. Managing yourself is hard when things get blurry, not to mention leading and managing teams. A bit of wisdom on remote engineering management can go a long way to help.

Leading virtual engineering teams in times of hyper growth

Although it’s a simple form, it’s probably enough to understand the amount of effort needed to build it.  Moreover, this strategy requires a lot of good and synchronous communication between frontend and backend engineers which also means that it’s error-prone. For example, what if the backend updates the minimum available PTO from 20 to 22 but the frontend is not updated? What if for some reason the name of the property changed on the backend from available_pto to available_pto_days?

This doesn’t scale when forms are part of your core. Causing a lot of frustration for developers, potentially being a source for bugs, and ultimately delaying your time to market.

Using JSON Schemas to supercharge our forms

We needed to take action and thought about solutions that could improve our developer experience, reduce all duplicated code and ultimately accelerate our time to market in functionalities that involved forms.

We decide to use JSON Schemas and delegate form composition and ergonomics, plus data validations to the backend. JSON Schema is a vocabulary that is primarily used to annotate and validate JSON data.

So going back to our example, we could represent our form with the following schema:

properties are the list of fields that feeds the DynamicForm.  Each schema property includes a presentation which is a non-standard property that includes “visual” data from a field. In our example, we can see that work_schedule has inputType set to radio which will map to a RadioField component with a list of options, and a position that represents its position in the form.

When it comes to validation, both fields are required and then each one has specific validation rules, available_pto requires a minimum value of 20 and work_schedule only accepts one of two values: full_time or part_time.

Using a JSON Schema parser on the frontend

We already have DynamicForm and input components so we only needed the last piece of the puzzle: build a JSON Schema parser, that could transform JSON into a list of fields (similar to the example above). Then validate that form and plug it into DynamicForm.

From a code standpoint this could be translated to the component below:

import { createHeadlessForm } from "@remote-com/json-schema-form";

export function MyComponent() {
  const [data, setData] = useState();

  useEffect(() => {
    fetch("api/v1/json-schema")
      .then((response) => response.json())
      .then((data) => setData(data));
  }, []);

  if (!data) {
    return null;
  }

  return <MyForm jsonSchema={data} />;
}

export function MyForm({ jsonSchema }) {
  /**
   * Example response: 
     {
      fields: [{
        description: "Can be either full time or part time"
        label: "Work schedule"
        name: "workSchedule"
        required: true,
        options: [
          { value: 'full_time', label: 'Full Time' },
          { value: 'part_time', label: 'Part Time' },
        ],
        type: "radio"
      },
      {
        description: "Time that employees can take off of work while still getting paid"
        label: "Available PTO"
        name: "availablePto"
        required: true
        type: "number"
      }],
      validationSchema: Yup.schema
     }
   */
  const { fields, validationSchema } = createHeadlessForm(jsonSchema);
  const handleSubmit = (values) => {
    // send to server
  };

  return (
    <DynamicForm
      fields={fields}
      validationSchema={validationSchema}
      handleOnSubmit={handleSubmit}
    />
  );
}

For the sake of simplicity and readability of this guide we’re using simple schemas. However, many use-cases are much more complex. 

A common scenario presents when field properties change based on another field value. This can be done with conditional logic in JSON Schemas and it’s handled by our parser, which means that validationSchema is recalculated when a field with dependencies changes. 

In our specific example, let’s say that the minimum number for available PTO changes depending on the selected work_schedule option.

{
  "additionalProperties": false,
  "properties": {
    "available_pto": {
      …
    },
    "work_schedule": {
      …
    }
  },
  "required": ["work_schedule", "available_pto"],
  "if": {
    "properties": {
      "work_schedule": {
        "const": "part_time"
      }
    }
  },
  "then": {
    "properties": {
      "available_pto": { "minimum": 0 }
    }
  },
  "else": {
    "properties": {
      "available_pto": { "minimum": 20 }
    }
  },
  "type": "object"
}

In the example, if part_time is selected, then available_pto takes a minimum of 0 — otherwise (if full_time was selected), the minimum is 20.

Since the adoption of this strategy, we already started to feel the differences when it comes to development velocity, just by reducing duplicated code, mostly related to validations, and all the required harmony between frontend and backend engineers.

This also brings an extra layer of security to the Remote Platform because all the validations live on the backend, making it impossible to brute force form submission. Moreover, in the long term, we expect to reduce our Javascript bundle size significantly as we migrate older forms to the new development pattern.

In conclusion, our engineering team is very happy with the come-up solution and we’re looking forward to a massive adoption in our platform.

We worked hard and we are wrapping up a beta version of our JSON Parser library that we hope to open-source pretty soon.

Finally, you might be thinking that at this point we need someone to manually create these schemas on the backend. I can attest this can work, but the execution can also be a very time-consuming and prone to error. 

The extra effort is worth it for our team, as we now have a single source of truth for such complex forms, and most importantly, we have the foundations to move to the next step: developing an interface to create JSON Schemas 🚀. 

Stay tuned for the next update on this one!

If you're a web developer, you have probably spent some time in your career developing a functionality involving forms. Depending on the form and the application on the frontend, this can mean a lot of work (particularly if you're determined to provide a solid user experience). 

On the front end you'll need to focus on three critical tasks:

write client-side validation before it's submitted to the server

And on the backend you'll need to move through the following three steps:

do whatever needs to be done with that data

send a response that may be reflected in the form

On top of that, you’ll need synched validations between the frontend and backend. In the long term, that can quickly get out of control. Without careful planning, you could end up with a nightmare scenario to maintain.

The Remote platform depends heavily on forms. The use case of a new country launch demonstrates this in practice. 

In this instance, we need to follow a process of gathering the data requirements for Remote to support a new Employer of Record country activation. For every new country launch in the platform, we need to create around eight forms with country-specific fields. Each of these fields have different specificities. 

If you think about the number of countries in the world (180+), multiplied by the number of forms with different fields each (approximately 8), you can start to understand the number of different possibilities, and all the coordination that needs to happen between frontend and backend. The complexity is immense.

But let’s dive into a simple example and see what needs to be done from a frontend perspective so you can have a better understanding of the problem with developing forms at scale.

We’ll assume we have to create a form with two required fields:

1. Work Schedule: This can take one of two values – full-time or part-time.

2. Available PTO: The time that employees can take off of work while still getting paid (which can’t be less than 20 days)

To create this form, we need to work through the following workflow:

Create input components (a radio button, and input text) which will represent work schedule and available PTO respectively. At Remote we already have a lot of input components and all of them include labels, descriptions and/or placeholders and also accommodate inline errors.

Create the form with the inputs and handle all the boilerplate code, like getting values in and out of form state, validating the payload, rendering error messages, and handling form submission. At Remote, we created a DynamicForm component, which is a headless form that renders the form inputs based on the fields prop. That means we need to manually create this fields list with the field name, description, options, validation schema, and some other specific properties depending on the input.

Orchestrate client-side validation (we use Yup for this) and in our case, we need to make sure both fields are required and that available_pto can’t be less than 20.

On the backend (which we won’t focus on in this blog post), we need to validate the payload coming from the frontend, store it in the database and return a response.

Depending on the form and application, on the frontend, this can mean a lot of work if you want to provide a solid user experience.

Guide to using JSON schema forms

Supercharged Forms Guide: Using JSON Schema In Forms

iex()> country = Repo.get(Country, 2)
%EctoExplorer.Schemas.Country{...}
iex()> country.addresses
#Ecto.Association.NotLoaded<association :addresses is not loaded>
iex()> country = Repo.preload(country, :addresses)
%EctoExplorer.Schemas.Country{...}
iex()> country.addresses
[%EctoExplorer.Schemas.Address{...}, ...]

The explicitness of calling Repo.preload/2 to load an association is more than welcome in Production given the safety it brings to the table, but it becomes a nuisance when we are in a local IEx shell trying to smoothly navigate our data model, hopping from Ecto association to Ecto association:

iex()> country = Repo.get(Country, 2)
%EctoExplorer.Schemas.Country{
  addresses: #Ecto.Association.NotLoaded<association :addresses is not loaded>,
  currencies: #Ecto.Association.NotLoaded<association :currencies is not loaded>,
  flag: #Ecto.Association.NotLoaded<association :flag is not loaded>,
  code: "ECU"
  id: 2,
  name: "Ecuador",
  ...
}
# 1st Repo.preload/2 for the country.flag
iex()> country = Repo.preload(country, :flag)
# ...
iex()> country.flag
%EctoExplorer.Schemas.Flag{
  colors: "YBR",
  country: #Ecto.Association.NotLoaded<association :country is not loaded>,
  country_id: 2,
  orientation: "horizontal"
}
# 2nd Repo.preload/2 for the country.currencies
iex()> country = Repo.preload(country, :currencies)
# ...
iex()> country.currencies
[
  %EctoExplorer.Schemas.Currency{
    code: "USD",
    symbol: "$"
  },
  %EctoExplorer.Schemas.Currency{
    code: "SUC",
    symbol: "Suc"
  }
]
# 3rd Repo.preload/2 for the country.addresses
iex()> country = Repo.preload(country, :addresses)
# ...
iex()> country.addresses |> Enum.at(0)
%EctoExplorer.Schemas.Address{
  city: "city_ECU_1",
  country_id: 2,
  first_line: "first_line_ECU_1",
  postal_code: "postal_code_ECU_1"
  ...
}

As you can see above, the pattern is always the same: 

You now need to explore one of its associations

You need to resort to Repo.preload/2 before actually checking any of the association records

I was getting really tired of the Repo.preload/2 dance, and it got to a point where I decided to scratch my own itch. 

At first, I even considered a quick and dirty hack 🔨 that would consist of a vim macro that would write the Repo.preload/2 call for me 🙈 . This approach might have solved my pain, since I always have both an IEx shell running (as my REPL) and vim inside tmux, but I'm pretty sure this wouldn't be much useful for anyone else but me 😅

I started thinking about what I needed to streamline the Repo.preload/2 usage:

The end goal would be to be able to "chain" one or more association accesses and the Repo.preload/2 calls would happen behind the scenes automatically; this way we would avoid the dreaded look of the Ecto.Association.NotLoaded 🪦

Repo.preload/2 already let's us pass an association "chain" on the second argument (e.g. Repo.preload(flag, [:country, :addresses])), so the new approach needs to improve on what's already provided out of the box by Ecto;

If we consider the last example, a streamlined version of the preload/2 function like X(flag, country.addresses) would already be an improvement. Note that X would be the new function and the second argument is not a string. This improvement would already save some keypresses every day, but I would need to turn the country.addresses part into [country: :addresses]. Maybe some metaprogramming sprinkles would help here? 🤓

By now I was almost certain that my path implied the usage of metaprogramming, so I tried to figure out what I'd get if I had no restrictions on the amount of metaprogramming I'm willing to use. The goal is to chain Ecto association "hops", and since these hops are similar in spirit to map accesses, if we support expressions like flag.country.addresses, we would be keeping a known Elixir pattern.

For this to work though, I would need to somehow override the . (dot) access to cater to our specific needs. From what I gathered from the Elixir source, extending the dot access would not be easy since its implementation is really intertwined with the language "core".

If I can't use the dot access for the Ecto navigation, I'd like to have something as similar as possible to it. I know that Elixir has a lot of operators provided by the Kernel module that are macro-based (check, for example, the [Kernel.<>/2](<https://github.com/elixir-lang/elixir/blob/49bac68affd2e94bda704d6207b6a34cd4f07e76/lib/elixir/lib/kernel.ex#L1903-L1906>)), so I dived into the source of these macros to understand how they come to life 🕵️‍♂️

What I found is that most of these macros use a different macro syntax than what I'm used to see, but this is exactly what allows one to write a ||| macro that would behave like an operator and used like foo|||bar.

iex()> defmodule Example do
...()>   # "operator" macro syntax
...()>   defmacro left ||| right do
...()>     "#{left} AND #{right}"
...()>   end
...()>
...()>   # instead of the "traditional" macro syntax
...()>   defmacro my_operator(left, right) do
...()>     "#{left} AND #{right}"
...()>   end
...()> end
{:module, Example, <<70, 79, 82, ...>>, {:my_operator, 2}}
iex()> import Example
Example
iex()> "foo"|||"bar" # note the usage of `|||` as an "operator"
"foo AND bar"
iex()> my_operator("foo", "bar")
"foo AND bar"

💡 Note the defmacro left X right, do: ... way of defining the macro instead of the more conventional defmacro X(left, right), do: ... that is commonly seen and used.

I felt I was on the right track, I just needed to define an operator symbol for the Ecto navigation that would somehow convey the meaning of "navigating through Ecto associations". I settled on the ~> operator since the ~ looks like a wave 🌊 and the > points forward, hence it would be used to sail through a sea of Ecto navigations (cheesy I know 🙈 , but I didn't come up with a better mnemonic 😅 ).

If I define the macro now and we simply inspect what we get on both left and right parameters, you'll see that we get the quoted form of both params, as every regular macro does 🌈

iex()> defmodule Example2 do
...()>   defmacro left ~> right do
...()>     IO.inspect(left, label: "Left")
...()>     IO.inspect(right, label: "Right")
...()>
...()>     :ok
...()>   end
...()> end
{:module, Example2, <<70, 79, 82, ...>>, {:~>, 2}}
iex()> import Example2
Example2
iex()> foo~>a
Left: {:foo, [line: 15], nil}
Right: {:a, [line: 15], nil}
:ok
iex()> foo~>a.b
Left: {:foo, [line: 16], nil}
Right: {{:., [line: 16], [{:a, [line: 16], nil}, :b]}, [no_parens: true, line: 16], []}
:ok
iex()> foo~>a.b.c.d
Left: {:foo, [line: 17], nil}
Right: {{:., [line: 17],
  [
    {{:., [line: 17],
      [
        {{:., [line: 17], [{:a, [line: 17], nil}, :b]},
         [no_parens: true, line: 17], []},
        :c
      ]}, [no_parens: true, line: 17], []},
    :d
  ]}, [no_parens: true, line: 17], []}
:ok

For each of these expressions, we need to convert the quoted form of the navigation part (the right-hand side of the expression, after the ~>) to a list of steps.

We'll create this list of steps by traversing the quoted right parameter, using the Macro.postwalk/3 function (you can check the full code here, which is a bit more long due to the handling of indexes, ie., steps with an index like country.addresses[3]):

@doc false
def _steps(quoted_right) do
  quoted_right
  |> Macro.postwalk(%{visited: [], steps: []}, fn
    # ...
    {:., _, _} = node, acc ->
      acc = accumulate_node(acc, node)
      {node, acc}
    {first_step, _, _} = node, acc when is_atom(first_step) ->
      acc = accumulate_node(acc, node, %Step{key: first_step})
      {node, acc}
    step, acc when is_atom(step) ->
      acc = accumulate_node(acc, step, %Step{key: step})
      {step, acc}
    # ...
    node, acc ->
      acc = accumulate_node(acc, node)
      {node, acc}
  end)
end
defp accumulate_node(%{visited: visited} = acc, node) do
  %{acc | visited: [node | visited]}
end
defp accumulate_node(%{steps: steps} = acc, node, %Step{} = step) do
  %{accumulate_node(acc, node) | steps: [step | steps]}
end

In a nutshell, the postwalk logic visits each node of the right-hand side AST and accumulates the steps as a list of %Step{} structs. The _steps/1 function tests illustrate how the function works (you can find the tests here):

test "makes steps for a basic right-hand side" do
  rhs = quote do: foo
  assert [%Step{key: :foo}] == Subject.steps(rhs)
end
test "makes steps for a 2-hop right-hand side" do
  rhs = quote do: foo.bar
  assert [%Step{key: :foo}, %Step{key: :bar}] == Subject.steps(rhs)
end
test "makes steps for a 5-hop right-hand side" do
  rhs = quote do: foo.bar.baz.bin.yas
  assert [
           %Step{key: :foo},
           %Step{key: :bar},
           %Step{key: :baz},
           %Step{key: :bin},
           %Step{key: :yas}
         ] == Subject.steps(rhs)
end
test "makes steps for a basic right-hand side with index" do
  rhs = quote do: foo[99]
  assert [%Step{key: :foo, index: 99}] == Subject.steps(rhs)
end

By converting each step into its own %Step{} structure we are able to keep things tidy 🧹. 

As you might have guessed by now, here's how the step list of the flag~>country.addresses expression looks like:

iex()> rhs = quote do: country.addresses
{{:., [], [{:country, [], Elixir}, :addresses]}, [no_parens: true], []}
iex()> EctoExplorer.Resolver._steps(rhs)
{^rhs,
 %{
   expected_index_steps: 0,
   steps: [
     %EctoExplorer.Resolver.Step{index: nil, key: :addresses},
     %EctoExplorer.Resolver.Step{index: nil, key: :country}
   ],
   visited: [
     {{:., [], [{:country, [], Elixir}, :addresses]}, [no_parens: true], []}, #4
     {:., [], [{:country, [], Elixir}, :addresses]}, #3
     :addresses, #2
     {:country, [], Elixir} #1
   ]
 }}

By looking at the visited list from the bottom up, we can figure out what happened:

Then visited the addresses AST node (#2);

Then visited the :. (dot) navigation node (#3);

And finally visited the full expression that matches exactly the rhs value (#4).

With the navigation steps turned into a pretty list of %Step{} structs, we now need to preload the "current" struct with the immediate next step of the list and we'll obtain a new "current" struct. This approach will be repeated until we don't have any more remaining steps.

@doc false
def _resolve(current, %Step{key: step_key, index: nil} = step) do
  case Map.get(current, step_key) do
    %Ecto.Association.NotLoaded{} ->
      current = Preloader.preload(current, step_key)
      _resolve(current, step)
    nil ->
      Logger.warn("[Current: #{inspect(current)}] Step '#{step_key}' resolved to `nil`")
      nil
    value ->
      value
  end
end

As you can see above, we try to get the current.step_key value and if it's an association that isn't loaded yet, we call the Preloader.preload/2 that behind the scenes relies on the Ecto.Repo.preload/2 function to fetch the association. Otherwise we just return the current.step_key value.

At this stage the gist of the streamlined Ecto navigation is behind us. We just need to offer an easy way for everyone to use the ~> navigation operator on their own IEx shells.

The EctoExplorer module defines its own __using__/1 macro so that people can use it to have access to the ~> operator. Since the Ecto navigation needs an actual Ecto Repo to fetch the data from the DB, the repo option is required when using the EctoExplorer module:

defmacro __using__(repo: repo) do

  if Mix.env() not in [:dev, :test] do

    IO.puts(

      "You're using EctoExplorer on the `#{Mix.env()}` environment.\\nEctoExplorer isn't in any way optimized for Production usage, and forces the preload of each association. Use with care!"

    )

  end

  {:ok, _pid} =

    repo

    |> Macro.expand(__ENV__)

    |> maybe_start_repo_agent()

  quote do

    import unquote(__MODULE__)

  end

end

For the navigation logic to be able to use the repo, it needs to have access to it, hence we start an Agent process to keep it. Notice that we Macro.expand/2 the repo value since its value is quoted (💡 recall that any macro parameter is passed in its quoted form, so Foo.Repo would look like {:__aliases__, [alias: false], [:Foo, :Repo]}).

To use the ~> operator, is now just a matter of using the EctoExplorer module:

iex()> use EctoExplorer, repo: EctoExplorer.Repo

EctoExplorer

iex()> f = Repo.get(Flag, 2)

%EctoExplorer.Schemas.Flag{

  __meta__: #Ecto.Schema.Metadata<:loaded, "flags">,

  colors: "YBR",

  country: #Ecto.Association.NotLoaded<association :country is not loaded>,

  country_id: 2,

  id: 2,

  orientation: "horizontal"

}

iex()> f~>country.addresses[0].postal_code

11:20:01.467 [debug] QUERY OK source="countries" db=0.0ms idle=27.6ms

SELECT c0."id", c0."name", c0."code", c0."population", c0."id" FROM "countries" AS c0 WHERE (c0."id" = ?) [2]

11:20:01.470 [debug] QUERY OK source="addresses" db=0.0ms idle=30.2ms

SELECT a0."id", a0."first_line", a0."postal_code", a0."city", a0."country_id", a0."country_id" FROM "addresses" AS a0 WHERE (a0."country_id" = ?) ORDER BY a0."country_id" [2]

"postal_code_ECU_1"

Note that the second f~>country.addresses[0].postal_code expression performed two queries for us, one to retrieve the country, and a second one to retrieve the country addresses association 🎉 .

You can find the library on Github, hope it's as useful to you as it is to me.

By now we walked through most of the EctoExplorer library, but so far we didn't talk about the context that made it possible 🗺️ . The pain with the bazillion of Repo.preload/2 calls, the initial thought process, the additional exploration, and the eventual solution all happened during my normal scope of work here at Remote.

As much as I like to dabble with Elixir metaprogramming, balancing my personal time (family, hobbies) with my daily work means that a library like could never be ready so quickly if it wasn’t for the incredible flexibility of Remote’s weekly learning time.

At Remote we are encouraged to invest two to four hours of our week to expand our knowledge and skills, and this isn't just a pretty bullet point in a slideshow presentation. Our team managers proactively encourage us to use and treasure this time. This learning window is something we truly believe makes us better as an organization. Every journey feels much better when you are allowed to grow each step of the way 🐾

This time I decided to scratch my Repo.preload/2 itch, let's see what comes next!

After joining Remote and starting to find my way around our extensive data model, I quickly noticed how often I was repeating the Repo.preload/2 call to navigate through our data model.

The explicit preload strategy, instead of the automatic eager loading provided by other frameworks (like Rails' ActiveRecord, for example) serves as a deterrent to the unfortunately common problem of having N+1 queries popping up in Production.

With Ecto, you need to be deliberate upfront about whether you need the association loaded for you. Otherwise, you'll get this result when trying to reach out to the country.addresses.

#Ecto.Association.NotLoaded<association :addresses is not loaded> 

The following snippet shows that an association isn't loaded by default. Only after "preloading" it the association becomes "filled" with records coming from the DB.

We've created a tidy and practical library that you can use to help navigate our Ecto data model that our team has been able to create during our learning and development time.

A library to help navigate Ecto Data models

The Remote team has been working hard to deliver the launch of our Remote API, designed to help HR and payroll companies leverage Remote’s powerful global infrastructure to help more businesses hire globally. By taking the international infrastructure that has allowed us to offer a superior experience for businesses and teams around the world and making it available to developers, we’re able to afford so many more businesses the chance to expand their global team. 

In this article, we want to focus on the journey we took to migrate from built-in documentation to using OpenAPI schemas. You will learn why this was so important, and we’ll explain the benefits of using OpenAPI for your Elixir projects. 

A large number of developers are working on the Remote platform, and our team continues to grow every month.

There are many different domains within the product, and different developers have different specialties, both for the domains and for the backend and frontend technologies powering them. This makes our internal APIs essential for effective communication.

We started by using Elixir's built-in documentation features to document our APIs. This served us well for a while. However, as our team grew, we needed more. A collection of issues began to surface that triggered our need for a better solution

The problems with relying on built-in documentation became more obvious, and they are familiar challenges for any team experiencing fast growth:

Maintaining example json responses, query parameters, and request bodies in @doc attributes: This involved copy/pasting json and keeping a list of query parameters.

Ensuring the documentation was up-to-date: at times the docs would go stale and not match the API, because this system relied on the team remembering to update whenever a change was made.

Exposing and browsing the documentation: including extra steps to build docs and opening and finding endpoints. Some developers are not as familiar with the Elixir tooling, another barrier to pass to get to the docs.

Thankfully, there is a better way to maintain API documentation. We want to explain how to simplify documentation management for a growing global team and reduce manual upkeep.

For introducing changes to the development process, we have an internal RFC (Request For Comments) process. In this case, we had two separate RFCs with some overlap around improving our API documentation. After some conversations, we opted to adopt OpenAPI specifications for our internal RESTful APIs.

OpenAPI is a specification for RESTful APIs. It's widely used and the Elixir ecosystem has good tooling for it. From the specification, you can then generate documentation. It has several other functions as well, like generating clients, servers, data validation, and more.

OpenAPI’s own introduction is quite comprehensive, but here's the TL;DR:

An OpenAPI specification can either be a yaml or json file.

You need to define a “Path” for every API endpoint.

Each Path can have multiple “Operations” (get, post, put, etc).

Each operation describes the parameters, payload, and possible server responses.

To make adding and editing specs more integrated to our normal development flow, we decided to use Open API Spex. Open API Spex is a library for Elixir to generate and serve the Open API Spec document from code.

To organize our specs, we mirror the Context structure of the Phoenix app.

The implementation of this has been intentionally gradual. The initial change started by writing specs for two controllers (six operations), with a bit of developer documentation on how to work with Open API Spex.

When working on functionality related to an endpoint, we add specs along the way. Sometimes, we add incomplete placeholder specs which we can later expand on.

Although we are not done yet, we're up to around 150 documented endpoints. Already our team is experiencing some tangible benefits, with cleaner alignment and less manual work leading to speedier delivery on critical projects.

Documentation is easy to find and browse, leading to less confusion about the APIs, fewer questions and a better async workflow.

We can test the specs! Now we can ensure our documentation is reliable and up to date, which ultimately reduces bugs.

We can provide mock data from the specs for the frontend, automatically. Less manual in this regard makes us faster as a collective.

We now have a single source of truth for API documentation!

This last result is such a win for our team and for Remote as a business.

The developer experience boost is significant. This new clear single source of truth makes it so much easier for our team to understand and deliver value. We are reducing the frustration involved in context shifting, which means less errors and faster delivery.

This solution is also more scalable, setting up our processes in the cleanest possible way for new starters - something of critical importance as the Remote team is growing so fast.

Open API means more transparency for Remote

More importantly, this move to migrate from built-in documentation to using OpenAPI schemas delivers on one our continual quest for Transparency; one of the fundamental values of Remote. To succeed in a Remote-first business, transparency is vital. 

Automation, documentation, and scalable processes are the lifeblood of our company. Indeed, any team trying to facilitate sustainable fast growth in a remote environment, either in an engineering team, or any other department, can benefit from more transparent work practices. At the very least, we’re confident that using OpenAPI for your Elixir projects will serve you well for the future.

How OpenAPI schemas can improve documentation, communication, and transparency

Engineering

Leading virtual engineering teams in times of hyper growth

Supercharged Forms Guide: Using JSON Schema In Forms

A library to help navigate Ecto Data models

Open API Specifications: Improving Documentation & Communication

Welcoming Elixir creator José Valim as an advisor to Remote

Written by • Marcelo Lebre

Practical tips to improve your Elixir codebase

How to structure your Sass codebase

How to add dark mode to your website

How to use current_user in Elixir's Phoenix Framework