# Lunar Engineering Blog - Full LLM Context

This file contains the full markdown content for selected Lunar Engineering blog posts. Use /llms.txt for the short index.

Site: https://engineering.lunar.app
RSS: https://engineering.lunar.app/feed.xml

# Data Anonymization at Scale

Source: https://engineering.lunar.app/blog/data-anonymization-at-scale
Date: 2024-05-16
Summary: How Lunar builds and operates a state-of-the-art data anonymization engine to handle sensitive data at scale.

Lunar's data anonymization pipeline removes or tokenizes sensitive fields before data reaches analytics systems. It runs on our standard Go service platform, scales from 3 to 30 pods, handles partitions up to 120 MB, and has operated for roughly four months without downtime.

## By the numbers

| Metric | Value |
| --- | --- |
| Default pod count | 3 pods |
| Autoscaling ceiling | 30 pods |
| Average memory per pod | 224 MB |
| Largest partition size handled | 120 MB |
| Runtime since launch | Approximately 4 months |
| Downtime since launch | 0 incidents mentioned |

## Why do we use Data Anonymization?

At Lunar we continuously produce data as part of our normal operations. A part of having this data is making sure it is safe for use and following regulations like GDPR. Data Anonymization is one such technique, it allows the other services to depend on the data while not being burdened with the Personally identifiable information (PII) that are often associated with this data. It not only makes it easier and safer to process data, but allows a complete inventory of all the data processed.

## What is Data Anonymization?

Data anonymization is the act of altering parts of a data object related to a person, or another sensitive entity, such that the data remains useful but no longer personally identifiable. This act may be undertaken for various purposes and reasons, but generally, it aims to improve governance and control over data usage, making it easier and safer to utilize the data for analytical purposes.

![We can see an event having certain fields removed (address) and replaced with a token (uuid)](/images/blog/data-anonymization-at-scale/image1.jpg)

There are many ways to anonymize data; one method involves replacing sensitive elements with a token, while another mutates the data in place to retain a similar meaning, but without the ability to identify an actual person or object from it. For example, "Hack Kampmanns Pl. 10" could simply be generalized to "Aarhus."

### Data anonymization pipeline

At Lunar, we have internally designed our anonymization pipeline with multiple stages:

Data Entry: Raw data arrives and is stored.

Tokenization: Data is processed, and sensitive data is extracted and replaced with tokens (such as the UUID mentioned above).

Anonymization: Data is either reintegrated into the original data objects or replaced with alternative information, depending on the use case and purpose.

Staging Area: The cleaned data is then placed in a staging area, from where it can be integrated into the rest of the data ingestion pipeline and fed into our data analytics platform.

![The 4 stages of data processing](/images/blog/data-anonymization-at-scale/image2.jpg)

In the following sections, we'll dig into each of the above-mentioned stages.

### Raw data

Raw data objects at Lunar adhere to a stringent and opinionated set of tools and follow a uniform format. Each object is encapsulated in an envelope that provides crucial metadata about the event, including a timestamp, a unique ID, and the originator. This metadata is essential for determining whether we have previously processed the event and which schema should be applied.

Additionally, events are partitioned based on their initial publication date. Our data anonymization platform is designed to handle time-traveling data seamlessly, treating it as a first-class citizen. It is not uncommon for our system to process an event published three years ago, applying the same or modified processing based on schema updates.

Our anonymization pipeline continuously scans our ingested data for any data it has not previously scanned. Upon identifying an unprocessed partition—a set of data objects—it schedules them for tokenization.

### Tokenization

Tokenization is the act of replacing sensitive data fields with tokens, which are pointers. In our case, each token is a globally unique identifier (GUID) that points to a piece of data, with the sensitive data itself stored elsewhere for safekeeping. This process facilitates easier data management and cleanup.

Data arriving from the raw staging area has not yet undergone schema application, making it difficult to interpret its structure natively. The first step, therefore, involves applying a schema to understand and structure the data.

Schemas are already implemented elsewhere in our data platform, especially when event data is packed into Parquet files for our analytics platform. At this stage, we ensure that the data delivered to the analytics platform adheres to a schema and is no longer considered unstructured.

However, the same processing cannot be directly applied to all incoming data due to potential schema incompatibilities with certain data objects. Moreover, given the critical role of the anonymization pipeline, we cannot discard data that does not conform to the expected schema.

We have developed a custom token-sweeping algorithm that optimistically searches through data objects for fields marked as sensitive. This process does not involve applying the schema directly; instead, it navigates through the data, identifying fields of interest, replacing them with tokens, and then forwarding them further in the pipeline. This is possible because the events are protected against breaking changes elsewhere, allowing us to reason about the data's structure directly.

This entire process is executed under a single transaction setup for the scheduled partition. As such, we either complete the handling of the set of data objects or, in case of failure, restart the processing in the next tokenization loop. This method ensures the system's robustness, though it comes at the cost of latency and performance. While we cannot guarantee the exact timing of event processing, we ensure that each event is processed at least once. Thanks to the idempotency built into the tokenization step, encountering the same data twice results in identical tokens.

Once tokenization is completed for a partition, it is moved to another staging area, ready to be picked up by the anonymization step.

#### Tying tokens to an ID

Because sensitive data is subject to various legal requirements, we need a structured and systematic approach to determine when to retire specific fields. Regulations such as GDPR and PSD2 impose stringent rules on when to anonymize sensitive data, such as addresses and social security numbers.

During the tokenization step, we always link tokens to an entity. This entity could be a person or another identifiable unit. When it becomes necessary to retire an entity after a specified period, we simply mark the associated entity ID as retired.

Once an entity ID is marked as retired, our system automatically processes all data objects linked to that entity. These data objects are completely removed from our system. Concurrently, our anonymization engine processes the data, ensuring that it is securely and safely expunged from our analytics platform.

### Anonymization

Currently at Lunar, our anonymization step is relatively basic. When we anonymize data, we replace sensitive data fields with the text "REDACTED". In the future, we plan to develop more sophisticated strategies, as illustrated by the "Address to City" example mentioned earlier. The primary goal is to effectively extract and manage sensitive data without compromising its utility.

These enhanced strategies may include deriving a more generic address, generating a fake yet random social security number, or creating a general but unique IP address. Each method is designed to maintain the usefulness of the data while ensuring individual privacy.

The core of the anonymization process involves taking data objects that have been tokenized in the previous step and processing them with our data anonymization engine. This step either reconstructs the data as it would have appeared originally or inserts the anonymized data where sensitive data previously existed. This allows our data analytics platform to process the data as usual, although it will now encounter the placeholder "REDACTED" in fields associated with retired entities.

In our setup, the anonymization step is essentially the inverse of the tokenization step. It utilizes the same token-sweeping algorithm previously described to reintroduce the original values into the data.

Once this process is complete, the data partition is marked as requiring reprocessing. During the next cycle, our data ingestion pipeline re-ingests the cleaned data, updating the analytics platform with the latest information.

This might seem redundant as if we are simply tokenizing data only to recombine it later. However, the anonymization step can be triggered independently of the tokenization step and usually occurs when an entity's data needs to be retired. Therefore, it serves various purposes beyond just data recombination.

### Cleaned data

The cleaned data is continuously backfilled into our data analytics platform. This process is triggered only for partitions that have undergone changes.

Additionally, the tokenized data partitions — before they have been anonymized or recombined — can be directly utilized if desired. This practice offers optimal data governance assurances.

As such data never contains sensitive information, it can be used for a broader range of analytical purposes without the risk of data leakage. This ensures a wider margin of utility while maintaining stringent security standards.

## Built using standard Golang tools

Lastly, I'd like to discuss the development of our anonymization pipeline. Unlike typical data processing tools, this pipeline was developed using the standard Golang service tools that are fundamental to Lunar's operations.

Data practitioners are likely familiar with tools like Kafka, Spark, and Flink, which are staples for data engineers. While Scala and Java are commonly used in these contexts, they are not the primary technologies at Lunar. Despite having capabilities in these languages, we chose to leverage Golang — the backbone of nearly 99% of all services at Lunar — for this complex, multifaceted pipeline. This approach allows us to utilise our established strengths and infrastructure.

This approach also showcases the capabilities of our Golang service preset. It demonstrates how we can scale to meet demand, ensure fault tolerance, and run on our Kubernetes container runtime. We also leverage Prometheus and Grafana to drive metrics and analytics. Finally, PostgreSQL is utilized for consistent data storage and management. This combination of technologies illustrates the robustness and versatility of our system.

![Overview of data anonymization solution, it is visible how much work the pods are doing, as well as where the work lies in the pipeline](/images/blog/data-anonymization-at-scale/image4.png)

The diagram above illustrates our typical setup of running three pods, which is the minimum default configuration. These pods operate efficiently, requiring no further scaling down. However, our horizontal pod autoscaler is set to a maximum of 30 pods, allowing us to scale up significantly during peak processing times when the system is initially activated.

Additionally, we utilize an estimate from PostgreSQL to monitor the backlog of partitions needing processing and to determine at which stage the bulk of the work is occurring. It's important to note that the names of the stages mentioned in this article differ slightly from their operational names:

"Redaction" corresponds to "Tokenization"

"Sanitize partition" corresponds to "Anonymization"

While not all stage names are detailed here, each plays a crucial role in ensuring safe operations and in the retiring or processing of these partitions.

Choosing Golang proved to be an excellent decision for efficiently processing a vast amount of data. Each pod operates with an average of 224 MB of RAM and is capable of handling partitions up to 120MB in size. Our design goal was to ensure that each application runs as efficiently as possible, allowing them to operate alongside our standard applications without consuming excessive resources.

![Overview of how fast partitions (unit of work) are processed, as well as the database utilization](/images/blog/data-anonymization-at-scale/image3.png)

To manage our backlog and ensure an "at least once" guarantee for our data, we utilized PostgreSQL as both our consistency mechanism and data store. During periods of heavy usage, we maximized resource utilization on AWS RDS (PostgreSQL), which occasionally became our bottleneck. However, this was primarily an issue during the initial data ingestion phase. Under normal operational conditions, AWS RDS performs efficiently, maintaining ample capacity even on smaller instances.

### Since launch

The data anonymization pipeline has been fully operational for approximately four months and has reliably handled our partitions with only minor issues and no downtime, thanks to our 'at least once' guarantee. It has also been instrumental in testing experimental tools for connecting to RDS. Despite occasional issues such as random disconnections or connectivity failures during development, the system autonomously recovered to a satisfactory level, eliminating the need for manual intervention.

## Conclusion

In this article, we've outlined Lunar's approach to processing sensitive data and upholding the highest data management standards. Our experience demonstrates the advantages of developing custom tooling tailored to specific needs, which often simplifies complex problems. By choosing Golang over more traditional platforms like Kafka, Spark, and Flink, we've shown that alternative technologies can not only meet but sometimes exceed the capabilities of conventional data processing ecosystems. This approach has enabled us to achieve robust data anonymization with enhanced efficiency and reliability.

---

# Migrating to Flux v2

Source: https://engineering.lunar.app/blog/migrating-to-flux-v2
Date: 2022-01-19
Summary: Our migration journey from Flux v1 to Flux v2 was not as straight forward as we had hoped, and we wanted to share some of the learnings we got during the migration.

_NOTICE: Originally posted on January 19th 2022._

Migrating Lunar from Flux v1 to Flux v2 reduced a slow monorepo reconciliation path from roughly 8 minutes to 2-3 minutes after we moved the controller to dedicated compute, raised Kubernetes API limits, and tuned controller resources. The migration solved real Flux v1 pain, but it also introduced more moving parts and operational risk.

## By the numbers

| Metric | Before | After |
| --- | --- | --- |
| Reconciliation time | Roughly 8 minutes | 2-3 minutes |
| Services managed | Roughly 250 microservices | Roughly 250 microservices |
| Controller interval | 5 minutes | 10 minutes |
| Kubernetes API QPS | 20 | 250 |
| Kubernetes API burst | 50 | 500 |
| CPU request | 100m | 3 cores |
| Memory request | 64 Mi | 3 GB |

At Lunar we name everything based on a space theme and Nasa in this context is a platform squad with the responsibilities around our developer and container platform. We work on improving the platform and everything around it. In this case we wanted to migrate from Flux v1 to Flux v2.

### Defining terms

#### What is GitOps

Quoting WeaveWorks on GitOps :

"An operating model for Kubernetes and other cloud native technologies, providing a set of best practices that unify Git deployment, management and monitoring for containerized clusters and applications." - [WeaveWorks GitOps definition](https://www.weave.works/technologies/gitops/)

Kasper Nissen wrote a great blogpost on GitOps - "[GitOps operations by pull requests](/blog/gitops-operations-by-pull-request)"

#### What is Flux v1 & v2

One big difference between Flux v1 and v2 is that Flux v1 is basically a simple loop that does `kubectl apply` on a given pool of manifests. It's a controller running in Kubernetes that watches a Git repository and sequentially applies the resources. This is all done by one process.

Flux v2 on the other hand is split up into multiple processes. One is in charge of fetching Git repositories, another for sending notifications when events happen, and another to reconcile. [Learn more about Flux](https://fluxcd.io/).

Flux v2 is also capable of using Kustomize which adds support for both overlays and defining resources for a given service. However, currently, we do not use the overlay functionality as we have a folder for each environment in our Git repository.

#### What is Kustomize

Kustomize is an alternative to Helm that can create overlays, define resources and patches without using templates.

To quote Kustomize :

"Kustomize introduces a template-free way to customize application configuration that simplifies the use of off-the-shelf applications" - [Kustomize.io](https://kustomize.io/)

Flux v2 can be pointed to a Kustomize file and reconcile the service defined in that Kustomize resource. This Kustomize file might point to other Kustomize files with more resources or patches that need to be applied to the data.

#### Mono/multi repo

Defining services and environments in Git you basically have two options. Either you have a Git repository for each service, resulting in a lot of Git repositories - multirepo, or you have all manifests in one big Git repository - monorepo. We have historically been using a monorepo with Flux v1 and automation in front of it to avoid drift in definitions between environments. This has worked fine in the past but will be important later on in the blog post.

#### Shuttle

In Lunar, we use [Shuttle](https://github.com/lunarway/shuttle) to centralize our CI pipelines.

"shuttle is a CLI for handling shared build and deploy tools between many projects no matter what technologies the project is using"

Shuttle uses the concept of a Plan which defines what Shuttle can do. Each micro-service references a plan.

#### Release Manager

[Release Manager](https://github.com/lunarway/release-manager) is a CLI for managing releases to a GitOps Kubernetes repository. It enables developers to release artifacts into a GitOps mono-repo using a single command.

### Current pains

#### Flux v1 deprecation

As Flux v2 was released, Flux v1 was soon deprecated. This means that all new development will happen exclusively on Flux v2 in the future. So we needed to either migrate to Flux v2 or something else to keep getting features and updates.

#### Duplicate definition

If two Kubernetes manifests are created with the same name in the same namespace, the API-server will reject the latter object from being applied. The big problem with Flux v1 is that it will stop reconciling any manifests from this point on in its queue. So, if one of the first objects happens to contain a duplicate definition, none of the services from that point on will be applied. This was a big problem for us as it basically prevented all squads in Lunar from releasing services.

Flux v1 does not support bulk-heading between deployable units. So one failure will block all other services.

#### Our monorepo grew and flux got slower

As our Git repository grew in size, Flux v1 naturally got slower at reconciling the entire repository. Flux v1 has a single process applying manifests to Kubernetes and the only way to scale it is vertically by adding more RAM and CPU.

### Fix in Theory

#### Flux 2 is made to scale

One of the big differences between Flux v1 and Flux v2 is that the new version is split up into multiple processes doing one thing well.

These are:

**Source controller**

A controller's role is to watch a source defined. This is typically a Git repository, but could also be object storage buckets, etc.

**Kustomize controller**

Reconciles the cluster state from multiple sources (provided by source-controller)

**Helm controller**

Watches for HelmRelease objects and generates HelmChart objects

**Notification controller**

The controller handles events coming from external systems (GitHub, GitLab, Bitbucket, Harbor, Jenkins, etc) and notifies the GitOps toolkit controllers about source changes

**Image automation controller**

The image-reflector-controller and image-automation-controller work together to update a Git repository when new container images are available

[Learn more from Flux's components documentation](https://fluxcd.io/docs/components/).

We currently use a Source controller, Kustomize controller, and Notification controller.

By splitting the capabilities of Flux 1 into multiple processes, Flux v2 is by design able to scale to running on various machines and scaling their resources independently.

#### Kustomize as a protocol

We choose to use the Kustomize controller, where we create Kustomize CustomResources (CRs) in Kubernetes that define our services. One Kustomize file per service. This is now how services are registered with Flux. So you could say that Kustomize is now our standard way of defining a service's resources.

Kustomize has several features. One of them is templating which we currently do not use. Another is that you can create a reference in one Kustomize file to another Kustomize file. This is used to have one Kustomize file per environment - and this Kustomize file refers to the services that should be running in the environment by referring to their Kustomize file. Another feature is that you can define dependencies in Kustomize, which Flux will consider when deciding the order of the reconciliation. One case could be that RabbitMQ is deployed before our micro-services are deployed.

#### Automating kustomize in plans

As of January 2022, we have roughly 250 micro-services running on our platform. Each of these has its own Git repository and a folder in each environment with its Kubernetes manifest in our mono repository. None of these had a Kustomize file, so we had to automate this.

First, we created a script that would go through the entire folder structure in our mono repo and create a basic Kustomize file (see service-a-kustomization.yaml in example below). This enabled us to start deploying our services to a given environment.

Second, we added a step to our pipelines for each service, that would ensure the desired Kustomize state would end up in its Kustomize file in the manifest repository.

And third, we added logic to our release manager that ensured that kustomize files would end up in the right location as we keep them seperate from our Kubernetes manifests.

```
GitOps-repo
.
|__ clusters
|   |__ dev
|      |__ serviceA
|      |  |__ service-a-kustomization.yaml
|      |__ serviceB
|         |__ service-b-kustomization.yaml
|__ dev
    |__ releases
       |__ serviceA
       |  |__ deployment.yaml
       |  |__ service.yaml
       |  |__ ingress.yaml
       |__ serviceB
          |__ deployment.yaml
          |__ service.yaml
          |__ ingress.yaml
```

### Observations

#### Monorepo was really slow

It turned out that when ever our monorepo was updated with a new release Flux v2 had to go through each and every Kustomization.

Reconciling a queue took roughly 8 minutes with this default setup. This means that from a developer pushes a new release to an environment in Git, it can take up to 8 minutes before it gets in a running state. This is in contrast to the old Flux v1 setup where it took about one minute from a Git commit to the service was running.

In our Grafana dashboard for Flux v2 we could see the queue depth and work queue rate and noticed that the work queue rate was always high, and never really settled. The system was basically always running a reconciliation.

This was not the performance we were hoping for. Now the hunt to find the cause started. We did a lot of Googling and eventually found that Flux v2 had to go through the entire queue for each commit to the mono repository. Had we used multiple repositories this would not have happened. The reason this happens in a mono repo setup is that one Kustomize file can reference another Kustomize file, so to be sure that all is reconciled, it has to go through all Kustomize files in a repository.

Now the choice was basically if we should cancel the upgrade or try and move forward with improvements. 8 minutes was not acceptable to us.

We had a couple of knobs to turn and we used our Grafana dashboard to monitor for changes in performance. For every change, we had to redeploy the controller. Our problem was that we did not get consistent results. Sometimes we would get better performance by turning a knob like `concurrent XXX` and other times we would not.

It turned out we had a bottleneck that outweighed the other bottlenecks in the system. Sometimes this big bottleneck would allow more throughput and other times less throughput, despite improving the bottleneck we were focusing on or not.

This bottleneck of bottlenecks - also called the constraint of the system, turned out to be CPU resources. The initial resource definition on the Kustomization controller was set as follows:

```
resources:
limits:
    cpu: 1000m
    memory: 1Gi
requests:
    cpu: 100m
    memory: 64Mi
```

#### Grafana dashboard was important

Lets have a look at the actual CPU usage in Grafana:

![Flux v2 initial CPU usage](/images/blog/migrating-to-flux-v2/flux-initial-cpu.png)

We can see that it flatlines a lot. And, another problem is that the Kustomization controller pod is scheduled based on the `resources.requests` field. So, sometimes it would get scheduled on a node where no CPU was used by other services and other times on nodes with heavy CPU usage.

Looking at the memory consumption, we can see that it is quite stable so we did not have any concerns about memory.

![Flux v2 initial memory usage](/images/blog/migrating-to-flux-v2/flux-initial-memory.png)

The graph showing the work queue depth and rate show each queue's start and end. On the work queue depth, we can see a peak every time someone releases a service. This is where Flux v2 will put all Kustomizations from that repository into the queue and start working its way through it. We can see that it takes roughly 8 minutes from start to end.

![Flux v2 initial work queue depth and rate](/images/blog/migrating-to-flux-v2/flux-initial-workqueue.png)

The work queue rate shows when it's actually doing something. It is using an average of more than two minutes. Flux v2 has an `interval` parameter that tells Flux how often it should re-check a Kustomization. By default, this is set to 5 minutes. So every 5 minutes, a service is being checked for changes. If you have more services than your Kustomization controller can process before the first service is set to be checked again, then you will always have work being done. It will never be at rest. This condition makes less room for the webhook reconciliation as well, so we sought to change this from 5 minutes to 10 minutes intervals.

We use small nodes (m5.large), which have 2 `vCPU` and 8 `GB` memory. To solve the CPU usage problem, we decided to put Flux v2 on a dedicated node (m5.xlarge) 4 `vCPU` and 16 `GB` memory in the cluster. This way we could ensure Flux the resources it needs. The CPU request is set to 3, and from the following graph, we can see that it utilizes that. There are also some `daemonsets` running on each node which is why it does not use all of the CPUs.

![Flux v2 dedicated CPU usage](/images/blog/migrating-to-flux-v2/flux-dedicated-cpu.png)

To get control of the memory usage, we set the `request` and `limit` to 3 G. As we can see here it does not necessarily use it all.

![Flux v2 dedicated memory usage](/images/blog/migrating-to-flux-v2/flux-dedicated-memory.png)

![Flux v2 dedicated work queue depth and rate](/images/blog/migrating-to-flux-v2/flux-dedicated-workqueue.png)

As seen on the following screenshot, the changes we made to our Flux 2 deployment meant that a system reconciliation (workqueue depth) now takes 2-3 minutes to process instead of 8 minutes.

Also, the workqueue rate shows that it actually has a chance of getting through the queue before a new reconciliation is promised.

Here are the changes we made:

```
> Kustomize controller interval went from 5m to 10m
> Flux running on a dedicated node with more resources
> kube-api-qps - default 20 - now 250 - QPS to use while talking with kubernetes API.
> kube-api-burst - default 50 - now 500 - Burst to use while talking with kubernetes API server.
> Removed CPU limit
> Set CPU request to 3
> Set memory request and limits to 3G
```

### Conclusion

In the end, the migration was good for us as it solves some of the pains we had with Flux v1, e.g. bulkheading and duplicate definitions. It also makes it possible to describe the order services are deployed in. For example, RabbitMQ will be deployed before microservices.

Flux v2 is very different from Flux v1. Therefore, we recommend that you play around with it before upgrading. The only thing Flux v1 and Flux v2 have in common is the name. Flux v2 is more complex, flexible, and extendable but introduces a higher risk from an operational point of view. At one point we killed one of our dev clusters while experimenting, as we wanted to move the main Kustomize manifest from the `flux-system` namespace to a different namespace. It introduced a race condition that ended up deleting the `flux-system` namespace before creating it again. With Flux v1 this isn't a problem, but we have all our Kustomize CRs in that namespace, and the Kustomize controller saw the deleting resources before closing down. When it does so, it will remove the services described in the Kustomize CR. As a result, Flux v2 deleted all services in that dev cluster.

One learning we can share is to scale the Kustomize controller to zero if you want to do anything related to Kustomize resources, flux-system namespace, etc.

Flux v2 requires more compute resources than Flux v1 in our case, so we ended up using a dedicated worker node. This is partly because we have a mono repository, which on every commit will trigger a reconciliation on all services. If we had multiple repositories it would only reconcile services in the given repository.

### Next steps

Dependency management between Kustomization will be a huge help between our platform squads to create a boundary that is understood by each squad. This will make it easier for us to do failovers between clusters, as it will remove manual steps that currently control the order of deployment. It will also make it clear what each squad will deliver as a product to its customers.

We are in the middle of splitting up our platform squad into two squads, and with Flux v2 we can create a Kustomization for each of these. The Kustomization resources then point to services delivered by the squads and can have dependencies underneath it. These high-level Kustomization resources can then depend on each other so that resources by the Container Platform squad will be deployed before the Application Platform squad.

A great resource for inspiration can be found here: [fluxcd/flux2-multi-tenancy](https://github.com/fluxcd/flux2-multi-tenancy).

---

# GitOps - Operations by Pull Request

Source: https://engineering.lunar.app/blog/gitops-operations-by-pull-request
Date: 2020-03-10
Summary: For the past year, or so, we have been investigating, built PoC's, discussed options, decided on a model, and actually implemented a GitOps model at Lunar.

_NOTICE: Originally posted on March 10th 2020._

For the past year, or so, we have been investigating, built PoC's, discussed options, decided on a model, and actually implemented a GitOps model at Lunar.

This blog post will walk you through our considerations, the different models we see in the current ecosystem, and the model that we decided to implement.

### Prerequisuites

But before diving into the details, let's first discuss a couple of terms and definitions, to make sure we are on the same page.

GitOps is an opinionated implementation of Continuous Delivery.

Continuous Delivery (CD) refers to the practice of being able to quickly and sustainably get changes into production and extends continuous integration by automating the release process, so that you can release your application at any point in time.

Continuous Deployment (CDE) goes one step further, by automatically releasing every change that passes the workflow of your production pipeline.

The last term I would like to discuss before moving into the details of GitOps, is reconciliation.

If you look up the term reconciliation in a dictionary, you will get something like this:

"the process of making two people or groups of people friendly again after they have argued seriously or fought and kept apart from each other, or a situation in which this happens"

or the more generic definition,

"the process of making two opposite beliefs, ideas, or situations agree"

By this definition, it referes to two different views agreeing and to become one.

In the context of Kubernetes, you have probably encountered this concept when applying a Deployment to your Kubernetes cluster. In Kubernetes, we specify a desired state in our yaml resource files or using kubectl, an example of this could be

```
replicas: 3
```

With this statement we tell Kubernetes that our desired state is 3 instances of our application. It's now the job of the Kubernetes controller-manager to try and make the two different states agree, hence drive the desired state to be the current state of the cluster.

Ok, now that we know what reconciliation means, let's see how this concept applies to GitOps.

### What is GitOps?

The term GitOps was coined by Alexis Richardson, CEO of Weaveworks a couple of years ago.

They define the term as:

GitOps is a way to do Kubernetes cluster management and application delivery. It works by using Git as a single source of truth for declarative infrastructure and applications. With Git at the center of your delivery pipelines, developers can make pull requests to accelerate and simplify application deployments and operations tasks to Kubernetes.

To put it in one sentence;

GitOps is leveraging git as the source of truth to reconcile applications and infrastructure.

In order to be able to use git as the source of truth, we need a way for the cluster to detect and apply changes to the cluster. We need a controller with a reconciliation loop running somewhere looking for changes in the desired state, and drive the current state towards the source of truth. Let's call that cluster reconciliation and could look something like this:

![Cluster reconciliation](/images/blog/gitops-operations-by-pull-request/reconciliation.png)

Now, in order to make changes to your cluster, you create a PR with the desired changes, and commit that to your cluster configuration repository. Eventually your changes will be detected and the control loop will drive your cluster towards the desired state.

You might be thinking, this seems like an unnecessary extra step, and further, what is wrong with just using `kubectl apply` either directly from local machine or in a CI/CD pipeline?

Many people, us at Lunar included, built their first pipelines to Kubernetes with some simple scripts to generate Kubernetes yaml, and apply the changes to Kubernetes using a simple `kubectl apply` from the CI/CD server.

However, this approach came with quite a few caveats and problems. The asynchronous nature of the controller-manager makes it hard to detect when a container in a pod is actually ready to receive traffic. You can come a long way with using the `kubectl rollout status` in your pipeline to wait for pods to be in `Running` state, but what happens when the container crashes 5 seconds later, because it can't reach the database? It's possible to implement a loop in a script that can detect these failure modes, but wouldn't it be much better if you just get a message when this occurs?

This way of doing it, works to some extent, and was a good place to start for us. But it came with drawbacks highlighted below:

The CI/CD system needs pretty permissive rights to your clusters. From a security standpoint, this might be a no-go.

Executing `kubectl` from a CI/CD system is a very imperative and commanding approach, instead of the declarative approach that is used in many aspects of the Cloud Native ecosystem.

All Kubernetes yaml was generated using shell scripts in the pipeline and dismissed once applied to the cluster. This made it very hard to track the diff between what was running and the new desired state. As a financial institution we need to be able to audit every change, and this implementation made that hard.

No single source of truth of what was actually runnning in our clusters. In case of a cluster going down, it would require us to run through all services and generate new configurations, which would take time, and be error prone.

In this section, I've provided you with an introduction to GitOps, and some reasoning behind choosing this approach from out standpoint at Lunar. Next, let's look at the different options available.

### Flavours of GitOps

Back when we decided to look at the different options of how we could implement GitOps at Lunar, these were the approaches we saw. I have named them and will in the next part of this blog post go through them one by one.

- Decentralized One-way flow
- Decentralized Two-way flow
- Centralized flow

There are no formally accepted naming of these different architectural approaches but to make the discussion clear and referable these are the names that will be used.

#### Decentralized One-way flow

The first approach we will look at, is what I refer to as the decentralized one-way flow. The following diagram provides an overview of the flow. The main thing to note, is where the controller is running, and how it gets its updates.

![Decentralized One-way flow](/images/blog/gitops-operations-by-pull-request/one-way-flow.png)

Let's go through it step by step. A developer pushes a change to a service repository (assuming you are using a polyrepo strategy). This push triggers a continuous integration job that will build, test, scan, push, and generate the Kubernetes configuration. Instead of applying this configuration directly to the cluster, it pushes the config to a centralized repo. A controller running in the Kubernetes cluster detects a change in git, either by polling or using webhooks, and applies the change to the cluster. The idea in this approach is to use the config repo as the trigger for new changes. Further it gathers all configuration for a given service in the service repository.

#### Decentralized Two-way flow

The Decentralized Two-way flow is the flow that Weaveworks and the [flux-cd](https://github.com/fluxcd/flux) project highlights.

This flow is a bit different from the Decentralized One-way flow in that a Docker registry is used as a trigger for new deployments, and the controller has the ability to update the config repo, based on the newly built image.

![Decentralized Two-way flow](/images/blog/gitops-operations-by-pull-request/two-way-flow.png)

Further, this flow separates configuration from the source-code, meaning that if you want to make any configuration changes to your service, e.g. change an environment variable, you have to create a PR against the config repository.

The flow starts with a developer pushing a new change, a CI server picks up this change, builds, tests, scans, and pushes an image to a docker registry. This triggers a controller in the cluster to apply the changes, and commit the updated deployment specification back to the configuration repo. If a developer or operator at some point has to change configuration for a given service, this has to be done in the config repository. This gives operators a great benefit as they can manage all configurations in one central place. In this setup the controller needs a read/write access to the config repository, whereas the one-way flow only requires read access.

#### Centralized flow

The last flow I will highlight in this blog post, is what I refer to as the centralized flow.

The idea in this flow is similar to the other flows. A developer pushes a source code change, a CI job builds, generates configuration, and pushes the configuration to the config repo. Next a centralized control-plane running in a centralized environment detects the change, and applies the state in the given environment. As you can see in the diagram below, we now have one component controlling the rest of our environments. This requires a centralized controller which have access to the Kubernetes API's in the environments it needs to control.

![Centralized flow](/images/blog/gitops-operations-by-pull-request/centralized.png)

### Where should service configuration live?

In the previous section, I provided a brief overview of the different flows, or architectural choices you need to decide on before embarking on a GitOps journey. An important differentiating factor between the 3 flavours of GitOps, is the question: Where should service config files live?

We've had many discussion internally at Lunar about this exact question, and we decided to put everything in the service repository. The reason for this decision is found in an ongoing mission - namely to reduce the operational overhead put on developers and services, and provide tooling that abstracts many of the nitty-gritty details away. We do not have a centralized operations team, or release team, and it's therefore of the utmost importance, that developers can build and run their services by themselves. The way we do this today is by using another open source project we built called [shuttle](https://github.com/lunarway/shuttle).

`shuttle` abstracts things such as Dockerfiles, Kubernetes yaml, common tooling, etc. and provides a CLI entrypoint that all services uses. Each service repository is using, what in `shuttle` terms, is called a plan. A plan is a set of centralized scripts, templates, etc. that is available for the given service using the `shuttle` CLI and a config file called `shuttle.yaml`. This file contains metadata about the service, but also how e.g. environment variables, some Kubernetes configurations or similar should differ between our environments.

`shuttle` will handle the rest and generate Kubernetes yaml files based on this simple yaml file. These files can now be committed into a config repository and applied by a controller.

### How to deal with multiple environments?

As you probably have encountered now, there's many ways to implement a GitOps approach. A question you definitely will encounter when embarking on this journey is: how to deal with multiple environments?

You have quite a few options, when deciding on how to deal with multiple environments:

- Multiple git repositories each representing an environment
- One git repository using branches to represent an environment
- One git repository using directories to represent an environment
- Keep the configuration in the service repository

The choice you have to make here, very much depend on whether you want to use an already accessible GitOps solution, like [Flux CD](https://github.com/fluxcd/flux) with [Weave Cloud](https://www.weave.works/product/cloud/), or [Argo CD](https://argoproj.github.io/argo-cd/), or maybe build your own tooling to support this.

Either way, you need a place to store your configuration, and a way to move configurations between environments.

### GitOps at Lunar

In the previous section on "Where should service configuration live", I provided some hints to how we at Lunar chose to architect our GitOps setup. In this section I will try to go a bit deeper, and touch upon the reasoning behind our choices.

Let's first start with the big question: why use GitOps at all? There's a couple of reasons why we thought this way of doing it was appealing. As you may know, we are building a bank. In this context, there are many requirements and regulations that we have to live up to. One of them being audit logging. Who did what and when? We need to be able to answer this question at all times.

Another reason or goal we had upfront was that we needed a good way to limit access to the environments - without losing too much agility. We walk a fine line, balancing speed and agility against compliance and security.

The last reason I will highlight in this section is disaster recovery. We need to be able to bring back our environments in case something goes terribly wrong, and we need to be able to do that easily and without shaking hands. It should in theory be a non-event if a cluster dies.

These three reasons were the main reasons why we found GitOps interesting.

#### Our solution

One of the first decisions we made was to go with a one-way decentralized flow. The reasons for this, as I explained earlier, are that we want to focus on the ease of use from a developer perspective. They are the ones who day to day uses the tooling, and it needs to support their workflow and help them become effective. This was as mentioned also the reasoning behind choosing to have all service configuration in the service repository.

The next decision was to decide whether we want the controller to commit back to our config repository, or not. We really like the simplicity of using the git repository as the trigger for new deployments, and disliked the fact that the controller could commit changes in our config repository. We felt more in control with a one-way flow.

We choose to use [Flux CD](https://github.com/fluxcd/flux) as our controller. Flux works great and is easy to setup and configure to be used in the flow that we wanted. Flux runs in each of our environments and is configured to listen for changes in a specific directory of the config repo. We have configured flux to be read-only and to not listen for changes in the docker registry, and thereby only react on changes in the given directory of the config repo.

But, the big question for us, was to find a way to move the configuration files between the different directories (environments) that flux listens on, e.g. how do we move a deployment from one environment to the other?

We couldn't really find an answer to this question in the market or in the open source community. At least not a solution that fits our needs. Instead we set out to build our own "release manager".

### Release-manager

The [release-manager](https://github.com/lunarway/release-manager) project consists of 4 components.

- release-manager server
- release-daemon
- hamctl
- artifact

The release-manager server is the heart of the system, and is the central point of contact. It receives webhooks from git, the release-daemon, and further the CLI, `hamctl`, also interacts with it.

You might be thinking, why did they name their CLI tool after pink meat? Actually Ham was the [first chimpanzee in space](<https://en.wikipedia.org/wiki/Ham_(chimpanzee)>). As you might have noticed, we have a space theme going on at the office.

The server is responsible for moving files around in the config repo, e.g. releasing artifacts to environments. artifacts in this context refers to kubernetes yaml for each of the given environments along with metadata.

In the following I will go into a bit more detail to explain each of the four components.

#### release-manager server

The release-manager server's primary job is to move artifacts to their destined environments. It acts based on a couple of different events. The CLI, `hamctl`, talks to the server to either get information or change state. When a new artifact is pushed to the config repo, the release-manager receives a webhook from GitHub to notify about the change. The release-manager then checks to see if there exist an auto-release policy to any environment, and if one exists, it will release the artifact to the given environment. If no policies exists, it will do nothing. Currently there exists two ways of releasing an application, `promote` or `release`.

The `promote` event will release software based on an implicit notion of the environments, in our case it's;

```
master > dev > staging > prod
```

This means that, if you instruct the release-server to promote to environment `staging`, the release-manager will figure out which artifact is running in `dev`, and move that artifact into the `staging` environment.

The other possible event, `release`, can either release based on a specific branch or an artifact-id. This allows developers to release a specific branch to a specific environment.

#### release-daemon

The next component we will discuss, is the release-daemon. The responsibility of the release-daemon is to report state changes from the given environment back to the release-manager server. It works as a Kubernetes controller running in each of the environments, and listens for changes of deployments and report each of the pods' states back to the release-manager. If the release-daemon sees a pod crash, either by `CreateContainerConfigError`, or `CrashLoopBackOff`, it will fetch the latest loglines of the container before reporting back to the release-manager.

#### hamctl

The third component, is the CLI-tool, `hamctl`. This is how our developers interacts with the system. If a developer wants to promote a version of their application, they can do that using hamctl, as follows

```
hamctl promote --service example --env staging
```

Besides the promote method, developers can also release their applications using the `release` command. Let's say I have a hotfix on a hotfix branch, and I need to release this into staging for testing.

```
hamctl release --service example --branch hotfix --env staging
```

The CLI also has a couple of other functions such as getting status from a specific service, e.g. which artifacts are deployed to the different environments. Policies can also be controlled using the CLI.

#### artifact

The last component, is the `artifact` CLI. The main idea behind this component, is to gather information from the CI pipeline, but is also responsible for reporting CI status to Slack.

`artifact` produces a json-blob called `artifact.json`. At each step of our pipeline, a call to artifact is executed to report the state of that step. This state change is written to the json-blob and reported to the git-author via Slack.

This was a fairly brief overview of the system we have implemented at Lunar. All code for our [release-manager](https://github.com/lunarway/release-manager) solution is publicly available in our GitHub account.

Let's revisit the three reasons why we wanted GitOps; audit logging, limited access to environments, and disaster recovery.

With the system we currently have deployed, we now have all the audit logging needed. All interactions with our systems are logged using commits, and can be found in the git history. Secondly, all releases are decoupled from our Kubernetes environments, which means our developers don't need access to the environments. Lastly, all our environments are now stored in git at any given time, which makes disaster recovery a lot easier.

### Everything in Git

You have now seen some of the different options in implementing GitOps, and also our specific implementation. I want to wrap up this blog post with a teaser for a possible upcoming blog post.

We are working towards a scenario where we want to leverage the Kubernetes API, and Kubernetes extensions, Custom Resource Definitions, to have even more of our infrastructure declared in git.

This could be creation of databases, users, but also cloud resources, such as AWS S3 buckets, and even machines.

I hope this blog post offers some perspective of the different choices you have when implementing GitOps, and also provides reasoning why this approach might be a good idea, especially in highly regulated environments.

---

# Consistency Guarantees in a Microservice Architecture

Source: https://engineering.lunar.app/blog/consistency-guarantees
Date: 2019-04-23
Summary: The shortcomings of the early Lunar Way platform and what we have done to improve the platform to support the plans we have for the future.

_NOTICE: Originally posted on April 23rd 2019. We were known as 'Lunar Way' until late 2019._

A year ago I shared our first learnings from entering microservice land in the blog post From Rails Monolith to Microservices — Part 1. Back then my intention was to write a follow-up post to finish the adventure and share all the key learnings we gained as part of our journey. Unfortunately, that blog post was never finished and at this point in time, it really doesn't make sense to write the intended part 2. Instead, this post will be about the shortcomings of the early Lunar Way platform and what we have done to improve the platform to support the plans we have for the future.

### Digging into the Lunar Way platform

In the Monolith to Microservices post, I left the Lunar Way microservice platform after we had implemented the first microservice. Since then, a lot more services have joined, but the fundamental platform architecture has not really changed since then.

Our backend is a microservice architecture with asynchronous message passing as the first choice of inter-service communication. Synchronous RPC using gRPC is used where the async mechanism is not possible.

Services may subscribe to whatever upstream events they like. This is how a service gains required knowledge about data owned by an upstream service and how business processes are implemented as a flow of messages between services taking part in the process.

Services are deployed in a Kubernetes cluster using a CI/CD pipeline triggered by git commits.

Our early microservice platform had a few other important properties:

- Most services were implemented as CRUD services with messages being published after state changes were persisted. Published events were not stored inside the producing service.
- Events were stored in what we call the Poor Man's Event Store: a consumer of all relevant events which stored these in a database. We have built a tool to replay specific events from this database.
- Bootstrapping a new service was a manual process. Sometimes done by constructing synthetic events based on the entity state in the source service, sometimes done by replaying events from the event store database.

### Houston, we have a challenge

We have had great success with the platform: over the last two years, we have deployed around 80 different services. This enables a great number of new features and products to our users. In this sense, the platform has proven its worth: without a platform based on loosely coupled microservices, we would probably never have been able to deliver the stuff we have. Also, the way our tech team is organized — in highly autonomous squads — is very dependent on the platform characteristics: each squad is able to develop, deploy and control its own services with a high degree of independence from the other squads.

However, over time we identified a number of challenges in the platform. Challenges which had an impact on not only our work as developers, but also on our users and our in-house support team.

Let's go through them.

#### Non-atomicity of message publishing

Messages were published after the state was changed in most of our microservices, i.e., the service received an external request, executed the required business logic and persisted the updated entity in the database and then it published a message.

The problem was that the event was not guaranteed to be published. Persistence of the updated entity and publication of the event was not atomic; hence the message publication could fail or the service could die at exactly this moment.

The severity of this problem was further enhanced by the fact that for obvious reasons we didn't know if and when it happened. We only knew at a later point in time if the missing event caused an inconsistency in downstream services or if a business process was not being completed.

#### Zero or once message delivery

We use RabbitMQ as our message broker. This piece of technology has proven to be a reliable and performant way for us to publish messages. Furthermore, the semantics of topic exchanges make it easy for us to add new services as subscribers to specific messages.

However, RabbitMQ was a far too important component in the message delivery chain: in failure scenarios where RabbitMQ for some reason failed to deliver a message to a subscriber, we had no way of re-delivering the message. In combination with the problem of non-atomicity, message delivery guarantees were, therefore, non-existing.

#### Consumers unable to reason about events

A producing service, in general, did not itself store the messages and events it produced. Hence, a producing service did not itself know about the events it produced and it could therefore not answer questions about them — it only knew about the current state of the entities it stored.

Messages had a sequence number which was essentially just a timestamp. However, no inherent guarantees existed about this sequence number: e.g. two events pertaining to the same user from the same service could have sequence numbers out of order if the events had their origin in two different instances of the service. Furthermore, when creating synthetic events in a bootstrapping use case, the sequence numbers would usually be different from what it would have been in the non-synthetic case.

These facts trickled downstream to consuming services. Since a received event was not a first-class citizen with a well-defined identity and ordering, a consuming service could not reason about it, e.g. determine whether it had already received the message or figure out if it had missed an earlier event.

#### The impact

The impact of the inherent characteristics on the early platform as a whole centers very much around the guarantees — or the lack of guarantees — in relation to consistency across services.

Due to the described characteristics, a downstream service was not guaranteed to receive all events it subscribed to and it had no way of knowing if it had lost events. For the same reason, there's no way in which a consuming service on its own demand could verify the consistency of the upstream data it had received.

Consequently, a downstream consumer had no other choice than to blindly trust the messages received from the upstream.

The bottom line was that there were no consistency guarantees in the early platform. Downstream services would maybe or maybe not have a consistent view of what happened in upstream dependencies. In these cases, the data was not lost from the perspective of the downstream service — it just required manual intervention to fix the situation.

### The consequences

#### Support cases and bad user-experience

The most problematic consequence of the problems described were of course when it directly influenced our users. This could be the case if events were lost somewhere along the chain from producer to consumer. The result of this would be inconsistent data in a downstream service or some business process being terminated before completion.

Inevitably, this would give rise to a support case and the only way to remedy the situation was by manual intervention of a developer.

Fortunately, this did not happen very often.

#### Manual bootstrap of new services

When a new service is introduced, it typically requires some information from other services in order to do its job. For example, almost all of our services require some user information in order to determine which actions to take for a specific user.

In the early platform, providing this information to a new service was a manual process, which required a developer to feed the proper messages into the new service. This was either done by side-loading relevant messages from the Poor Man's event store into the new service or by implementing functionality in an upstream service to provide the required data.

This was a cumbersome process and due to the events not having a strict order, we had to take care that historic events replayed from the event store wouldn't overwrite live events.

### Desired characteristics

With the realisation of the inherent problems of the early platform also came a desire to fix them and instead build consistency guarantees into the platform.

The key to unlocking these guarantees was to solve the identified problems.

Hence, we set out to improve the platform with a set of characteristics being the logical opposites of the problems above:

- **Atomic message publication** — no state change without an event
- **At least once delivery** of messages
- **Events as first class citizens** with a persistent nature
- **Reliable event ordering** with reproducibility of events

#### Implications

These characteristics have a number of important implications for the architecture:

Producing services will never perform state changes without publishing an event, and the producing service will itself know about this event — it's as real as entities.

Downstream services have the ability to make sure that they have a correct version of the data in an upstream service: they can use the strict ordering of events to deduce if they have missed any data, and use the reproducibility of events to ask for any missing information.

Downstream services may take care of bootstrapping by themselves without any manual intervention: this can be done either by demand as part of a request from the outside, or it can be done in a batch manner.

These properties in combination deliver the consistency guarantees we regard as a requirement for the Lunar Way platform of the future — a platform which we can scale to 100.000s of users with services always having a consistent view of data of upstream dependencies.

### Implementation

There is more than one way to design an architecture with the 4 desired characteristics. One such solution is event sourcing and this is the design pattern we have chosen to introduce into the Lunar Way platform. I will not give any introduction to event sourcing here — there are a lot of good introductions to this architectural pattern to be found.

Here's a short list of resources we have found helpful:

- [Confluent blog posts on event sourcing](https://www.confluent.io/blog/tag/event-sourcing/)
- Martin Fowler on [Event Sourcing](https://martinfowler.com/eaaDev/EventSourcing.html), [CQRS](https://martinfowler.com/bliki/CQRS.html) and [domain driven design](https://martinfowler.com/tags/domain%20driven%20design.html)
- [Daniel Whittaker's blog](http://danielwhittaker.me/)

By design, event sourcing provides a way to achieve the 4 desired characteristics:

- **Atomic message publication**: Since the internal event stream of an event sourced service is the only persisted entity, there is no difference between the state and the event — the events are the state. Also, publication of domain events derived from the internal events of the service may be implemented as projections thereby providing similar guarantees on publication of domain events.
- **At least once delivery**: If implemented as projections keeping track of their position in the events stream, domain events can be guaranteed to be published at least once. Of course the chosen message technology also plays an important part here, since the producing side can only guarantee that the message was published. The actual delivery guarantee rests on the message technology.
- **Events as first class citizens**: Again, this is a feature which by design is a property of event sourcing.
- **Reliable event ordering with reproducibility of events**: Event ordering comes for free in event sourcing and since the events are persisted, reproducibility of events is built-in.

### Implementation challenges

#### Mindset

Event sourcing is a very different way of thinking about a service. Coming from traditional CRUD services, developers are used to thinking about the state of an entity and requests coming in from the user modifying this state. The modification is not persisted — it only lives in the code processing the request and maybe in a notification message afterward. In event sourcing the current state is secondary — it's something we delegate to views to care about. What really matters is the actual change.

Thinking this way is a change of paradigm and requires an effort to get used to.

Also, event sourcing comes with a lot of new terminology and concepts, which can be overwhelming, to begin with.

At Lunar Way we started out with event sourcing as a hackathon project. This project was eventually promoted to a real service and deployed into production without being exposed to user requests. We used this service as a way of getting to know the concepts and terminology and actually ended up implementing our own event sourcing library in Go based on the experience from the first service. We're planning on open sourcing this library when we think it's ready.

#### Event sourcing events versus domain events

One source of confusion when talking about event sourcing is the very word "event". People often tend to think of the events in an event sourced system as events published to the outside world. This is a misconception. The "event" in event sourcing is fundamentally internal to the service — it's the entity used by the service to store state. It's a big mistake to publish these internal events as available events for downstream services to consume. Doing this exposes the inner workings of the service to the outside and introduces a very hard coupling between services.

Instead, an event sourced service must publish "domain events" in the DDD meaning of domain. Due to the nature of the internal event stream of an aggregate root in the event sourced service, these domain events may be implemented as projections with the same guarantees about order and reproducibility as the inner event stream. (Check out this [blog](https://www.innoq.com/en/blog/domain-events-versus-event-sourcing/) for an excellent discussion of this)

#### Coexistence with non-event sourced services

Implementing an event sourced service as part of an existing platform of non-event sourced services — as in our case — can be a bit of a challenge. If the new event sourced service is at the very top of the dependency hierarchy, it's not a problem.

However, if the service must consume events from upstream services, there is a challenge if the new event sourced service expects upstream events also to provide the same set of guarantees around delivery, order and reproducibility. If retrofitting upstream services with these guarantees is not an option, you have no other choice than to implement an adapting layer between the upstream and the event sourced service. This adapter must guard the event sourced service against replay of the same events and augment events with an order. The first event sourced service we implemented did have upstream dependencies which we had to adapt to. We did this by implementing an adapting layer inside the service itself.

### The benefits of it all

Apart from the consistency guarantees which is what we are really after, event sourcing has other benefits too.

#### Audit log for free

One of the selling points you often hear when it comes to event sourcing is that you get an audit log for free. This is true to the extent that the events in the event sourced system contain the required information to act as an audit log. What you get is a complete log of all the changes in the system, but if all relevant information from the action triggering the change is not available in the events, it is not really an audit log.

#### Domain logic encapsulation in pure functions

Implementing an aggregate root (AR) in an event sourced service follows a very strict pattern. When processing a command, the AR cannot perform any side-effecting actions — i.e. everything the AR requires to determine whether to execute the command must be a part of the command. This means that the command processing implementation is a pure function without side effects. Thus, the domain logic becomes a pure function which can be tested easily without requiring mocks or complicated test setups. Along the same lines, reproducing bugs from production is possible by replaying the events from production and execute the same command.

### Wrapping up

At Lunar Way we have big plans for the future and in order to achieve these goals, our platform must be able to scale. We believe that building consistency guarantees into the platform is a key element for this.

If you have had similar challenges and found different solutions, we'd love to hear about it. Feel free to leave your comments below.

### What is Lunar Way?

Lunar Way is a fintech company motivated by rethinking the experience of banking, and the way people perceive money and spending in general. That is why we are using the most innovative and smart technology in order to create the banking solution for tomorrow directly in our app.

Read more on [lunarway.com](https://lunarway.com/)

---

# Domain-Driven Design at Lunar

Source: https://engineering.lunar.app/blog/domain-driven-design
Date: 2022-07-26
Summary: How Lunar uses Domain-Driven Design principles to manage complexity and organize our microservice architecture.

At Lunar we are organized within Squads, each working independently on their mission. To achieve focus and efficiency we follow Domain-Driven principles by taking ownership and dividing it into different domains. A domain can be [defined](https://www.merriam-webster.com/dictionary/domain) as a 'sphere of knowledge, influence, or activity' and in development, it often refers to its intended application. Domain-Driven Design is a term coined by Eric Evans, in his 2003 book of the same name, but it has since evolved in the hands of the developer community into much more. At Lunar it has become a critical part of our growth mindset.

### What is Domain-Driven Design?

[According to Martin Fowler](https://martinfowler.com/bliki/DomainDrivenDesign.html#:~:text=Domain%2DDriven%20Design%20is%20an,through%20a%20catalog%20of%20patterns.), who we mentioned as an inspiration in [this 2019 post](/blog/consistency-guarantees) on how we designed our architecture, Domain-Driven Design is an approach that centers software development on creating a domain model that offers a rich understanding of the processes. It is described as particularly well suited for complex domains, which admittedly running a FinTech operation can sometimes be.

> "A model is a selectively simplified and consciously structured form of knowledge"
>
> — Eric Evans, Software Design Consultant at Domain Language Inc. 2003

#### Ubiquitous Language

The idea of basing software systems on a model of a domain has long been around, but importantly, Evans developed a vocabulary to talk about it. He called this the 'ubiquitous language' and its purpose is to have all team members use it to connect all the activities of the team. On top of this identifying conceptual elements and embedding them directly in the code, imbues it with knowledge and a sense of its domain. Through this approach, you are able to evolve software models throughout their lifecycle. Antonio Theodorides, our Solution Architect at Lunar describes the most important aspect of Domain-Driven design as having a unified language between tech and the business. Being able to reason about problems using the same language to find solutions minimizes the gap in communications and supports alignment. It decreases cognitive load on the individual developer. But still, Domain-Driven Design is more than that:

#### Understanding and defining domains in Bounded Contexts

Domain-Driven Design is an understanding of one's domain. Working in a domain and taking ownership of your software means you become the expert. At Lunar, we give developers code-ship-run ownership. This lets developers drive - and thrive and with it, we have adopted the saying:

> "You build it, you run it."
>
> — Werner Vogels, CTO of Amazon, 2006

Theodorides describes it as being able to understand all the parts that make up your business and how they interact. To be effective, a model needs to be consistent and have no contradictions. At the same time, a model can only be understood in the right context.

> "If sophisticated domain experts don't understand the model, there is something wrong with the model."
>
> — Eric Evans, Software Design Consultant at Domain Language Inc. 2003

As a domain may grow larger, it becomes harder to build a single unified model. To combat this Domain-Driven Design divides larger systems into what is known as a Bounded Context, each with its own unified model.

![Source: BoundedContext, Martin Fowler](/images/blog/domain-driven-design/bounded-context.png)

Bounded Contexts can have unrelated concepts inside but also share several. Different contexts may have completely different models of shared concepts. Various factors draw boundaries between these contexts. The boundaries are created and defined by the work of a specific team or squad.

### How we practice Domain-Driven Design at Lunar

Some aspects of the approach may seem obvious, and many software developers already practice aspects of it intuitively. However, it is critical that developers use the business language consciously. Sharing the same vocabulary in naming classes in code as in strategic management, creates value for our business. Part of the job as a developer is working with non-coders to understand what you are meant to deliver - likewise, for any newcomer to the Squad, this might help them learn and catch up quicker. Theodorides puts it simply: It's clarity. And that clarity brings much value; Better understanding, easier maintenance of code, better ways of integrating the code, and easier manners of changing the code. Domain-Driven Design is part of how we build things at Lunar, part of our architecture, and part of the onboarding process for developers. Each Squad should be domain-centric and understand the business reason behind every change.

> "It's part of our thinking here at Lunar."
>
> — Antonio Theodorides, Solution Architect at Lunar, 2022

#### Event Storming - Part of our Strategy Design

Theodorides shared also how Domain-Driven Design becomes part of our business strategy through what is known as Event Storming (although he calls this 'the world according to Antonio' and emphasizes there are other methodologies for doing this). He explains that somewhere in the context of Domain-Driven Design making its stride in the early 2010s, an Italian IT-Consultant by the name of Alberto Brandolini invented Event Storming. Event Storming allows us to understand what is going on in the business end-to-end. It allows us to find our pain points. Say, for example, we onboard a new customer:

- What triggers us onboarding the customer?
- Where does that trigger happen?
- What events does that trigger?
- Who receives these events?
- Commands or actions executed.
- Actors execute these commands or actions.

> "We get to learn about the business while simultaneously revealing this knowledge to the business itself. These shared insights and knowledge are extremely valuable."
>
> — Antonio Theodorides, Solution Architect at Lunar, 2022

Big Picture Event Storming like this can be extensive, so we do not always want to do it. However, it gives us the ability to ask what problems we can solve, when we want, and which pain points we can do something about. Suddenly Domain-Driven Design becomes strategic. We can now figure out the steps we need to take to find a solution, what can and cannot be solved right now, and how to prioritize. Strategic Domain-Driven Design means what we need to do and which objectives we need to work on, while tactical Domain-Driven Design more becomes a question of how. Strategic Domain-Driven Design also splits domains into different subdomains:

- **Core domains** - this is where we can differentiate ourselves as a business and part of what makes our organization special. This is where we need to build our own software for where there is no common solution. This should receive the highest priority and biggest effort.
- **Supporting subdomains** - these support our core domains. They are necessary for the organization to succeed, but do not fall under the core domain category. They are not generic either as they require some level of specialization.
- **Generic subdomains** - Here we can make use of what is already available, for example for regulatory reporting (although this still does spawn complex situations in our world from time to time). We can save a lot of resources by picking up off-the-shelf software for these subdomains.

![An example of different kinds of subdomains at Lunar. Inspiration: Strategic DDD, Peter Holmström](/images/blog/domain-driven-design/subdomains.png)

#### User Obsession & Extreme Programming

Evans is also a big proponent of what is known as Extreme Programming (XP), an agile methodology to programming. He views Domain-Driven Design as a natural component of XP. It is an iterative process that features short development cycles and frequent releases and is intended to improve responsiveness to changing customer requirements. So while this may not yet be a formalized part of our approach, it fits perfectly well with Lunar's efforts to be not just user-centric, but user obsessed. In the world of finance, things are constantly changing.

> "The heart of software is its ability to solve domain-related problems for its user."
>
> — Eric Evans, Software Design Consultant at Domain Language Inc. 2003

By now you may be thinking to yourself that much of this is just one man's opinion and while that very well may be, we have not just built our approach on his words alone. The Domain-Driven Design community has since 2003 grown large and spawned many additional learning materials which have further evolved the approach. Today DDD is truly in the hands of its practitioners and in true crowdsourcing fashion, all the quotes from Evans' book shared in this blog post have each been [highlighted by at least 1000+ Kindle readers.](https://www.amazon.com/gp/product/0321125215/ref=as_li_tl?camp=1789&creative=9325&creativeASIN=0321125215&ie=UTF8&linkCode=as2&tag=martinfowlerc-20)

#### Challenges

Being a scale-up is another contributor to things constantly changing and some of our Squads are still in their infancy. Domain-Driven Design certainly has a strong foundation here, but the level of implementation varies throughout the organization. Some Squads have got their language, allowing for clear conversations about their domain events and integration. Some still have to figure out what their domain is and what entities even are in theirs. In Domain-Driven Design the language and model should evolve alongside the team's understanding of the domain, but with many new hires over the last couple of years, this can be a challenge for us as Domain-Driven Design encourages keeping the same domains with the same people.

> "What you are trying to do is enriching your ability to work in these domains, by enriching your domain knowledge."
>
> — Antonio Theodorides, Solution Architect at Lunar 2022

When you expand rapidly, there is no getting around that people have got to learn and time needs to be set aside for it. There is effort required and it has to be maintained. This is important because, if it is not maintained, then it becomes inaccurate and then the domain expertise crumbles as nobody trusts it. There is a cost to this, but the benefit definitely outweighs the cost, says Theodorides.

![Source: Public talk at 2022 AWS Financial Services Denmark by Kenneth Fiil (Head of Tech at Lunar), April 2022](/images/blog/domain-driven-design/lunar-architecture.png)

### Domain-Driven Design fits our culture

Reinventing banking is not all that easy - but it is after all what Lunar is all about. We dabble in things here that have never been done before and because of the many unknowns this brings, we sometimes need to navigate domains where we are not necessarily the experts - at least not yet. Lunar is built on a foundation of technology and a growth mindset. Independence is key and we use Domain-Driven principles to achieve this goal and narrow down an area of interest to its core. This is why we delegate specific problems for each Squad to solve, allowing Squads in time, to become experts at what they do - even if they are first to do it. Traditionally, companies used to strive for total unification for the entire business. Domain-Driven Design instead recognizes that this is not feasible, nor cost-effective. The success of our Squads is apparent and even through organizational changes as we have grown from a small startup to a scale-up, this element of our business persists.

### About Lunar

Lunar is a FinTech company founded in Aarhus, Denmark 2015 motivated by rethinking the banking experience. Unlike many other FinTechs Lunar operates as its own bank, with a license provided by the Danish Financial Supervisory Authority in August 2019. Lunar is a 100% digital bank, with a mission of giving you back control over your money by making managing your finances understandable, accessible, and easy for anyone. We now serve more than 500.000 customers across Denmark, Norway, and Sweden - and not just private banking customers, but businesses as well!

For more information visit our main website [here](https://www.lunar.app/).

#### Want to work for Lunar?

Lunar is full of talented people working on incredible things all the time. With more than 550 people across the Nordics - Aarhus, Copenhagen, Stockholm, and Oslo we are constantly evolving. This site is about just that: The technologies we utilize and create, our ways of working, our learnings, and company culture. If you are interested in joining us, check out our [careers page](https://jobs.lunar.app/jobs?department=Technology)!

---

# Introducing shuttle

Source: https://engineering.lunar.app/blog/introducing-shuttle
Date: 2019-01-15
Summary: shuttle is a small project, written in Go, that centralizes shared scripts and configuration for microservices. We built it to solve the problem of replicated scripts across 70+ services.

_NOTICE: Originally posted on January 15th 2019. We were known as 'Lunar Way' until late 2019._

At Lunar Way we have a lot of microservices. Currently, we have around 70 microservices running in our production kubernetes environment along with multiple infrastructure services that provide monitoring, logging, etc. A good deal of them is developed using the same language, architecture, and libraries.

### Decentralized vs centralized?

When we started our microservice journey, we chose to go with a decentralized repository strategy (also known as a polyrepo opposed to a monorepo). The main reason for this was a clear separation between services, but also keeping all configuration needed for deployment in one place. The project structure was, therefore, similar to the one below:

```
├── <project files>
├── Dockerfile-service
├── Dockerfile-unittest
├── Jenkinsfile
├── Makefile
└── kubernetes
    ├── deployment.sh
    ├── dev
    ├── env.sh
    ├── prod
    ├── secrets.sh
    ├── staging
    └── verify.sh
```

The directories and scripts above are what was replicated between individual service repositories. We used Makefiles to create a common interface on top of each service making it easier for developers to move between services.

However, as the number of microservices grew from the cozy 5–10 services to now 70 services, these replicated scripts became a big problem — and we, therefore, felt the need to do something.

Further, we didn't want to burden our developers with kubernetes configuration or Dockerfiles if it was not necessary - they can go fast and far with a set of sane defaults.

### Building a high-level abstraction with centralized configuration

Instead of drowning ourselves with tedious configuration changes across 70 microservices, we wanted to build a tool that could provide a common higher level of abstraction on service repositories with centralized configuration.

At Lunar Way we currently have two main types of services; Go and Node.js services. Their pipelines and kubernetes configurations are a bit different, and therefore the tool also needed to support this.

We looked at the multitude of projects for service templating and configuration for kubernetes, such as helm, forge.sh, ksonnet, and many others, but none of them matched our exact need. Many of them were solely focusing on kubernetes and didn't provide an option to build the generalized abstraction for all the services we needed. This abstraction should, therefore, be able to wrap actions such as code generation for Go, run tests, build docker images, push docker images, and generate kubernetes configuration, and much more.

To include all of these tools and tasks, we needed a dynamic CLI tool that allowed for centralized configuration and customization.

So we started out investigating different ways to build such a tool, and after our research, we came up with the following requirements for our initial iteration:

- It should be easy to iterate on developer tooling and CI scripts without too much replication work across projects
- Projects should specify how they differ from the norm, not how they are alike to limit boilerplate coding
- Scripts for the CI server and for the developers should share as much code as possible, so scripts that run locally would give the same results on the CI
- Testing CI scripts locally should be easy and without pain

Going over the requirements and options, we eventually found that the tool we needed didn't exist.

Tada: shuttle!

### Introducing shuttle

shuttle is a small project, written in Go, that is installed as a binary on a CI server and developer machines.

It acts as the Makefile executor, but with some main differences:

- The targets are stored external to the project (with an optional feature to add some project specific ones internally)
- Variables and configuration are stored internally on the service repository
- shuttle centralizes shared scripts and configuration, and further, it provides a common abstraction on each service repository. shuttle uses a centralized repository for configuration, also called a `plan`.

A simple overview is shown below 👇

A plan can be comprised of all things imaginable. At Lunar Way we have created 3 different plans, i.e., 3 different plan configuration repositories:

- lw-shuttle-go-plan
- lw-shuttle-node-plan
- lw-shuttle-infrastructure-plan

There are similarities between some of these plans, especially the Go and Node.js plan. In the future, we will like to look into options for sharing actions between plans, but this is a future enhancement. Below is an example of the available scripts in our shuttle Go plan at Lunar Way:

Let's turn our attention to the plan repository and its contents. We've set up a simple example of a Go project that can be found [here](https://github.com/lunarway/shuttle-example-go-plan).

Let's have a look at this simple example plan 👀

```
scripts:
  build:
    description: Build the docker image
    actions:
    - shell: shuttle template -o Dockerfile Dockerfile.tmpl
    - shell: docker build -f $tmp/Dockerfile -t $(shuttle get docker.destImage):$(shuttle get docker.destTag) .
  push:
    description: Push the docker image
    actions:
    - shell: docker push $(shuttle get docker.destImage):$(shuttle get docker.destTag)
  test:
    description: Run test for the project
    actions:
    - shell: go test
  deploy:
    description: Deploys the image to a kubernetes environment
    actions:
    - shell: shuttle template -o deployment.yaml deployment.tmpl
    - shell: kubectl apply -f $tmp/deployment.yaml
```

This simple plan provides 4 scripts at the service repository. Now, when you build more and more services you use the same centralized configuration along with custom variables for the given service. If you are interested, the example service repository can be found here.

This was a short introduction to shuttle, why we built it, and how you can leverage it for your own cloud native architecture.

### Proving the value

To provide some experience with proving the value of adopting shuttle, we did a lot of testing of different docker and dependency scanning tools. Because our services rely on this centralized configuration, which includes a centralized pipeline, we could easily test 3 different docker container scanning tools at the same time in the pipeline. We could easily test our setup locally because these were just built as shuttle actions. This also demonstrated that we could use shuttle to minimize our Jenkins configurations, and instead move our logic to the service level, and thereby minimizing the job of Jenkins to only invoke shuttle actions. This is awesome!

Because it allows us to change our CI server rather easily because we don't rely on custom logic. If we compared this test of different security solutions with our old setup with decentralized config, it would have taken ages to get the validation of the different tools that we needed.

With shuttle we merely just build an action for each tool that ran the scanning, which took a few hours.

### Future

We have many ideas to what and where we want to take shuttle, but first and foremost, we think this tool could help a lot of other people finding the right abstraction for their projects. We are thrilled to take contributions in the form of issues, PR's, ideas, etc.

To read more, and keep yourself posted — please follow us at [GitHub](https://github.com/lunarway/shuttle).

### What is Lunar Way?

Lunar Way is a fintech company motivated by rethinking the experience of banking, and the way people perceive money and spending in general. That is why we are using the most innovative and smart technology in order to create the banking solution for tomorrow directly in our app.

Read more on [lunar.app](https://lunar.app/)

---

# Adopting Go at Lunar Way

Source: https://engineering.lunar.app/blog/adopting-go-at-lunar-way
Date: 2017-12-01
Summary: Our experience of introducing the Go programming language to our backend, including runtime, concurrency, and learning curve.

_NOTICE: Originally posted on December 1st 2017. We were known as 'Lunar Way' until late 2019._

A previous blog post,"Lunar Way's journey towards Cloud Native Utopia", covered our motivation for building Cloud Native services, and highlighted how it helps us achieve velocity and autonomy in our feature squads.

This blog post will cover our experience of introducing the Go programming language to our backend.

### Background

Lunar Way's production backend is comprised of around 30 microservices deployed in a Kubernetes cluster. We are currently undergoing a "service explosion" in our pursuit of decomposing our monolithic Ruby on Rails service into several services. The majority of our services are written in TypeScript using the Node.js runtime (Node), and to simplify the integration with some partners we have taken advantage of code generation from WSDLs to Java.

### Introducing a New Language

Before introducing a new language we needed to consider if the magnitude of such a task was worth the benefits. We use around 10 different internal npm packages in our Node applications, for common concerns and conventions such as log format, event communication, gRPC, database repos, and localisation. In addition, we use Swagger for specifying our services' APIs. Our build and deployment process is handled by Jenkins (using Docker), and as a consequence the dependency management, build process, and test phase is needed for a new language.

Before settling on Go we had a good look at the language, followed what revolved around it, and tested it against a couple of concrete use cases during a hackathon in March 2017. We feared Go would be too low-level (pointers), or lack the appropriate libraries given the smaller community.

Due to differences between TypeScript and Go, some of the internal packages ended up looking different than we expected. The lack of generics led us to use code generation for our database "library", instead of using database repositories with generics. Our event consumer/publisher package for Go is utilising go-routines and channels heavily, which has made it very performant.

At the time of writing, we have two non-user facing Go services in production and a handful in the pipeline.

### Why Go?

Go caught our attention for several reasons, such as a powerful concurrency model, small runtime, performance, statically typed, and maintainability through simplicity. The fact that most components in our infrastructure are written in Go strengthened our interest in learning some of the internals. Better support for CPU bound work was one of the features we were looking for in a language to supplement Node due to its single threaded model. The creator of Node, Ryan Dahl, had an interesting comment on Node and Go on a podcast a couple of months ago:

"If you're building a server, I can't imagine using anything other than Go. That said, I think Node's non-blocking paradigm worked out well for JavaScript, where you don't have threads" - [Ryan Dahl](https://www.mappingthejourney.com/single-post/2017/08/31/episode-8-interview-with-ryan-dahl-creator-of-nodejs/)

Apart from the above, Go seems to have a lot of momentum, and is being described as the language of the cloud. Stack Overflow's Developer Survey has had Go in the top 5 in the category "Most loved", and in 2017 Go came in at #3 in the "Most wanted" category, its first appearance on this list.

### Runtime

As the amount of services increase, the relevance of the runtime's size increases. The figure below shows the minimum and maximum pod memory metrics from Kubernetes over the last week, grouped by runtime. So far we have seen 6–14 MB used in our Go services. To be fair, only event handling and gRPC are used so far in our Go services production. It is, however, pretty good compared to our smallest Node service, which is using 53 MB only handling events. From the measurements below it's seen that around 71 Go services can run to one Java service, using the highest observed pod memory.

Go compiles to binaries, which we add to empty Docker scratch images, which results in image sizes ranging from 4 to 8 MB, where the largest images include packages for event handling, REST endpoints and gRPC endpoints. In addition to image size, we get the security benefit of leaving out several unnecessary dependencies compared to larger base images. The image sizes shown below aren't only due to the language runtimes size, but also how well we construct the images. Go makes that easy by only requiring a binary.

### CPU Bound Work

Another area where Go really shines is CPU bound work and concurrency. As we are strangling Ruby on Rails, we aren't interested in Node for tasks like PDF generation. A previous example with image manipulation in Node showed us how CPU bound work blocked the event loop and caused long response times. This was solved by offloading the work to an AWS Lambda function to utilize Node's async I/O.

Go's concurrency model makes this a lot easier by utilising the simple concurrency primitives that are handled underneath, by the Go scheduler on top of threads.

### Learning Curve

Go is a simple language that is easy to get started with, and the "Tour of Go" works very well. Structuring a project and understanding how the conventions around the GOPATH, workspaces, dependencies, and multiple repositories fit together initially caused a bit of hassle. A couple of concepts in Go differ from most other languages. State belongs in structs, while behaviour belongs in functions, instead of having both on a class for example. Interfaces are implicitly implemented instead of explicitly stated. Inheritance doesn't exist, but [embedding](https://go.dev/doc/effective_go#embedding) allows you to "borrow" pieces from other structs, or combine an interface of interfaces. Concepts such as channels and go-routines build on [CSP](https://blog.golang.org/share-memory-by-communicating) and have some overlap with the actor model. These concepts were new to some of us, but they wrap the error-prone parts of concurrency in a simple, effective way.

The standard library provides a lot of what you need in high quality. The example below only uses the standard library. It shows how a custom struct can be written to a buffer, or a file, using the built in io.Writer interface. This example is inspired by the [following blog post](https://medium.com/@as27/a-simple-beginners-tutorial-to-io-writer-in-golang-2a13bfefea02).

```
package main
import (
 "bytes"
 "encoding/json"
 "fmt"
 "io"
 "os"
)
// User struct wraps the state and json mappings
type User struct {
 ID   string `json:"id"`
 Name string `json:"name"`
 Age  int    `json:"age"`
}
// Write method takes the io.Writer interface as input
func (p *User) Write(w io.Writer) {
 b, _ := json.Marshal(*p)
 // Writes to the io.Writer whether it is a buffer, file, or another implementation of io.Writer
 // Error handling is left out of this example
 w.Write(b)
}
func main() {
 me := User{
  ID:   "828a4b25-daf9-4857-96de-4af40aa6da2e",
  Name: "Martin",
  Age:  28,
 }
 // The first io.Writer is a buffer
 // The json marshalled user is written to the buffer and printed out
 var b bytes.Buffer
 me.Write(&b)
 fmt.Printf("%s", b.String())

 // The second io.Writer is a file
 // Defer closes the file before leaving the current scope
 // The file is saved using the same method as above
 file, _ := os.Create("demo.json")
 defer file.Close()
 me.Write(file)
}
```

Handling JSON is less flexible than in TypeScript, but on the flip side you can trust your runtime that a variable is whatever you specified it to be.

The tooling around Go is, in my opinion, exceptional. Building, testing and running is blazingly fast, and every detail seems thoroughly thought out. Among my other favourite features are: table-testing, code formatter, documentation, race detector, and simple cross-compiling.

### Dependencies

Dependency management is [mentioned](https://www.youtube.com/watch?v=5UG57xQL_RE) as one of the big challenges of Go. A working group is steadily working on an "official experiment" called [dep](https://github.com/golang/dep), that we have been using with great success so far. When we started out the documentation was limited, but it has improved a lot. Dep may not be quite there yet, but it allows us to lock down our dependencies, among other things, which is a great start even though it's still a bit slow. Coming from npm, we had an issue of a transitive dependency introducing a memory leak, even though all our level 1 dependencies were locked to a specific version. 😬

### Conclusion

Overall, we have been very satisfied with Go. It fits well into our existing setup with its small runtime. The amount of services becomes less significant in terms of resource overhead, and allows for fast and frequent deployment of smaller pieces. The powerful concurrency model fills a gap that we were about to encounter, and Go gives us a lot of control without feeling too low-level. The lack of generics is still a bit annoying, but using code generation works better than expected, and maybe that's the price of simplicity.

The simplicity of the language makes it explicit and aligns our code style internally, which (hopefully) will make it easier to get acquainted with a new service.

Introducing a new language takes time and effort, but it has been a very positive experience, and Go has definitely found its place at Lunar Way.

---

# From Rails Monolith to Microservices

Source: https://engineering.lunar.app/blog/from-rails-monolith-to-microservices
Date: 2018-03-20
Summary: A traveller's report of a journey from a monolithic backend architecture to microservices, with discoveries and takeaways about the best approach and pitfalls.

_NOTICE: Originally posted on March 20th 2018. We were known as 'Lunar Way' until late 2019._

In earlier tech focused blog posts a number of my colleagues have talked about aspects of the Lunar Way platform and the organisation of the tech team building it.

The term "microservice architecture" either explicitly or implicitly played an important role in these posts:

- In Lunar Way's journey towards true autonomy (Part 1) the focus was on the ability to organise tech teams in to highly autonomous feature squads. This would be a lot more difficult if we did not have a microservice architecture.
- The ease with which we adopted a new language (Go) into our backend tech stack, as described in Adopting Go at Lunar Way, was only possible due to our microservice architecture.
- Lunar Way's journey towards Cloud Native Utopia focused on our introduction of Kubernetes and other technologies from the cloud native landscape. A fundamental reason for introducing a cluster orchestration framework like Kubernetes was specifically to enable independent deployment of individual microservices.

In this post I will dive deeper into exactly how we use microservices at Lunar Way.

I will also explain why we have chosen this type of architecture and what it allows us to do.

The post will be a bit like a traveller's report of a journey from a monolithic backend architecture to microservices. Along the way I'll describe some of our discoveries and hopefully there'll also be some takeaways about the best approach and pitfalls that'll be helpful if you're setting out on a similar journey.

### Microservices. What is it?

Before we get to our microservice journey, let's get some terminology defined. What is a microservice architecture in the first place?

There is no precise definition, which everyone in the industry agrees upon, but for the purpose of this post we will use the following characteristics, which hopefully everyone will agree upon is some of the basic characteristics of a microservice architecture:

A microservice architecture is an application design, where a number of independent, small services work together to achieve the purpose of the application. Each service must be independently deployable and have a clear, coherent purpose in the sense that it must be possible to describe it as doing "just one thing". The services must communicate through well defined, simple communication protocols and they should only share data through these protocols.

Typically, a microservice is designed as the owner of a specific data domain and the service implements the business rules and interfaces related to this data domain. This is what is called the "bounded context" of a microservice.

### Event based systems and asynchronous communication

A microservice is never alone. An application is typically made up of tens or hundreds of services, and these services must communicate somehow in order to fulfill the purpose of the application. You can choose to use synchronous communication between services by using some kind of RPC in the broadest sense (think REST, gRPC or a similar technology).

However, no matter which RPC method you choose, the RPC call introduces a tight coupling between the two communicating services, a coupling both at compile time and at runtime. At runtime, the coupling spans both space and time. This means the receiving service must be up and available for the sending service to reach it.

This coupling is bad for many reasons. For example, if service B is down or unreachable, you still want service A to be able to work, although it may not be able to fulfil its purpose completely.

This is why when you say microservice, there's a good chance the next tech buzzword coming out of your mouth will be "event sourcing" or "asynchronous communication by message passing".

If the synchronous communication between services is changed to an asynchronous one, the tight coupling between the services at runtime is removed and reduced to a dependency only involving the data in the message. Service A can broadcast its event to a queue and it does not matter if service B is not immediately available to process it. Eventually, when service B is up, it will process the event and perform whatever action it's designed to carry out.

Asynchronous communication using events opens up a number of other significant possibilities:

#### Open ended communication

Broadcasting messages, i.e. publishing events without an explicit receiver and free for everyone to consume makes it easy to implement new services, which make use of already existing events.

#### Event sourcing

If all business model modifications are broadcast as events and these events are recorded, you get a log of all changes to the system.

Essentially, you get the complete history of the system in a single source.

This is the fundamental idea behind event sourcing, an architecture where all changes to data are stored as events, which may be replayed to get to the current state of the system. This design is often combined with microservices since it also offers a model for how a service may gain knowledge about a business domain outside its own domain — it can simply subscribe to all relevant events and build its own view of that domain.

### What are the challenges?

There's no such thing as a free lunch!

This is also true when it comes to microservices. In many ways, building a functioning microservice architecture is a lot more difficult than building a monolith.

You must consider the following questions when embarking on building a microservice architecture.

#### Data duplication

If service B needs to know something in a domain which is owned by service A, how does it get this knowledge? Should it do a synchronous RPC request to service A each time it requires this knowledge? Or should it instead subscribe to business events from service A and build its own model of the world, as described above?

#### Service domains

How do you divide your services and organize the communication in order to avoid building a distributed monolith with a huge amount of dependencies between services?

#### Debugging and tracing

How do you debug and trace across service boundaries?

#### Asynchronous communication

Adding asynchronous communication to the equation makes it even more complex. How can you be sure events are consumed correctly? In the case where a certain business workflow is implemented across several microservices involving multiple asynchronous events, how can you be certain the flow is completed, that it does not stop somewhere down the chain due to an event being lost?

### What are the benefits?

Why dive into this challenge when it's all so complicated? The answer is very much centred around different aspects of scalability and independence:

#### Runtime/deployment

- In combination with a clever deployment tool, microservices provide the ability to scale services independently of each other. Services with a high load or costly processes can be scaled and deployed independently of other services.
- Microservices have an inbuilt resilience to faults — an error in service A does not prevent service B from functioning.
- Microservices fit nicely into a continuous delivery mindset where services are deployed continuously in order to minimize work in progress.

#### Development

- Microservices allow for autonomous teams which can build and deploy services independently of each other.
- Microservices allow for fast experiments to quickly try out and evaluate new ideas to lower time to market.
- Microservices naturally encourage a system design with high coherence and low coupling, where different parts of the code base only interact through well defined interfaces.

### Lunar Way's microservice architecture

With the terminology, concepts, challenges and benefits well defined, we can continue with Lunar Way's journey towards a microservice architecture.

When I joined the company back in early summer 2016, the backend consisted of a single Rails application. The API exposed by the backend to the app was nicely divided into domains, but the code and data model was tightly coupled. During the summer we decided to embark on rebuilding the backend into a scalable architecture, allowing us to meet the requirements of the business and to be able to deliver new features fast.

There was really no doubt that what we wanted was ultimately to kill the Rails monolith and build a microservice architecture instead.

Furthermore, we wanted as much as possible of the communication in the platform to be asynchronous. At that point we did not have answers to all the challenging questions posed above. However, we thought the benefits of a microservice architecture were too significant to ignore, and hoped we would be able to solve the challenges along the way.

### Our first "microservice"

The first microservice we built was the so called "Feed service". Its sole purpose was to supply the app with a feed of transactions and other items displayed alongside transactions in a chronological order. Up until then, the data model used for the data displayed in the app was very tightly coupled to the data model in the Rails backend. This was an obstacle for data model changes on both sides. It was therefore also a main goal of the feed service to decouple the data model of the app from that of the Rails backend, by letting the feed service implement a new, simple API for the app centred around feed items.

The feed service was supposed to generate feed items based on data received in events published by the Rails monolith.

This was where we made our first rookie mistake

Instead of embedding the complete transaction data for the feed items into these events, they only contained a unique identifier of the transaction, and the feed service would then have to look up the actual transaction in the old Rails database. In hindsight this was a really bad decision, which created a very tight coupling between the feed service and the Rails monolith at the database level. It goes against all the principles of microservices being independent and only sharing data through well defined communication protocols, and certainly not by letting one service read from another service's database.

The main reason behind this bad design choice was to buy ourselves some time. Due to technicalities, which are beyond the scope of this post, it was the easiest solution at the time, although we knew it wasn't the best choice.

We have regretted the shortcut ever since. The time we won back has easily been lost due to the coupling it introduced.

The first key takeaway: Never, ever let one microservice access another service's data directly.

The journey will continue soon in my next post. Stay tuned!