
<Sponsorships>

🎉 [**JetBrains Rider**](https://www.jetbrains.com/rider/?utm_campaign=rider_free&utm_content=site&utm_medium=cpc&utm_source=milan_jovanovic_newsletter)
is now free for non-commercial use.
Great news for all hobbyists, students, content creators, and open source contributors!
Now you can use Rider, a cross-platform .NET and game dev IDE, for non-commercial development for free.
[**Download and start today**](https://www.jetbrains.com/rider/?utm_campaign=rider_free&utm_content=site&utm_medium=cpc&utm_source=milan_jovanovic_newsletter)!

Remember when APIs were an afterthought?
We're [**not going back**](https://shortclick.link/djx5wv).
74% of devs surveyed in the 2024 State of the API Report are API-first, up from 66% in 2023.
What else? We're getting faster. 63% of devs can produce an API within a week, up from 47% in 2023.
Collaboration on APIs is still a challenge.
APIs are increasingly revenue generators.
Read the [**rest of the report**](https://shortclick.link/djx5wv) for fresh insights.

</Sponsorships>

[**Caching**](caching-in-aspnetcore-improving-application-performance) is essential for building fast, scalable applications.
ASP.NET Core has traditionally offered two caching options: in-memory caching and distributed caching.
Each has its trade-offs.
In-memory caching using `IMemoryCache` is fast but limited to a single server.
Distributed caching with `IDistributedCache` works across multiple servers using a backplane.

.NET 9 introduces `HybridCache`, a new library that combines the best of both approaches.
It prevents common caching problems like cache stampede.
It also adds useful features like tag-based invalidation and better performance monitoring.

In this week's issue, I'll show you how to use `HybridCache` in your applications.

## What is HybridCache?

The traditional caching options in ASP.NET Core have limitations.
In-memory caching is fast but limited to one server.
Distributed caching works across servers but is slower.

[HybridCache](https://learn.microsoft.com/en-us/aspnet/core/performance/caching/hybrid) combines both approaches and adds important features:

- Two-level caching (L1/L2)
  - L1: Fast in-memory cache
  - L2: Distributed cache (Redis, SQL Server, etc.)
- Protection against [cache stampede](https://en.wikipedia.org/wiki/Cache_stampede) (when many requests hit an empty cache at once)
- Tag-based cache invalidation
- Configurable serialization
- Metrics and monitoring

The L1 cache runs in your application's memory.
The L2 cache can be Redis, SQL Server, or any other distributed cache.
You can use HybridCache with just the L1 cache if you don't need distributed caching.

## Installing HybridCache

Install the `Microsoft.Extensions.Caching.Hybrid` NuGet package:

```powershell
Install-Package Microsoft.Extensions.Caching.Hybrid
```

Add `HybridCache` to your services:

```csharp
builder.Services.AddHybridCache(options =>
{
    // Maximum size of cached items
    options.MaximumPayloadBytes = 1024 * 1024 * 10; // 10MB
    options.MaximumKeyLength = 512;

    // Default timeouts
    options.DefaultEntryOptions = new HybridCacheEntryOptions
    {
        Expiration = TimeSpan.FromMinutes(30),
        LocalCacheExpiration = TimeSpan.FromMinutes(30)
    };
});
```

For custom types, you can add your own serializer:

```csharp
builder.Services.AddHybridCache()
    .AddSerializer<CustomType, CustomSerializer>();
```

## Using HybridCache

`HybridCache` provides several methods to work with cached data.
The most important ones are `GetOrCreateAsync`, `SetAsync`, and various remove methods.
Let's see how to use each one in real-world scenarios.

### Getting or Creating Cache Entries

The `GetOrCreateAsync` method is your main tool for working with cached data.
It handles both cache hits and misses automatically.
If the data isn't in the cache, it calls your factory method to get the data, caches it, and returns it.

Here's an endpoint that gets product details:

```csharp
app.MapGet("/products/{id}", async (
    int id,
    HybridCache cache,
    ProductDbContext db,
    CancellationToken ct) =>
{
    var product = await cache.GetOrCreateAsync(
        $"product-{id}",
        async token =>
        {
            return await db.Products
                .Include(p => p.Category)
                .FirstOrDefaultAsync(p => p.Id == id, token);
        },
        cancellationToken: ct
    );

    return product is null ? Results.NotFound() : Results.Ok(product);
});
```

In this example:

- The cache key is unique per product
- If the product is in the cache, it's returned immediately
- If not, the factory method runs to get the data
- Other concurrent requests for the same product wait for the first one to finish

### Setting Cache Entries Directly

Sometimes you need to update the cache directly, like after modifying data.
The `SetAsync` method handles this:

```csharp
app.MapPut("/products/{id}", async (int id, Product product, HybridCache cache) =>
{
    // First update the database
    await UpdateProductInDatabase(product);

    // Then update the cache with custom expiration
    var options = new HybridCacheEntryOptions
    {
        Expiration = TimeSpan.FromHours(1),
        LocalCacheExpiration = TimeSpan.FromMinutes(30)
    };

    await cache.SetAsync(
        $"product-{id}",
        product,
        options
    );

    return Results.NoContent();
});
```

Key points about `SetAsync`:

- It updates both L1 and L2 cache
- You can specify different timeouts for L1 and L2
- It overwrites any existing value for the same key

### Using Cache Tags

Tags are powerful for managing groups of related cache entries.
You can invalidate multiple entries at once using tags:

```csharp {7,18,31}
app.MapGet("/categories/{id}/products", async (
    int id,
    HybridCache cache,
    ProductDbContext db,
    CancellationToken ct) =>
{
    var tags = [$"category-{id}", "products"];

    var products = await cache.GetOrCreateAsync(
        $"products-by-category-{id}",
        async token =>
        {
            return await db.Products
                .Where(p => p.CategoryId == id)
                .Include(p => p.Category)
                .ToListAsync(token);
        },
        tags: tags,
        cancellationToken: ct
    );

    return Results.Ok(products);
});

// Endpoint to invalidate all products in a category
app.MapPost("/categories/{id}/invalidate", async (
    int id,
    HybridCache cache,
    CancellationToken ct) =>
{
    await cache.RemoveByTagAsync($"category-{id}", ct);

    return Results.NoContent();
});
```

Tags are useful for:

- Invalidating all products in a category
- Clearing all cached data for a specific user
- Refreshing all related data when something changes

### Removing Single Entries

For direct cache invalidation of specific items, use `RemoveAsync`:

```csharp
app.MapDelete("/products/{id}", async (int id, HybridCache cache) =>
{
    // First delete from database
    await DeleteProductFromDatabase(id);

    // Then remove from cache
    await cache.RemoveAsync($"product-{id}");

    return Results.NoContent();
});
```

`RemoveAsync`:

- Removes the item from both L1 and L2 cache
- Works immediately, no delay
- Does nothing if the key doesn't exist
- Is safe to call multiple times

Remember that `HybridCache` handles all the complexity of distributed caching, serialization, and stampede protection for you.
You just need to focus on your cache keys and when to invalidate the cache.

## Adding Redis as L2 Cache

To use [Redis](https://redis.io/) as your distributed cache:

1. Install the `Microsoft.Extensions.Caching.StackExchangeRedis` NuGet package:

```powershell
Install-Package Microsoft.Extensions.Caching.StackExchangeRedis
```

2. Configure Redis and `HybridCache`:

```csharp
// Add Redis
builder.Services.AddStackExchangeRedisCache(options =>
{
    options.Configuration = "your-redis-connection-string";
});

// Add HybridCache - it will automatically use Redis as L2
builder.Services.AddHybridCache();
```

`HybridCache` will automatically detect and use Redis as the L2 cache.

## Summary

`HybridCache` simplifies caching in .NET applications.
It combines fast in-memory caching with distributed caching, prevents common problems like cache stampede,
and works well in both single-server and distributed systems.

Start with the default settings and basic usage patterns - the library is designed to be simple to use while solving complex caching problems.

Thanks for reading.

And stay awesome!

---



<Sponsorships>

You know that feeling when you have an idea in your head but can't think of the word for it?
Or you have documents in MongoDB Atlas but not the exact text to search them?
[**Vector search**](https://shortclick.link/1oo137) might be the answer.
Adding MongoDB Atlas Vector Search is pretty [**straightforward**](https://shortclick.link/1oo137), and can come in handy.

Auto-Generate [**enterprise-grade C# SDKs**](https://www.speakeasy.com/docs/sdk-design/csharp/oss-comparison-csharp?utm_source=net-weekly)
from OpenAPI specs.
Create customizable, strongly-typed C# SDKs with Speakeasy's SDK generator - designed for enterprise APIs where open-source solutions fall short.
Compare our SDK generator to popular OSS alternatives and [**create your first SDK for free**](https://www.speakeasy.com/docs/sdk-design/csharp/oss-comparison-csharp?utm_source=net-weekly).

</Sponsorships>

Functional programming patterns can feel academic and abstract.
Terms like "monads" and "functors" scare many developers away.
But beneath the intimidating terminology are practical patterns that can make your code safer and more maintainable.

C# has embraced many functional programming features over the years.

- Records for immutability
- LINQ for functional transformations
- Lambda expressions for first-class functions

These features aren't just syntax sugar - they help prevent bugs and make code easier to reason about.

Let's look at five practical patterns you can use in your C# projects today.

## Higher-Order Functions

Higher-order functions can take other functions as parameters or return them as results.
They let you write code that's more flexible and composable because you can pass behavior around like data.

Common examples of higher-order functions are LINQ's `Where` and `Select`, which take functions to transform data.

Let's refactor this validation example with higher-order functions:

```csharp
public class OrderValidator
{
    public bool ValidateOrder(Order order)
    {
        if (order.Items.Count == 0) return false;
        if (order.TotalAmount <= 0) return false;
        if (order.ShippingAddress == null) return false;
        return true;
    }
}

// What if we need:
// - different validation rules for different countries?
// - to reuse some validations but not others?
// - to combine validations differently?
```

Here's how higher-order functions make this more flexible:

```csharp
public static class OrderValidation
{
    public static Func<Order, bool> CreateValidator(string countryCode, decimal minimumOrderValue)
    {
        var baseValidations = CombineValidations(
            o => o.Items.Count > 0,
            o => o.TotalAmount >= minimumOrderValue,
            o => o.ShippingAddress != null
        );

        return countryCode switch
        {
            "US" => CombineValidations(
                baseValidations,
                order => IsValidUSAddress(order.ShippingAddress)),
            "EU" => CombineValidations(
                baseValidations,
                order => IsValidVATNumber(order.VatNumber)),
            _ => baseValidations
        };
    }

    private static Func<Order, bool> CombineValidations(params Func<Order, bool>[] validations) =>
        order => validations.All(v => v(order));
}

// Usage
var usValidator = OrderValidation.CreateValidator("US", minimumOrderValue: 25.0m);
var euValidator = OrderValidation.CreateValidator("EU", minimumOrderValue: 30.0m);
```

The higher-order function approach makes validators composable, testable, and easy to extend.
Each validation rule is a simple function that we can compose.

## Errors as Values

Error handling in C# often looks like this:

```csharp
public class UserService
{
    public User CreateUser(string email, string password)
    {
        if (string.IsNullOrEmpty(email))
        {
            throw new ArgumentException("Email is required");
        }

        if (password.Length < 8)
        {
            throw new ArgumentException("Password too short");
        }

        if (_userRepository.EmailExists(email))
        {
            throw new DuplicateEmailException(email);
        }

        // Create user...
    }
}
```

The problem?

- [Exceptions are expensive](https://youtu.be/E3dU9Y1CsnI)
- Callers often forget to handle exceptions
- The method signature lies - it claims to return a User but might throw

We can make errors explicit using the [OneOf](https://github.com/mcintyre321/OneOf) library.
It provides discriminated unions for C#, using a custom type `OneOf<T0, ... Tn>`.

```csharp
public class UserService
{
    public OneOf<User, ValidationError, DuplicateEmailError> CreateUser(string email, string password)
    {
        if (string.IsNullOrEmpty(email))
        {
            return new ValidationError("Email is required");
        }

        if (password.Length < 8)
        {
            return new ValidationError("Password too short");
        }

        if (_userRepository.EmailExists(email))
        {
            return new DuplicateEmailError(email);
        }

        return new User(email, password);
    }
}
```

By making the errors explicit:

- The method signature tells the whole truth
- Callers must handle all possible outcomes
- No performance overhead from exceptions
- The flow is easier to follow

Here's how you use it:

```csharp
var result = userService.CreateUser(email, password);

result.Switch(
    user => SendWelcomeEmail(user),
    validationError => HandleError(validationError),
    duplicateError => HandleError(duplicateError)
);
```

## Monadic Binding

A **monad** is a container for values - like `List<T>`, `IEnumerable<T>`, or `Task<T>`.
What makes it special is that you can chain operations on the contained values without dealing with the container directly.
This chaining is called monadic binding.

You use monadic binding daily with LINQ, but you might not know it.
It's what allows us to chain operations that transform data.

Map (`Select`) transforms values:

```csharp
// Simple transformations with Select (Map)
var numbers = new[] { 1, 2, 3, 4 };

var doubled = numbers.Select(x => x * 2);
```

Bind (`SelectMany`) transforms and flattens:

```csharp
// Operations that return multiple values use SelectMany (Bind)
var folders = new[] { "docs", "photos" };

var files = folders.SelectMany(folder => Directory.GetFiles(folder));
```

A popular example of applying monads in practice is the [**Result pattern**](functional-error-handling-in-dotnet-with-the-result-pattern),
which provides a clean way to chain operations that might fail.

## Pure Functions

Pure functions are predictable: they depend only on their inputs and don't change anything in the system.
No database calls, no API requests, no global state.
This constraint makes them easier to understand, test, and debug.

```csharp
// Impure - relies on hidden state
public class PriceCalculator
{
    private decimal _taxRate;
    private List<Discount> _activeDiscounts;

    public decimal CalculatePrice(Order order)
    {
        var price = order.Items.Sum(i => i.Price);

        foreach (var discount in _activeDiscounts)
        {
            price -= discount.Calculate(price);
        }

        return price * (1 + _taxRate);
    }
}
```

Here's the same example as a pure function:

```csharp
// Pure - everything is explicit
public static class PriceCalculator
{
    public static decimal CalculatePrice(
        Order order,
        decimal taxRate,
        IReadOnlyList<Discount> discounts)
    {
        var basePrice = order.Items.Sum(i => i.Price);

        var afterDiscounts = discounts.Aggregate(
            basePrice,
            (price, discount) => price - discount.Calculate(price));

        return afterDiscounts * (1 + taxRate);
    }
}
```

Pure functions are thread-safe, easy to test, and simple to reason about because all dependencies are explicit.

## Immutability

Immutable objects can't be changed after creation.
Instead, they create new instances for every change.
This simple constraint eliminates entire categories of bugs: race conditions, accidental modifications, and inconsistent state.

Here's an example of a mutable type:

```csharp
public class Order
{
    public List<OrderItem> Items { get; set; }
    public decimal Total { get; set; }
    public OrderStatus Status { get; set; }

    public void AddItem(OrderItem item)
    {
        Items.Add(item);
        Total += item.Price;
        // Bug: Thread safety issues
        // Bug: Can modify shipped orders
        // Bug: Total might not match Items
    }
}
```

Let's make this an immutable type:

```csharp
public record Order
{
    public ImmutableList<OrderItem> Items { get; init; }
    public OrderStatus Status { get; init; }
    public decimal Total => Items.Sum(x => x.Price);

    public Order AddItem(OrderItem item)
    {
        if (Status != OrderStatus.Created)
        {
            throw new InvalidOperationException("Can't modify shipped orders");
        }

        return this with
        {
            Items = Items.Add(item)
        };
    }
}
```

The immutable version:

- Is thread-safe by default
- Makes invalid states impossible
- Keeps data and calculations consistent
- Makes changes explicit and traceable

## Takeaway

[**Functional programming**](how-to-apply-functional-programming-in-csharp) isn't just about writing "cleaner" code.
These patterns fundamentally change how you handle complexity:

- **Push errors to compile time** - Catch problems before running the code
- **Make invalid states impossible** - Don't rely on documentation or conventions
- **Make the happy path obvious** - When everything is explicit, the flow is clear

You can adopt these patterns gradually.
Start with one class, one module, one feature.
The goal isn't to write purely functional code.
The goal is to write code that's safer, more predictable, and easier to maintain.

Hope this was helpful.
See you next week.

---



<Sponsorships>

Remember when APIs were an afterthought?
We're [**not going back**](https://shortclick.link/djx5wv).
74% of devs surveyed in the 2024 State of the API Report are API-first, up from 66% in 2023.
What else? We're getting faster. 63% of devs can produce an API within a week, up from 47% in 2023.
Collaboration on APIs is still a challenge.
APIs are increasingly revenue generators.
Read the [**rest of the report**](https://shortclick.link/djx5wv) for fresh insights.

New Report: [**State of Designer-Developer Collaboration 2024**](https://www.telerik.com/design-system/designer-developer-collaboration-survey-2024/?utm_medium=cpm&utm_source=milanjovanovich&utm_campaign=dt_survey_design_story_dev_collab_report).
Whether you're on the development or design end of creating web applications, it's easy to lose track of the world beyond your daily work.
This brand-new report will give you some interesting insights and in-depth analysis on the relationship between designers and developers,
including excellent ideas for making it more efficient and satisfying.
[**Check it out**](https://www.telerik.com/design-system/designer-developer-collaboration-survey-2024/?utm_medium=cpm&utm_source=milanjovanovich&utm_campaign=dt_survey_design_story_dev_collab_report).

</Sponsorships>

I see the same mistake happen over and over again.

Developers discover Clean Architecture, get excited about its principles, and then... they turn the famous Clean Architecture diagram into a project structure.

But here's the thing: **Clean Architecture is not about folders**.
It's about dependencies.

Simon Brown wrote a "missing chapter" for Uncle Bob's Clean Architecture book that addresses exactly this issue.
Yet somehow, this crucial message got lost along the way.

Today, I'll show you what Uncle Bob's Clean Architecture diagram really means and how you should actually organize your code.
We'll look at practical examples that you can use in your projects right now.

Let's clear up this common misconception once and for all.

## The Problem With Traditional Layering

Almost every .NET developer has built a solution that looks like this:

- `MyApp.Web` for controllers and views
- `MyApp.Business` for services and business logic
- `MyApp.Data` for repositories and data access

It's the default approach. It's what we see in tutorials. It's what we teach juniors.

And it's completely wrong.

### Why Layer-Based Organization Fails

When you organize code by technical layers, you scatter related components across multiple projects.
A single feature, like managing policies, ends up spread across your entire codebase:

- Policies controller in the Web layer
- Policy service in the Business layer
- Policy repository in the Data layer

Here's what you'll see when looking at the folder structure:

```text
📁 MyApp.Web
|__ 📁 Controllers
    |__ #️⃣ PoliciesController.cs
📁 MyApp.Business
|__ 📁 Services
    |__ #️⃣ PolicyService.cs
📁 MyApp.Data
|__ 📁 Repositories
    |__ #️⃣ PolicyRepository.cs
```

Here's a visual representation of the layer-based architecture:

<div className="centered">
  <Image
    src="blogs/mnw_114/layered_architecture.png"
    width={490 * 0.5}
    height={1247 * 0.5}
    alt="Feature Scattering in Layer-Based Architecture."
  />
</div>

This fragmentation creates several problems:

1. **Violates Common Closure Principle** - Classes that change together should stay together.
   When your "Policies" feature changes, you're touching three different projects.

2. **Hidden dependencies** - Public interfaces everywhere make it possible to bypass layers.
   Nothing stops a controller from directly accessing a repository.

3. **No business intent** - Opening your solution tells you nothing about what the application does.
   It only shows technical implementation details.

4. **Harder maintenance** - Making changes requires jumping between multiple projects.

The worst part? This approach doesn't even achieve what it promises.
Despite the separate projects, you often end up with a "big ball of mud" because public access modifiers allow any class to reference any other class.

### The Real Intent of Layers

Clean Architecture's circles were never meant to represent projects or folders.
They represent different levels of policy, with dependencies pointing inward toward business rules.

You can achieve this without splitting your code into artificial technical layers.

Let me show you a better way.

## Better Approaches to Code Organization

Instead of splitting your code by technical layers, you have two better options: **package by feature** or **package by component**.

Let's look at both.

### Package by Feature

Organizing by feature is a solid option.
Each feature gets its own namespace and contains everything needed to implement that feature.

```text
📁 MyApp.Policies
|__ 📁 RenewPolicy
    |__ #️⃣ RenewPolicyCommand.cs
    |__ #️⃣ RenewPolicyHandler.cs
    |__ #️⃣ PolicyValidator.cs
    |__ #️⃣ PolicyRepository.cs
|__ 📁 ViewPolicyHistory
    |__ #️⃣ PolicyHistoryQuery.cs
    |__ #️⃣ PolicyHistoryHandler.cs
    |__ #️⃣ PolicyHistoryViewModel.cs
```

Here's a diagram representing this structure:

<div className="centered">
  <Image
    src="blogs/mnw_114/feature_folder_architecture.png"
    width={490 * 0.5}
    height={1247 * 0.5}
    alt="Vertical slice architecture for package by feature organization."
  />
</div>

This approach:

- Makes features explicit
- Keeps related code together
- Simplifies navigation
- Makes it easier to maintain and modify features

If you want to learn more, check out my article about [**vertical slice architecture**](vertical-slice-architecture).

### Package by Component

A component is a cohesive group of related functionality with a well-defined interface.
Component-based organization is more coarse-grained than feature folders.
Think of it as a mini application that handles one specific business capability.

This is very similar to how I define modules in a [**modular monolith**](what-is-a-modular-monolith).

Here's what a component-based organization looks like:

```text
📁 MyApp.Web
|__ 📁 Controllers
    |__ #️⃣ PoliciesController.cs
📁 MyApp.Policies
|__ #️⃣ PoliciesComponent.cs     // Public interface
|__ #️⃣ PolicyService.cs         // Implementation detail
|__ #️⃣ PolicyRepository.cs      // Implementation detail
```

The key difference? Only `PoliciesComponent` is public.
Everything else is internal to the component.

<div className="centered">
  <Image
    src="blogs/mnw_114/component_architecture.png"
    width={491 * 0.5}
    height={1247 * 0.5}
    alt="Feature Scattering in Layer-Based Architecture."
  />
</div>

This means:

- No bypassing layers
- Clear dependencies
- Real encapsulation
- Business intent visible in the structure

### Which One Should You Choose?

Choose **Package by Feature** when:

- You have many small, independent features
- Your features don't share much code
- You want maximum flexibility

Choose **Package by Component** when:

- You have clear business capabilities
- You want strong encapsulation
- You might split into microservices later

Both approaches achieve what Clean Architecture really wants: proper dependency management and business focus.

Here's a side-by-side comparison of these architectural approaches:

<figure>
  <div className="centered">
    <Image
      src="blogs/mnw_114/architecture_comparison.png"
      width={1934}
      height={1247}
      alt="Comparison between layered, vertical slice and component architectural approaches."
    />
  </div>
  <figcaption>
    Greyed-out types are internal to the defining assembly.
  </figcaption>
</figure>

In the Missing Chapter of Clean Architecture, Simon Brown argues strongly for package by component.
The key insight is that components are the natural way to slice a system.
They represent complete business capabilities, not just technical features.

My recommendation?
Start with package by component.
Within the component, organize around features.

## Practical Examples

Let's transform a typical layered application into a clean, component-based structure.
We'll use an insurance policy system as an example.

### The Traditional Way

Here's how most developers structure their solution:

```csharp
// MyApp.Data
public interface IPolicyRepository
{
    Task<Policy> GetByIdAsync(string policyNumber);
    Task SaveAsync(Policy policy);
}

// MyApp.Business
public class PolicyService : IPolicyService
{
    private readonly IPolicyRepository _repository;

    public PolicyService(IPolicyRepository repository)
    {
        _repository = repository;
    }

    public async Task RenewPolicyAsync(string policyNumber)
    {
        var policy = await _repository.GetByIdAsync(policyNumber);
        // Business logic here
        await _repository.SaveAsync(policy);
    }
}

// MyApp.Web
public class PoliciesController : ControllerBase
{
    private readonly IPolicyService _policyService;

    public PoliciesController(IPolicyService policyService)
    {
        _policyService = policyService;
    }

    [HttpPost("renew/{policyNumber}")]
    public async Task<IActionResult> RenewPolicy(string policyNumber)
    {
        await _policyService.RenewPolicyAsync(policyNumber);
        return Ok();
    }
}
```

The problem?
Everything is public.
Any class can bypass the service and go straight to the repository.

### The Clean Way

Here's the same functionality organized as a proper component:

```csharp
// The only public contract
public interface IPoliciesComponent
{
    Task RenewPolicyAsync(string policyNumber);
}

// Everything below is internal to the component
internal class PoliciesComponent : IPoliciesComponent
{
    private readonly IRenewPolicyHandler _renewPolicyHandler;

    // Public constructor for DI
    public PoliciesComponent(IRenewPolicyHandler renewPolicyHandler)
    {
        _renewPolicyHandler = renewPolicyHandler;
    }

    public async Task RenewPolicyAsync(string policyNumber)
    {
        await _renewPolicyHandler.HandleAsync(policyNumber);
    }
}

internal interface IRenewPolicyHandler
{
    Task HandleAsync(string policyNumber);
}

internal class RenewPolicyHandler : IRenewPolicyHandler
{
    private readonly IPolicyRepository _repository;

    internal RenewPolicyHandler(IPolicyRepository repository)
    {
        _repository = repository;
    }

    public async Task HandleAsync(string policyNumber)
    {
        var policy = await _repository.GetByIdAsync(policyNumber);
        // Business logic for policy renewal here
        await _repository.SaveAsync(policy);
    }
}

internal interface IPolicyRepository
{
    Task<Policy> GetByIdAsync(string policyNumber);
    Task SaveAsync(Policy policy);
}
```

The key improvements are:

1. **Single public interface** - Only `IPoliciesComponent` is public. Everything else is internal.

2. **Protected dependencies** - No way to bypass the component and access the repository directly.

3. **Clear dependencies** - All dependencies flow inward through the component.

4. **Proper encapsulation** - Implementation details are truly hidden.

This is how you would register the services with dependency injection:

```csharp
services.AddScoped<IPoliciesComponent, PoliciesComponent>();
services.AddScoped<IRenewPolicyHandler, RenewPolicyHandler>();
services.AddScoped<IPolicyRepository, SqlPolicyRepository>();
```

This structure enforces Clean Architecture principles through compiler-checked boundaries, not just conventions.

The compiler won't let you bypass the component's public interface.
That's much stronger than hoping developers follow the rules.

## Best Practices and Limitations

Let's discuss something that is often overlooked: the practical limitations of enforcing Clean Architecture in .NET.

### The Limits of Encapsulation

The `internal` keyword in .NET provides protection within a single assembly.
Here's what that means in practice:

```csharp
// In a single project:
public interface IPoliciesComponent { } // Public contract
internal class PoliciesComponent : IPoliciesComponent { }
internal class PolicyRepository { }

// Someone could still do this:
public class BadPoliciesComponent : IPoliciesComponent
{
    public BadPoliciesComponent()
    {
        // Nothing stops them from creating a bad implementation
    }
}
```

While `internal` helps, it doesn't prevent all architectural violations.

### The Trade-offs

Some teams split their code into separate assemblies for stronger encapsulation:

```plaintext
MyCompany.Policies.Core.dll
MyCompany.Policies.Infrastructure.dll
MyCompany.Policies.Api.dll
```

This comes with trade-offs:

1. **More complex build process** - Multiple projects need to be compiled and referenced.
2. **Harder navigation** - Jumping between assemblies in the IDE is slower.
3. **Deployment complexity** - More DLLs to manage and deploy.

### A Pragmatic Approach

Here's what I recommend:

1. **Use a single assembly**

   - Keep related code together
   - Use `internal` for implementation details
   - Make only the component interfaces public
   - Add `sealed` to prevent inheritance when possible

2. **Enforce through architecture testing**

   - Add architecture tests to verify dependencies
   - Automatically check for architectural violations
   - Fail the build if someone bypasses the rules

```csharp
[Fact]
public void Controllers_Should_Only_Depend_On_Component_Interfaces()
{
    var result = Types.InAssembly(Assembly.GetExecutingAssembly())
        .That()
        .ResideInNamespace("MyApp.Controllers")
        .Should()
        .OnlyDependOn(type =>
            type.Name.EndsWith("Component") ||
            type.Name.StartsWith("IPolicy"))
        .GetResult();

    result.IsSuccessful.Should().BeTrue();
}
```

Want to learn more about enforcing architecture through testing?
Check out my article on [**architecture testing**](enforcing-software-architecture-with-architecture-tests).

Remember: Clean Architecture is about managing dependencies, not about achieving perfect encapsulation.
Use the tools the language gives you, but don't over-complicate things chasing an impossible ideal.

## Conclusion

Clean Architecture isn't about projects, folders, or perfect encapsulation.

It's about:

- Organizing code around business capabilities
- Managing dependencies effectively
- Keeping related code together
- Making boundaries explicit

Start with a single project.
Use components.
Make interfaces public and implementations internal.
Add architecture tests if you need more control.

And remember: **pragmatism beats purism**.
Your architecture should help you ship features faster, not slow you down with artificial constraints.

Want to learn more?
Check out my [**Pragmatic Clean Architecture**](/pragmatic-clean-architecture) course,
where I'll show you how to build maintainable applications with proper boundaries, clear dependencies, and business-focused components.

That's all for today. Stay awesome, and I'll see you next week.

---



<Sponsorships>

Looking for a developer-friendly, affordable eSignature solution?
[**BoldSign**](https://boldsign.com/?utm_source=milanjovanovic&utm_medium=newsletter_ad&utm_campaign=boldsign) offers highly responsive and secure API integration for your .NET applications with transparent pricing—making it the perfect
[**alternative to DocuSign**](https://boldsign.com/docusign-alternative/?utm_source=milanjovanovic&utm_medium=newsletter_ad&utm_campaign=boldsign).
Start your free trial today.

Transform your team's API development.
See how [**Postman Flows**](https://app.zuddl.com/p/a/event/b718e16b-9456-4da3-9af1-f990b252ceb5?utm_source=influencer&utm_medium=Newsletter&utm_campaign=Flows_collab&utm_term=Milan_Jovanovic&utm_content=Webinar)
eliminates bottlenecks & enables seamless collaboration.
Join the finale of our "Visualize, Accelerate, & Collaborate" series to supercharge your workflow.
[**Register now**](https://app.zuddl.com/p/a/event/b718e16b-9456-4da3-9af1-f990b252ceb5?utm_source=influencer&utm_medium=Newsletter&utm_campaign=Flows_collab&utm_term=Milan_Jovanovic&utm_content=Webinar).

</Sponsorships>

Idempotency is a crucial concept for REST APIs that ensures the reliability and consistency of your system.
An idempotent operation can be repeated multiple times without changing the result beyond the initial API request.
This property is especially important in distributed systems, where network failures or timeouts can lead to repeated requests.

Implementing idempotency in your API brings several benefits:

- It prevents unintended duplicate operations
- It improves reliability in distributed systems
- It helps handle network issues and retries gracefully

In this week's issue, we'll explore how to implement idempotency in ASP.NET Core APIs, ensuring your system remains robust and reliable.

## What is Idempotence?

Idempotence, in the context of web APIs, means that making multiple identical requests should have the same effect as making a single request.
In other words, no matter how many times a client sends the same request, the server-side effect should only occur once.

The [RFC 9110](https://www.rfc-editor.org/rfc/rfc9110) standard about HTTP Semantics offers a definition we could use.
Here's what it says about **idempotent methods**:

<figure>

> A request method is considered "idempotent" if the intended effect on the server of multiple identical requests with
> that method is the same as the effect for a single such request.
>
> Of the request methods defined by this specification,
> PUT, DELETE, and safe request methods [(GET, HEAD, OPTIONS, and TRACE) - author's note] are idempotent.

  <figcaption>
    _&mdash; [RFC 9110 (HTTP Semantics), Section 9.2.2, Paragraph 1](https://www.rfc-editor.org/rfc/rfc9110#section-9.2.2-1)_
  </figcaption>
</figure>

However, the following paragraph is quite interesting.
It clarifies that the server can implement "other non-idempotent side effects" that don't apply to the resource.

<figure>

> ... the idempotent property only applies to what has been requested by the user;
> a server is free to log each request separately, retain a revision control history, or implement other non-idempotent side effects for each idempotent request.

  <figcaption>
    _&mdash; [RFC 9110 (HTTP Semantics), Section 9.2.2, Paragraph 2](https://www.rfc-editor.org/rfc/rfc9110#section-9.2.2-2)_
  </figcaption>
</figure>

The benefits of implementing idempotency extend beyond just adhering to HTTP method semantics.
It significantly improves the reliability of your API, especially in distributed systems where network issues can lead to retried requests.
By implementing idempotency, you prevent duplicate operations that could occur due to client retries.

## Which HTTP Methods are Idempotent?

Several HTTP methods are inherently idempotent:

- `GET`, `HEAD`: Retrieve data without modifying the server state.
- `PUT`: Update a resource, resulting in the same state regardless of repetition.
- `DELETE`: Remove a resource with the same outcome for multiple requests.
- `OPTIONS`: Retrieve communication options information.

`POST` is not inherently idempotent, as it typically creates resources or processes data.
Repeated `POST` requests could create multiple resources or trigger multiple actions.

However, we can implement idempotency for `POST` methods using custom logic.

**Note**: While `POST` requests aren't naturally idempotent, we can design them to be.
For example, checking for existing resources before creation ensures that repeated `POST` requests don't result in duplicate actions or resources.

## Implementing Idempotency in ASP.NET Core

To implement idempotency, we'll use a strategy involving idempotency keys:

1. The client generates a unique key for each operation and sends it in a custom header.
2. The server checks if it has seen this key before:
   - For a new key, process the request and store the result.
   - For a known key, return the stored result without reprocessing.

This ensures that retried requests (e.g., due to network issues) are processed only once on the server.

We can implement idempotency for controllers by combining an `Attribute` and `IAsyncActionFilter`.
Now, we can specify the `IdempotentAttribute` to apply idempotency to a controller endpoint.

**Note**: When a request fails (returns 4xx/5xx), we don't cache the response.
This allows clients to retry with the same idempotency key.
However, this means a failed request followed by a successful one with the same key will succeed - make sure this aligns with your business requirements.

```csharp
[AttributeUsage(AttributeTargets.Method)]
internal sealed class IdempotentAttribute : Attribute, IAsyncActionFilter
{
    private const int DefaultCacheTimeInMinutes = 60;
    private readonly TimeSpan _cacheDuration;

    public IdempotentAttribute(int cacheTimeInMinutes = DefaultCacheTimeInMinutes)
    {
        _cacheDuration = TimeSpan.FromMinutes(minutes);
    }

    public async Task OnActionExecutionAsync(
        ActionExecutingContext context,
        ActionExecutionDelegate next)
    {
        // Parse the Idempotence-Key header from the request
        if (!context.HttpContext.Request.Headers.TryGetValue(
                "Idempotence-Key",
                out StringValues idempotenceKeyValue) ||
            !Guid.TryParse(idempotenceKeyValue, out Guid idempotenceKey))
        {
            context.Result = new BadRequestObjectResult("Invalid or missing Idempotence-Key header");
            return;
        }

        IDistributedCache cache = context.HttpContext
            .RequestServices.GetRequiredService<IDistributedCache>();

        // Check if we already processed this request and return a cached response (if it exists)
        string cacheKey = $"Idempotent_{idempotenceKey}";
        string? cachedResult = await cache.GetStringAsync(cacheKey);
        if (cachedResult is not null)
        {
            IdempotentResponse response = JsonSerializer.Deserialize<IdempotentResponse>(cachedResult)!;

            var result = new ObjectResult(response.Value) { StatusCode = response.StatusCode };
            context.Result = result;

            return;
        }

        // Execute the request and cache the response for the specified duration
        ActionExecutedContext executedContext = await next();

        if (executedContext.Result is ObjectResult { StatusCode: >= 200 and < 300 } objectResult)
        {
            int statusCode = objectResult.StatusCode ?? StatusCodes.Status200OK;
            IdempotentResponse response = new(statusCode, objectResult.Value);

            await cache.SetStringAsync(
                cacheKey,
                JsonSerializer.Serialize(response),
                new DistributedCacheEntryOptions { AbsoluteExpirationRelativeToNow = _cacheDuration }
            );
        }
    }
}

internal sealed class IdempotentResponse
{
    [JsonConstructor]
    public IdempotentResponse(int statusCode, object? value)
    {
        StatusCode = statusCode;
        Value = value;
    }

    public int StatusCode { get; }
    public object? Value { get; }
}
```

**Note**: There's a small [**race condition**](solving-race-conditions-with-ef-core-optimistic-locking) window between checking and setting the cache.
For absolute consistency, we should consider using a distributed lock pattern, though this adds complexity and latency.

Now, we can apply this attribute to our controller actions:

```csharp {6}
[ApiController]
[Route("api/[controller]")]
public class OrdersController : ControllerBase
{
    [HttpPost]
    [Idempotent(cacheTimeInMinutes: 60)]
    public IActionResult CreateOrder([FromBody] CreateOrderRequest request)
    {
        // Process the order...

        return CreatedAtAction(nameof(GetOrder), new { id = orderDto.Id }, orderDto);
    }
}
```

**Idempotency with Minimal APIs**

To implement idempotency with Minimal APIs, we can use an `IEndpointFilter`.

```csharp
internal sealed class IdempotencyFilter(int cacheTimeInMinutes = 60)
    : IEndpointFilter
{
    public async ValueTask<object?> InvokeAsync(
        EndpointFilterInvocationContext context,
        EndpointFilterDelegate next)
    {
        // Parse the Idempotence-Key header from the request
        if (TryGetIdempotenceKey(out Guid idempotenceKey))
        {
            return Results.BadRequest("Invalid or missing Idempotence-Key header");
        }

        IDistributedCache cache = context.HttpContext
            .RequestServices.GetRequiredService<IDistributedCache>();

        // Check if we already processed this request and return a cached response (if it exists)
        string cacheKey = $"Idempotent_{idempotenceKey}";
        string? cachedResult = await cache.GetStringAsync(cacheKey);
        if (cachedResult is not null)
        {
            IdempotentResponse response = JsonSerializer.Deserialize<IdempotentResponse>(cachedResult)!;
            return new IdempotentResult(response.StatusCode, response.Value);
        }

        object? result = await next(context);

        // Execute the request and cache the response for the specified duration
        if (result is IStatusCodeHttpResult { StatusCode: >= 200 and < 300 } statusCodeResult
            and IValueHttpResult valueResult)
        {
            int statusCode = statusCodeResult.StatusCode ?? StatusCodes.Status200OK;
            IdempotentResponse response = new(statusCode, valueResult.Value);

            await cache.SetStringAsync(
                cacheKey,
                JsonSerializer.Serialize(response),
                new DistributedCacheEntryOptions
                {
                    AbsoluteExpirationRelativeToNow = TimeSpan.FromMinutes(cacheTimeInMinutes)
                }
            );
        }

        return result;
    }
}

// We have to implement a custom result to write the status code
internal sealed class IdempotentResult : IResult
{
    private readonly int _statusCode;
    private readonly object? _value;

    public IdempotentResult(int statusCode, object? value)
    {
        _statusCode = statusCode;
        _value = value;
    }

    public Task ExecuteAsync(HttpContext httpContext)
    {
        httpContext.Response.StatusCode = _statusCode;

        return httpContext.Response.WriteAsJsonAsync(_value);
    }
}
```

Now, we can apply this endpoint filter to our Minimal API endpoint:

```csharp {4}
app.MapPost("/api/orders", CreateOrder)
    .RequireAuthorization()
    .WithOpenApi()
    .AddEndpointFilter<IdempotencyFilter>();
```

An alternative to the previous two implementations is implementing idempotency logic in a custom middleware.

## Best Practices and Considerations

Here are the key things I always keep in mind when implementing idempotency.

Cache duration is tricky.
I aim to cover reasonable retry windows without holding onto stale data.
A reasonable cache time typically ranges from a few minutes to 24-48 hours, depending on your specific use case.

Concurrency can be a pain, especially in high-traffic APIs.
A thread-safe implementation using a distributed lock works great.
It keeps things in check when multiple requests hit at once.
But this should be a rare occurrence.

For distributed setups, Redis is my go-to.
It's perfect as a shared cache, keeping idempotency consistent across all your API instances.
Plus, it handles distributed locking.

What if a client reuses an idempotency key with a different request body?
I return an error in this case.
My approach is to hash the request body and store it with the idempotency key.
When a request comes in, I compare the request body hashes.
If they differ, I return an error.
This prevents misuse of idempotency keys and maintains the integrity of your API.

## Summary

Implementing idempotency in REST APIs enhances service reliability and consistency.
It ensures identical requests yield the same result, preventing unintended duplicates and gracefully handling network issues.

While our implementation provides a foundation, I recommend adapting it to your needs.
Focus on critical operations in your APIs, especially those that modify the system state or trigger important business processes.

By embracing idempotency, you're building more robust and user-friendly APIs.

That's all for today.

See you next week.

---



<Sponsorships>

[**Telerik UI for ASP.NET Core Grid**](https://www.telerik.com/blogs/editing-data-telerik-ui-aspnet-core-grid-no-code-required?utm_medium=cpm&utm_source=milanjovanovic&utm_campaign=core-general-awareness):
Enable data editing without code.
Multiple options for customizing user experience.
Configure the grid easily to meet your UX needs.
[**Learn more here**](https://www.telerik.com/blogs/editing-data-telerik-ui-aspnet-core-grid-no-code-required?utm_medium=cpm&utm_source=milanjovanovic&utm_campaign=core-general-awareness).

Webinar: [**Accelerating with AI using Postman Flows**](https://app.zuddl.com/p/a/event/aa49e0b7-9b7b-43d2-afce-282fcf08a756?utm_source=influencer&utm_medium=Newsletter&utm_campaign=Flows_AI&utm_term=Milan_Jovanovic&utm_content=Webinar).
Second in our series on visualizing, accelerating, and collaborating with Postman Flows.
Learn how AI enhances API development and speeds up your workflow.
[**Register here for free**](https://app.zuddl.com/p/a/event/aa49e0b7-9b7b-43d2-afce-282fcf08a756?utm_source=influencer&utm_medium=Newsletter&utm_campaign=Flows_AI&utm_term=Milan_Jovanovic&utm_content=Webinar).

</Sponsorships>

When developing HTTP APIs, providing consistent and informative error responses is crucial for a smooth developer experience.
**Problem Details** in ASP.NET Core offers a standardized solution to this challenge, ensuring your APIs communicate errors effectively and uniformly.

In this article, we'll explore the latest developments in **Problem Details**, including:

- The new [RFC 9457](https://www.rfc-editor.org/rfc/rfc9457) that refines the Problem Details standard
- Using the .NET 8 `IExceptionHandler` for global exception handling
- Using the `IProblemDetailsService` for customizing Problem Details

Let's dive into these features and see how they can improve your API's error handling.

## Understanding Problem Details

Problem Details is a machine-readable format for specifying errors in HTTP API responses.
HTTP status codes don't always contain enough details about errors to be helpful.
The Problem Details specification defines a JSON (and XML) document format to describe problems.

Problem Details includes:

- `type`: A URI reference that identifies the problem type
- `title`: A short, human-readable summary of the problem type
- `status`: The HTTP status code
- `detail`: A human-readable explanation specific to this occurrence of the problem
- `instance`: A URI reference that identifies the specific occurrence of the problem

[RFC 9457](https://www.rfc-editor.org/rfc/rfc9457), which replaces [RFC 7807](https://www.rfc-editor.org/rfc/rfc7807),
introduces improvements such as clarifying the use of the type field and providing guidelines for extending **Problem Details**.

Here's an example Problem Details response:

```json
Content-Type: application/problem+json

{
  "type": "https://tools.ietf.org/html/rfc9110#section-15.5.5",
  "title": "Not Found",
  "status": 404,
  "detail": "The habit with the specified identifier was not found",
  "instance": "PUT /api/habits/aadcad3f-8dc8-443d-be44-3d99893ba18a"
}
```

## Implementing Problem Details

Let's see how to implement Problem Details in ASP.NET Core.
We want to return a Problem Details response for unhandled exceptions.
By calling `AddProblemDetails`, we're configuring the application to use the Problem Details format for failed requests.
With `UseExceptionHandler`, we introduce an exception handling middleware to the request pipeline.
By adding `UseStatusCodePages`, we're introducing a middleware that will convert error responses with an empty body to a Problem Details response.

```csharp {4,9,12}
var builder = WebApplication.CreateBuilder(args);

// Adds services for using Problem Details format
builder.Services.AddProblemDetails();

var app = builder.Build();

// Converts unhandled exceptions into Problem Details responses
app.UseExceptionHandler();

// Returns the Problem Details response for (empty) non-successful responses
app.UseStatusCodePages();

app.Run();
```

When we encounter an unhandled exception, it will be translated to a Problem Details response:

```json
Content-Type: application/problem+json

{
  "type": "https://tools.ietf.org/html/rfc9110#section-15.6.1",
  "title": "An error occurred while processing your request.",
  "status": 500
}
```

Now, let's explore how we can customize this response.

## Global Error Handling

We have a few options for [implementing global error handling](global-error-handling-in-aspnetcore-8).
The most popular approach is creating a custom exception handling middleware.
You wrap the API request in a `try-catch` statement and return a response based on any caught exception.

With .NET 8, we can use the `IExceptionHandler` that runs in the built-in exception handling middleware.
This handler allows you to tailor the Problem Details response for specific exceptions.
Returning `true` from the `TryHandleAsync` method short-circuits the pipeline and returns the API response.
If we return `false`, the next handler in the chain attempts to handle the exception.

We can map different exception types to appropriate HTTP status codes, providing more precise error information to API consumers.

Here's an example `CustomExceptionHandler` implementation:

```csharp
internal sealed class CustomExceptionHandler : IExceptionHandler
{
    public async ValueTask<bool> TryHandleAsync(
        HttpContext httpContext,
        Exception exception,
        CancellationToken cancellationToken)
    {
        int status = exception switch
        {
            ArgumentException => StatusCodes.Status400BadRequest,
            _ => StatusCodes.Status500InternalServerError
        };
        httpContext.Response.StatusCode = status;

        var problemDetails = new ProblemDetails
        {
            Status = status,
            Title = "An error occurred",
            Type = exception.GetType().Name,
            Detail = exception.Message
        };

        await httpContext.Response.WriteAsJsonAsync(problemDetails, cancellationToken);

        return true;
    }
}

// In Program.cs
builder.Services.AddExceptionHandler<CustomExceptionHandler>();
```

## Using The ProblemDetailsService

Calling `AddProblemDetails` registers a default implementation of the `IProblemDetailsService`.
The `IProblemDetailsService` will set the response status code based on the `ProblemDetails.Status`.

Here's how we can use it in the `CustomExceptionHandler`:

```csharp
public class CustomExceptionHandler(IProblemDetailsService problemDetailsService) : IExceptionHandler
{
    public async ValueTask<bool> TryHandleAsync(
        HttpContext httpContext,
        Exception exception,
        CancellationToken cancellationToken)
    {
        var problemDetails = new ProblemDetails
        {
            Status = exception switch
            {
                ArgumentException => StatusCodes.Status400BadRequest,
                _ => StatusCodes.Status500InternalServerError
            },
            Title = "An error occurred",
            Type = exception.GetType().Name,
            Detail = exception.Message
        };

        return await problemDetailsService.TryWriteAsync(new ProblemDetailsContext
        {
            Exception = exception,
            HttpContext = httpContext,
            ProblemDetails = problemDetails
        });
    }
}
```

This approach seems very similar to the previous one, where we wrote to the response body.
However, using the `IProblemDetailsService` gives an easy way to customize all Problem Details responses.

We can return Problem Details in controllers using the `Problem` method, or `Results.Problem` in Minimal APIs.
These methods respect the configured Problem Details customizations (more on this in the next section).

```csharp
IdentityUser identityUser = new() { UserName = registerUserDto.UserName, Email = registerUserDto.Email };
IdentityResult result = await userManager.CreateAsync(identityUser, registerUserDto.Password);

if (!result.Succeeded)
{
    // return Results.Problem - Minimal APIs
    return Problem(
        type: "Bad Request",
        title: "Identity failure",
        detail: result.Errors.First().Description,
        statusCode: StatusCodes.Status400BadRequest);
}
```

## Customizing Problem Details

We can pass a delegate to the `AddProblemDetails` method to set the `CustomizeProblemDetails`.
You can use this to add extra information to all Problem Details responses.

This is an excellent place for solving cross-cutting concerns, like setting the `instance` value and adding diagnostics information.

```csharp
builder.Services.AddProblemDetails(options =>
{
    options.CustomizeProblemDetails = context =>
    {
        context.ProblemDetails.Instance =
            $"{context.HttpContext.Request.Method} {context.HttpContext.Request.Path}";

        context.ProblemDetails.Extensions.TryAdd("requestId", context.HttpContext.TraceIdentifier);

        Activity? activity = context.HttpContext.Features.Get<IHttpActivityFeature>()?.Activity;
        context.ProblemDetails.Extensions.TryAdd("traceId", activity?.Id);
    };
});
```

This customization adds the request path, a request ID, and a trace ID to every Problem Details response, enhancing debuggability and traceability of errors.

```json {7-9}
Content-Type: application/problem+json

{
  "type": "https://tools.ietf.org/html/rfc9110#section-15.5.5",
  "title": "Not Found",
  "status": 404,
  "instance": "PUT /api/habits/aadcad3f-8dc8-443d-be44-3d99893ba18a",
  "traceId": "00-63d4af1807586b0d98901ae47944192d-9a8635facb90bf76-01",
  "requestId": "0HN7C8PRNMGIA:00000001"
}
```

You can use the `traceId` to find the distributed traces and logs in a monitoring system like Seq.

<div className="bordered">
  <Image
    src="blogs/mnw_112/seq_tracing.png"
    width={1919}
    height={905}
    alt="Seq user interface showing a distributed trace with the same trace identifier as the problem details response."
  />
</div>

## Handling Specific Exceptions (Status Codes)

.NET 9 introduces a simpler way to map exceptions to status codes.
Great news for fans of throwing exceptions.
You can use the `StatusCodeSelector` to define the mappings.
This makes it easier to maintain consistent error responses across your API.

```csharp
app.UseExceptionHandler(new ExceptionHandlerOptions
{
    StatusCodeSelector = ex => ex switch
    {
        ArgumentException => StatusCodes.Status400BadRequest,
        NotFoundException => StatusCodes.Status404NotFound,
        _ => StatusCodes.Status500InternalServerError
    }
});
```

If you use this together with an `IExceptionHandler` that sets the `StatusCode`, then the `StatusCodeSelector` is ignored.

## Takeaway

Implementing Problem Details in your ASP.NET Core APIs is more than just a best practice -
it's a standard for improving the developer experience of your API consumers.
By providing consistent, detailed, and well-structured error responses, you make it easier for clients to understand and handle error scenarios gracefully.

As you implement these practices in your own projects, you'll discover even more ways to tailor Problem Details to your specific needs.
I shared what worked well for my use cases.

{/* Update this when the course is out */}
Problem Details is just one of the best practices covered in my upcoming [**REST APIs course**](/rest-apis-in-aspnetcore).
Check it out if you're looking for a comprehensive guide.

Good luck out there, and I'll see you next week.

---



<Sponsorships>

Visually Building API-Driven Applications with [**Postman Flows**](https://app.zuddl.com/p/a/event/d2d123ee-fb4c-4359-9b38-cb471fe1a844?utm_source=influencer&utm_medium=referral&utm_campaign=fy25q3_global_ent_postman_flows_webinar&utm_term=existing_influencer_relationships&utm_content=milan_jovanovic).
Explore how Postman Flows demystifies the composition of API applications API workflows through a low-code experience that lets you visualize complex API workflows.
[**Register here**](https://app.zuddl.com/p/a/event/d2d123ee-fb4c-4359-9b38-cb471fe1a844?utm_source=influencer&utm_medium=referral&utm_campaign=fy25q3_global_ent_postman_flows_webinar&utm_term=existing_influencer_relationships&utm_content=milan_jovanovic).

Deliver a great UI with easily customizable [**Page Templates and Building Blocks**](https://www.telerik.com/page-templates-and-ui-blocks?utm_medium=cpm&utm_source=milanjonanovich&utm_campaign=dt_design_templates_building_blocks).
Utilize a collection of over 50 building blocks and 10 page templates designed to help you build modern UI in no time.
All you need to do is copy and paste!
[**Learn more here**](https://www.telerik.com/page-templates-and-ui-blocks?utm_medium=cpm&utm_source=milanjonanovich&utm_campaign=dt_design_templates_building_blocks).

</Sponsorships>

In distributed systems, we often face the challenge of keeping our database and external systems in sync.
Imagine saving an order to a database and then publishing a message to a message broker.
If either operation fails, your system ends up in an inconsistent state.

The [**Outbox pattern**](outbox-pattern-for-reliable-microservices-messaging) solves this problem by treating message publication as part of your database transaction.
Instead of publishing messages directly, we save them to an Outbox table in our database, ensuring atomic operations.
A separate process then reliably publishes these messages.

In this newsletter, we'll dive into implementing this pattern in .NET, covering everything from setup to scaling.

## Why Do We Need the Outbox Pattern?

The transactional Outbox pattern fixes a common problem in distributed systems.
This problem happens when you need to do two things at once: save data and communicate with an external component.

Consider scenarios like sending order confirmation emails, notifying other systems about new client registrations,
or updating inventory levels after an order is placed.
Each of these involves a local data change coupled with an external communication or update.

For example, imagine a microservice that needs to:

- Save a new order in its database
- Tell other systems about this new order

If one of these steps fails, your system could end up in an inconsistent state.
Maybe the order is saved, but no one else knows about it.
Or everyone thinks there's a new order, but it's not actually in the database.

Here's a `CreateOrderCommandHandler` without the Outbox pattern:

```csharp
public class CreateOrderCommandHandler(
    IOrderRepository orderRepository,
    IProductInventoryChecker inventoryChecker,
    IUnitOfWork unitOfWork,
    IEventBus eventBus) : IRequestHandler<CreateOrderCommand, OrderDto>
{
    public async Task<OrderDto> Handle(CreateOrderCommand request, CancellationToken cancellationToken)
    {
        var order = new Order(request.CustomerId, request.ProductId, request.Quantity, inventoryChecker);

        await orderRepository.AddAsync(order);

        await unitOfWork.CommitAsync(cancellationToken);

        // The database transaction is completed at this point.

        await eventBus.Send(new OrderCreatedIntegrationEvent(order.Id));

        return new OrderDto { Id = order.Id, Total = order.Total };
    }
}
```

This code has a potential consistency problem.
After the database transaction is committed, two things could go wrong:

1. The application might crash right after the transaction is committed but before the event is sent.
   The order would be created in the database, but other systems wouldn't know about it.

2. The event bus might be down or unreachable when we try to send the event.
   This would result in the order being created without notifying other systems.

The transactional Outbox pattern helps solve this problem by ensuring that the database update and event publication are treated as a single atomic operation.

<Image
  src="blogs/mnw_110/outbox_pattern.png"
  width={2102}
  height={1372}
  alt="Flow diagram explaining how the Outbox Pattern works."
/>

The sequence diagram illustrates how the Outbox pattern solves our consistency challenge.
Instead of trying to save data and send a message as separate steps, we save both the order and an Outbox message in a **single database transaction**.
This is an all-or-nothing operation - we can't end up in an inconsistent state.

A separate Outbox processor handles the actual message sending.
It continuously checks for unsent messages in the Outbox table and publishes them to the message queue.
The processor marks messages as sent after successful publishing, preventing duplicates.

An important thing to realize here is that the Outbox pattern gives us [at-least-once delivery](https://www.cloudcomputingpatterns.org/at_least_once_delivery/).
The Outbox message will be sent at least once, but it could also be sent multiple times in case of retries.
This means we have to make our message consumers idempotent.

## Implementing the Outbox Pattern

First, let's create our Outbox table where we will store messages:

```sql
CREATE TABLE outbox_messages (
    id UUID PRIMARY KEY,
    type VARCHAR(255) NOT NULL,
    content JSONB NOT NULL,
    occurred_on_utc TIMESTAMP WITH TIME ZONE NOT NULL,
    processed_on_utc TIMESTAMP WITH TIME ZONE NULL,
    error TEXT NULL
);

-- We can consider adding this index since we will be querying for unprocessed messages often
-- and it will contain the rows in the correct sort order for our query.
CREATE INDEX IF NOT EXISTS idx_outbox_messages_unprocessed
ON outbox_messages (occurred_on_utc, processed_on_utc)
INCLUDE (id, type, content)
WHERE processed_on_utc IS NULL;
```

I'll use PostgreSQL as the database for this example.
Notice the `jsonb` type for the `content` column.
It allows for indexing and querying of the JSON data if needed in the future.

Now, let's create a class to represent our Outbox entry:

```csharp
public sealed class OutboxMessage
{
    public Guid Id { get; init; }
    public string Type { get; init; }
    public string Content { get; init; }
    public DateTime OccurredOnUtc { get; init; }
    public DateTime? ProcessedOnUtc { get; init; }
    public string? Error { get; init; }
}
```

Here's how we can add a message to the Outbox:

```csharp
public async Task AddToOutbox<T>(T message, NpgsqlDataSource dataSource)
{
    var outboxMessage = new OutboxMessage
    {
        Id = Guid.NewGuid(),
        OccurredOnUtc = DateTime.UtcNow,
        Type = typeof(T).FullName, // We'll need this for deserialization
        Content = JsonSerializer.Serialize(message)
    };

    await using var connection = await dataSource.OpenConnectionAsync();
    await connection.ExecuteAsync(
        @"""
        INSERT INTO outbox_messages (id, occurred_on_utc, type, content)
        VALUES (@Id, @OccurredOnUtc, @Type, @Content::jsonb)
        """,
        outboxMessage);
}
```

An elegant approach to implementing this is using [**domain events**](how-to-use-domain-events-to-build-loosely-coupled-systems) to represent notifications.
When something significant happens in the domain, we will raise a domain event.
Before completing the transaction, we can pick up all events and store them as Outbox messages.
You could do this from the unit of work or with an [**EF Core interceptor**](how-to-use-ef-core-interceptors).

## Processing the Outbox

The Outbox processor is the next component we'll need.
This could be a _physically_ separate process or a background worker in the same process.

I'll use Quartz to [**schedule background jobs**](scheduling-background-jobs-with-quartz-net) for Outbox processing.
It's a robust library with excellent support for scheduling recurring jobs.

Now, let's implement the `OutboxProcessorJob`:

```csharp
[DisallowConcurrentExecution]
public class OutboxProcessorJob(
    NpgsqlDataSource dataSource,
    IPublishEndpoint publishEndpoint,
    Assembly integrationEventsAssembly) : IJob
{
    public async Task Execute(IJobExecutionContext context)
    {
        await using var connection = await dataSource.OpenConnectionAsync();
        await using var transaction = await connection.BeginTransactionAsync();

        // You can make the limit a parameter, to control the batch size.
        // We can also select just the id, type, and content columns.
        var messages = await connection.QueryAsync<OutboxMessage>(
            @"""
            SELECT id AS Id, type AS Type, content AS Content
            FROM outbox_messages
            WHERE processed_on_utc IS NULL
            ORDER BY occurred_on_utc LIMIT 100
            """,
            transaction: transaction);

        foreach (var message in messages)
        {
            try
            {
                var messageType = integrationEventsAssembly.GetType(message.Type);
                var deserializedMessage = JsonSerializer.Deserialize(message.Content, messageType);

                // We should introduce retries here to improve reliability.
                await publishEndpoint.Publish(deserializedMessage);

                await connection.ExecuteAsync(
                    @"""
                    UPDATE outbox_messages
                    SET processed_on_utc = @ProcessedOnUtc
                    WHERE id = @Id
                    """,
                    new { ProcessedOnUtc = DateTime.UtcNow, message.Id },
                    transaction: transaction);
            }
            catch (Exception ex)
            {
                // We can also introduce error logging here.

                await connection.ExecuteAsync(
                    @"""
                    UPDATE outbox_messages
                    SET processed_on_utc = @ProcessedOnUtc, error = @Error
                    WHERE id = @Id
                    """,
                    new { ProcessedOnUtc = DateTime.UtcNow, Error = ex.ToString(), message.Id },
                    transaction: transaction);
            }
        }

        await transaction.CommitAsync();
    }
}
```

This approach uses polling to periodically fetch unprocessed messages from the database.
Polling can increase the load on the database, as we'll need to query for unprocessed messages frequently.

An alternative way to process Outbox messages is by using [Transaction log tailing](https://microservices.io/patterns/data/transaction-log-tailing.html).
We can implement this using [Postgres logical replication](https://www.npgsql.org/doc/replication.html).
The database will stream changes from the Write-Ahead Log (WAL) to our application, and we'll process these messages and publish them to the message broker.
You can use this to implement a push-based Outbox processor.

## Considerations and Tradeoffs

The Outbox pattern, while effective, introduces additional complexity and database writes.
In high-throughput systems, it's crucial to monitor its performance to ensure it doesn't become a bottleneck.

I recommend implementing retry mechanisms in the Outbox processor to [**improve reliability**](building-resilient-cloud-applications-with-dotnet).
Consider using exponential backoff for transient failures and a circuit breaker for persistent issues to prevent system overload during outages.

It's essential that you implement [**idempotent message consumers**](idempotent-consumer-handling-duplicate-messages).
Network issues or processor restarts can lead to multiple deliveries of the same message, so your consumers must handle repeated processing safely.

Over time, the Outbox table can grow significantly, potentially impacting database performance.
It's important to implement an archiving strategy early on.
Consider moving processed messages to cold storage or deleting them after a set period.

## Scaling Outbox Processing

As your system grows, you may find that a single Outbox processor can't keep up with the volume of messages.
This can lead to increased latency between when an event occurs and when it's processed by consumers.

One straightforward approach is to increase the frequency of the Outbox processor job.
You should consider running it every few seconds.
This can significantly reduce the delay in message processing.

Another effective strategy is to increase the batch size when fetching unprocessed messages.
By processing more messages in each run, you can improve throughput.
However, be cautious not to make the batches so large that they cause long-running transactions.

For high-volume systems, processing the Outbox in parallel can be very effective.
Implement a locking mechanism to claim batches of messages, allowing multiple processors to work simultaneously without conflict.
You can use `SELECT ... FOR UPDATE SKIP LOCKED` to claim a batch of messages.
This approach can dramatically increase your processing capacity.

## Wrapping Up

The Outbox pattern is a powerful tool for maintaining data consistency in distributed systems.
By decoupling database operations from message publishing, the Outbox pattern ensures that your system remains reliable even in the face of failures.

Remember to keep your consumers idempotent, implement proper scaling strategies, and manage your Outbox table growth.

While it adds some complexity, the benefits of guaranteed message delivery make it a valuable pattern in many scenarios.

If you're looking to implement the Outbox pattern in a robust, production-ready way, you can check out [**Pragmatic Clean Architecture**](/pragmatic-clean-architecture).
It includes an entire section on implementing the Outbox pattern, along with other essential patterns for building maintainable and scalable .NET applications.

That's all for today. Stay awesome, and I'll see you next week.

---



<Sponsorships>

Integrate e-signatures into your workflows with the [**BoldSign .NET SDK**](https://boldsign.com/esignature-api/?utm_source=milanjovanovic&utm_medium=newsletter_ad&utm_campaign=boldsign).
The e-signature API lets you embed signature requests, verify signer identity, and add custom branding.
Free coding assistance and migration support is available.
Try the [**free API sandbox**](https://boldsign.com/esignature-api/?utm_source=milanjovanovic&utm_medium=newsletter_ad&utm_campaign=boldsign) today!

[**Webinar: "Unlocking AI's Potential: Why the Future Depends on APIs"**](https://app.zuddl.com/p/a/event/8e8f96c1-99d5-4392-96a1-c68b8c8b9d2d?utm_source=influencer&utm_medium=referral&utm_campaign=fy25q3_global_ent_unlocking_ai_potential_webinar&utm_term=existing_influencer_relationships&utm_content=milan_jovanovic).
Join Postman CTO, Ankit Sobti, and Head of Customer Experience and Success, Kristine Chin,
at this free webinar to maximize the success of your API products
and provide a world-class experience for your developers, partners, and customers.
[**Join here**](https://app.zuddl.com/p/a/event/8e8f96c1-99d5-4392-96a1-c68b8c8b9d2d?utm_source=influencer&utm_medium=referral&utm_campaign=fy25q3_global_ent_unlocking_ai_potential_webinar&utm_term=existing_influencer_relationships&utm_content=milan_jovanovic).

</Sponsorships>

As your application grows, you might find yourself considering a move from a modular monolith to microservices.
This transition isn't just a technical shift.
It's a strategic move that can reshape how your entire system operates.
But let's be clear: it's not a magic solution and comes with its own challenges.

In this article, I'll share my experience of migrating from a modular monolith to microservices.

We'll explore why you might consider microservices, how to prepare for the migration and the critical steps in the migration process -
from choosing your first module to implementing inter-service communication and managing data migration.

Let's dive in!

## Why Consider Microservices?

A [**modular monolith**](what-is-a-modular-monolith) serves as an excellent starting point for many applications.
It's simpler to develop, easier to understand, and faster to deploy than a distributed system.
But as your application grows, you might start facing some problems.

Here are the key challenges I've encountered with modular monoliths:

- High-load modules can become bottlenecks, affecting the entire system's performance
- As the monolith grows, deployments become riskier and more time-consuming
- Large teams working on a single codebase often step on each other's toes
- You're locked into a single technology stack for the entire application

Microservices can address these issues, but it's not a decision to be taken lightly.

<Image
  src="blogs/mnw_109/modular_monolith_vs_microservices.png"
  width={1646}
  height={685}
  alt="Modular monolith to microservices comparison."
/>

Here's how microservices can help:

- **Scalability**: You can scale individual services based on their specific load, optimizing resource use.
- **Independent deployability**: Updates to one service don't require redeploying the entire application, reducing risk and downtime.
- **Team autonomy**: Separate teams can own different services, leading to faster development cycles.

The good news? If you've built a well-designed modular monolith, you're already halfway there.
The clear boundaries between modules in your monolith can serve as a blueprint for your microservices architecture.

Remember, microservices come with their own complexities in areas like data consistency and inter-service communication.
But for the right use cases, they can provide the flexibility and scalability needed for growing applications.

## Preparing for Migration

The success of your migration largely depends on how well you prepare.
Your best starting point is a well-structured modular monolith.

Here are key areas to focus on:

1. **Review module boundaries**:
   I can't stress this enough: clear module boundaries are crucial.
   Each module should have a distinct responsibility and minimal dependencies on others.
   If boundaries are blurry, refactor before migrating.

2. **Ensure proper data encapsulation**:
   Modules should not directly access each other's data stores.
   I make sure each module owns and manages its data exclusively.
   No shared tables, no direct database access across modules.
   This clean separation makes it much easier to extract modules into microservices later.

3. **Implement clean public APIs**:
   Define clear contracts between modules to facilitate future separation.
   Modules should communicate through well-defined APIs, not by accessing each other's internals.

Here's an example of the above:

```csharp {7,20}
// Good: Clear module boundary
namespace BookingsModule;

public class CreateBooking
{
    private readonly IBookingRepository _bookingRepository;
    private readonly IPaymentGateway _paymentGateway; // Public API

    public CreateBooking(IBookingRepository bookingRepository, IPaymentGateway paymentGateway)
    {
        _bookingRepository = bookingRepository;
        _paymentGateway = paymentGateway;
    }

    public async Task<BookingResult> CreateBookingAsync(BookingRequest request)
    {
        var booking = await _bookingRepository.CreateAsync(Booking.FromRequest(request));

        // Accessing the other module's data through an abstraction
        var paymentResult = await _paymentGateway.ProcessPaymentAsync(booking.Id, booking.TotalAmount);

        return new BookingResult(booking, paymentResult);
    }
}
```

By focusing on these areas, you're not just preparing for migration - you're improving your modular monolith.
Even if you decide not to migrate, these steps will make your system more maintainable and scalable.

## Choosing and Extracting the First Module

Selecting the right module to extract first can set the tone for your entire migration.
In my experience, it's crucial to start with a module that's self-contained and has clear boundaries.

When I'm evaluating modules, I look for these characteristics:

- Low coupling with other modules
- High cohesion within the module
- A distinct business function
- Potential performance or scalability gains from separation

For instance, in an e-commerce system, I might choose the product catalog module.
It typically has a clear purpose, doesn't heavily depend on other modules, and could benefit from independent scaling.

Once you've chosen your module, here's the extraction process I follow:

1. Create a new project for the microservice.
2. Move the module's code to the new project.
   This often reveals hidden dependencies, which is valuable information.
3. Update the dependencies, ensuring the microservice is self-contained.
   This might involve copying some shared code or refactoring to remove unnecessary dependencies.
4. Set up a separate database for the microservice.
   This enforces data independence.
5. Implement a data migration strategy (more on this later)

Here's what this might look like in practice:

<Image
  src="blogs/mnw_109/modular_monolith_extraction.png"
  width={1088}
  height={1184}
  alt="Modular monolith extraction process into microservices."
/>

Remember, the goal isn't perfection on the first try.
I often iterate on this process, gradually refining the separation between the new microservice and the monolith.

The first extraction is a learning experience.
It'll show you how to approach subsequent modules and help you refine the overall migration strategy.

## Implementing Inter-Service Communication

Once you've extracted a module into a microservice, you have to update the [**module's communication**](modular-monolith-communication-patterns)
with the rest of the system.
This typically involves transitioning from direct method calls to network-based communication.

Here's how I approach this transition:

1. Replace direct method calls with HTTP API calls.
   I often use libraries like [**Refit**](refit-in-dotnet-building-robust-api-clients-in-csharp) to simplify API interactions.
   Here's a before and after example:

```csharp
// Before: Direct method call
BookingDto booking = await _bookingService.GetAsync(bookingId);

// After: HTTP API call
var response = await _httpClient.GetAsync($"https://bookings-service/api/bookings/{bookingId}");

var booking = await response.Content.ReadFromJsonAsync<BookingDto>();
```

2. For asynchronous communication, consider implementing a messaging system.
   While the implementation details vary, tools like [**RabbitMQ**](using-masstransit-with-rabbitmq-and-azure-service-bus),
   [**Amazon SQS and SNS**](complete-guide-to-amazon-sqs-and-amazon-sns-with-masstransit),
   or [**Azure Service Bus**](messaging-made-easy-with-azure-service-bus)
   can significantly improve system resilience and decoupling.

3. Network communication introduces new failure modes, so I always implement [**resilience patterns**](building-resilient-cloud-applications-with-dotnet).
   Here's how I use Polly with resilience pipelines:

```csharp
using Polly;

var pipeline = new ResiliencePipelineBuilder<HttpResponseMessage>()
    .AddRetry(new RetryStrategyOptions
    {
        ShouldHandle = new PredicateBuilder().Handle<ConflictException>(),
        Delay = TimeSpan.FromSeconds(1),
        MaxRetryAttempts = 2,
        BackoffType = DelayBackoffType.Exponential,
        UseJitter = true
    })
    .AddTimeout(new TimeoutStrategyOptions
    {
        Timeout = TimeSpan.FromSeconds(10)
    })
    .Build();

var response = await pipeline.ExecuteAsync(
    async ct => await _httpClient.GetAsync($"https://bookings-service/api/bookings/{bookingId}", ct),
    cancellationToken);
```

Transitioning to HTTP-based communication always brings new problems to solve.
Serialization, error handling, and [**API versioning**](api-versioning-in-aspnetcore) become crucial.
I've learned to plan for these aspects from the start to avoid headaches down the line.

## Implementing an API Gateway

As you extract more services, managing the communication between clients and your microservices can become complex.
An API Gateway can help manage this complexity by providing a single entry point for all clients.

YARP (Yet Another Reverse Proxy) is an excellent tool for [**implementing an API Gateway**](implementing-an-api-gateway-for-microservices-with-yarp) in .NET.
Here's how I typically set it up:

```csharp
var builder = WebApplication.CreateBuilder(args);

builder.Services.AddReverseProxy()
    .LoadFromConfig(builder.Configuration.GetSection("ReverseProxy"));

var app = builder.Build();

app.MapReverseProxy();

app.Run();
```

Then, I configure the routing in `appsettings.json`:

```json
{
  "ReverseProxy": {
    "Routes": {
      "bookings-route": {
        "ClusterId": "bookings-cluster",
        "Match": {
          "Path": "/api/bookings/{**catch-all}"
        }
      },
      "payments-route": {
        "ClusterId": "payments-cluster",
        "Match": {
          "Path": "/api/payments/{**catch-all}"
        }
      }
    },
    "Clusters": {
      "bookings-cluster": {
        "Destinations": {
          "destination1": {
            "Address": "https://bookings-service/"
          }
        }
      },
      "payments-cluster": {
        "Destinations": {
          "destination1": {
            "Address": "https://payments-service/"
          }
        }
      }
    }
  }
}
```

This setup routes requests to the appropriate microservice based on the path.
For example, any request to `/api/bookings/*` will be routed to the `bookings-service`.

<Image
  src="blogs/mnw_109/modular_monolith_extraction_api_gateway.png"
  width={1050}
  height={737}
  alt="Modular monolith extraction process into microservices with an API gateway introduced."
/>

In my projects, I've found that an API Gateway often becomes a critical point for implementing cross-cutting concerns.
As your architecture evolves, consider adding features like:

- [**Rate limiting**](advanced-rate-limiting-use-cases-in-dotnet) to protect your services from overload
- [**Caching**](caching-in-aspnetcore-improving-application-performance) to improve response times for frequently requested data
- Request/response transformation to adapt your internal APIs for external consumption

Start simple, but be prepared to evolve your gateway as you learn more about your system's needs and usage patterns.

## Data Migration Strategy

Data migration is often the trickiest part of moving from a modular monolith to microservices. In my experience, there are two main approaches:

1. **The "One and Done" Approach**

   For simpler systems or those that can handle some downtime, I use this method:

   - Create a new database for the microservice
   - Copy the relevant schema and data from the monolith
   - Switch the application to use the new database

   It's quick and simple but requires a brief downtime.

2. **The Synchronization Approach**

   For complex systems needing minimal disruption, I use this method:

   - Copy the schema and initial data to the new database
   - Set up a sync mechanism (like [change data capture](https://en.wikipedia.org/wiki/Change_data_capture)) to keep the new database updated
   - Gradually shift traffic to the new microservice
   - Eventually, remove the old schema from the monolith

   This approach is more complex but allows for a smoother transition.

In both cases, I always ensure I have a solid backup and rollback plan.
The key is to choose the method that best fits your system's needs and tolerance for complexity.

Remember, if your modular monolith already uses [**separate database schemas for each module**](modular-monolith-data-isolation), you're starting with an advantage.
This logical isolation makes the migration process significantly easier.

## Summary

Migrating from a modular monolith to microservices is a journey I've taken several times, and it's always both challenging and rewarding.
The process we've covered here - from preparing your monolith and choosing the right module to extract
to implementing proper communication patterns - forms the foundation of a successful migration.

Don't rush the process.
Careful planning and incremental changes are key.
Each step teaches you something valuable about your system.

While we've covered the fundamentals, there's always more to learn. I recommend exploring these advanced topics:

- [**Service discovery**](service-discovery-in-microservices-with-net-and-consul) and API gateway patterns
- [**Monitoring and observability**](introduction-to-distributed-tracing-with-opentelemetry-in-dotnet) in a microservices environment
- Advanced [**deployment strategies**](how-to-build-ci-cd-pipeline-with-github-actions-and-dotnet), like blue-green deployment

If you're ready to master this process and gain hands-on experience, I've put together a comprehensive
[**Modular Monolith Architecture**](/modular-monolith-architecture) course.
There's an entire chapter dedicated to extracting modules and moving to microservices.
You'll gain practical skills to confidently navigate your microservices transformation.

Remember, the goal isn't to blindly adopt microservices but to evolve your architecture to best serve your business needs.
Sometimes, a well-structured modular monolith is the right solution.
Other times, a full microservices architecture is the way to go.
The key is understanding the trade-offs and making informed decisions.

Good luck out there, and I'll see you next week.

---



<Sponsorships>

[**Postman v11**](https://blog.postman.com/announcing-postman-v11/) takes your dev collaboration to next level using AI and APIs.
Check out [**Postman v11**](https://blog.postman.com/announcing-postman-v11/) to get started with it's features designed to help bring API collaboration to the next level.

Deliver a great UI with easily customizable [**Page Templates and Building Blocks**](https://www.telerik.com/page-templates-and-ui-blocks?utm_medium=cpm&utm_source=milanjonanovich&utm_campaign=dt_design_templates_building_blocks).
Utilize a collection of over 50 building blocks and 10 page templates designed to help you build modern UI in no time.
All you need to do is copy and paste!
[**Learn more here**](https://www.telerik.com/page-templates-and-ui-blocks?utm_medium=cpm&utm_source=milanjonanovich&utm_campaign=dt_design_templates_building_blocks).

</Sponsorships>

So, I've got this blog I've been running for a couple of years now.
It's a Next.js and TypeScript static site, all optimized for SEO.
Sounds pretty standard, right?
Well, it was, until I ran into a problem.

As my content grew, finding specific articles became a real pain.
It wasn't just me - readers were struggling too.
What's the point of writing all this stuff if no one can find it when they need it?

That's when I decided to look into full-text search.
Now, implementing full-text search on a static site isn't exactly straightforward.
It's not like you can just slap a database on there and call it a day.
You've got to get creative.

In this article, I'll walk you through how I turned my site's search from useless to actually functional.

## The Old Search Approach

Here's how the search function worked before:

1. I would generate a JSON document at build time containing each blog post's metadata.
   This document contains the data needed to perform the search and render the results.
2. When searching the articles, I would fetch the JSON document on the client and perform the search.
   Since this document changes only once a week, I can cache it on the client to improve performance.

Here's the code that would generate the search data JSON document:

```ts
export const generateSearchData = () => {
  const posts = getAllPosts();

  const searchData = posts.map((post) => ({
    slug: post.meta.slug,
    title: post.meta.title,
    excerpt: post.meta.excerpt,
    coverImage: post.meta.coverImage,
    date: post.meta.date,
    searchableContent:
      `${post.meta.title} ${post.meta.excerpt} ${post.content}`.toLowerCase()
  }));

  fs.mkdirSync('./search', { recursive: true });
  fs.writeFileSync(
    path.join(process.cwd(), 'search', 'search-data.json'),
    JSON.stringify(searchData)
  );
};
```

And then the search function is pretty simple (but not very smart):

```typescript
export const search(keyword) = () => {
  const res = await fetch('/search/search-data.json');

  const searchData = await res.json();

  return searchData.filter((p) =>
    p.searchableContent.includes(keyword.toLowerCase())
  );
}
```

This approach has a few issues:

- The search data JSON document is large (861KB) and will continue to grow
- It's slow on large datasets and won't scale well
- It uses an exact match to find results
- It doesn't rank results by relevance

I wasn't happy with this, so here's how I implemented a much better solution.

## Introducing Full-Text Search

To fix these problems, I turned to **full-text search** using [Lunr.js](https://lunrjs.com/).
But what exactly is full-text search?

Full-text search is a technique that allows fast and efficient searching of large volumes of text by creating an index of all words in a document collection
and returning ranked results based on relevance to the search query.

Full-text search works by:

1. **Indexing**: Breaking down text into individual words (tokens)
2. **Stemming**: Reducing words to their base form (e.g., "running" to "run")
3. **Ranking**: Scoring results based on relevance

Another common operation is stop word removal.
Common words that don't add much meaning (like "the", "and", "is") are often removed to save space and improve relevance.

At the heart of full-text search is an [inverted index](https://en.wikipedia.org/wiki/Inverted_index).
It's a data structure that maps each unique term to the documents containing it.
It's "inverted" because it goes from terms to documents, rather than documents to terms.

When searching, results are often ranked using [TF-IDF](https://en.wikipedia.org/wiki/Tf-idf) (Term Frequency-Inverse Document Frequency).
This gives higher scores to terms that are frequent in a document but rare across all documents.

That's the rundown on full-text search. Let's see how to implement it.

## Implementing Full-Text Search

I chose Lunr.js because it's lightweight and works great for static websites. No server required.

I updated the `generateSearchData` function to create and store a full-text search index.
This function runs once at build time, so it's not expensive.

```typescript {6-14,19-22}
export const generateSearchData = () => {
  const posts = getAllPosts();

  // Generate search data as before.

  const index = lunr(function () {
    this.ref('slug');
    this.field('title', { boost: 10 });
    this.field('content', { boost: 5 });

    searchData.forEach((doc) => {
      this.add(doc);
    });
  });

  // Store search data as before.

  // And store the search index.
  fs.writeFileSync(
    path.join(process.cwd(), 'search', 'search-index.json'),
    JSON.stringify(index)
  );
};
```

I'm **boosting** the title with a factor of 10 and the content with a factor of 5.
If the search term matches the title, it will have a higher relevance score.

The downside (for now) is that the index file is big, 2.5MB.
If I were to download this on the client every time, the costs would quickly add up.
I have 100,000 unique visitors per month, which would equate to `250GB` of network egress.

Now, I have to update the search function to use the full-text index:

```typescript {6}
export const search(keyword) = () => {
  const res = await fetch('/search/search-data.json');
  const searchData = await res.json();

  const res = await fetch('/search/search-index.json');
  const searchIndex = lunr.Index.load(indexJson);

  return searchIndex
    .search(keyword)
    .map((result) => {
      const post = searchData?.find((post) => post.slug === result.ref);
      return post ? { ...post, score: result.score } : null;
    })
    .filter((result) => result !== null)
    .sort((a, b) => b.score - a.score);
}
```

What's happening here:

- We're loading the search index from a JSON document using `lunr.Index.load`
- The index has a `search` function allowing us to perform full-text search
- We're expanding the result to also include the relevance score

This allows me to sort the search results based on relevance instead of chronologically.

Here's what the old search implementation returns when searching for "resilience":

<div className="bordered centered">
  <Image
    src="blogs/mnw_108/old_search.png"
    width={814}
    height={912}
    alt="The web page showing search results for the keyword 'resilience' with the most relevant result highlighted."
  />
</div>

The most relevant article is at the fourth spot, sorted chronologically.
You can see how this isn't a good user experience.

But with full-text search, when you search for a keyword like "resilience", you get nicer results:

<div className="bordered centered">
  <Image
    src="blogs/mnw_108/full_text_search.png"
    width={814}
    height={863}
    alt="The web page showing full-text search results for the keyword 'resilience' with the most relevant result highlighted."
  />
</div>

I also added a relevance score next to each article because it looks cool.

## Optimizing The Search

Now, I have a powerful search, but the index was huge (a whopping 2.5MB) and will continue to grow.
The search data file is also 861KB.

Enter [Brotli](https://en.wikipedia.org/wiki/Brotli) compression:

- I compressed the search index and data using Brotli on the server side
- In the browser, I decompress the files before using them to perform the search

```typescript {8,12}
import { compress } from 'brotli';

export const generateSearchData = () => {
  // Create the search data and index

  fs.writeFileSync(
    path.join(process.cwd(), 'search', 'search-index.br'),
    compress(Buffer.from(JSON.stringify(index)))
  );
  fs.writeFileSync(
    path.join(process.cwd(), 'search', 'search-data.br'),
    compress(Buffer.from(JSON.stringify(searchData)))
  );
};
```

The results:

- 2.5MB → 193KB **(-92%)**
- 861KB → 180KB **(-79%)**

If you want to learn more about compression algorithms, check out this [**article about response compression**](response-compression-in-aspnetcore).

After fetching the compressed search data and full-text index on the client, I cache them for subsequent searches.

The searches are lightning-fast now, and the results are more relevant.

## Server-Side Options for Full-Text Search

While Lunr.js works great for my static site, larger apps with more data need a robust full-text search solution.
So, here are some some server-side options you could explore.

[Lucene](https://lucene.apache.org/) is the foundation of many search engines.
It's written in Java but has ports to other languages.
Lucene provides robust full-text indexing and search capabilities.
It is also highly efficient and customizable, making it a popular choice for developers who need fine-grained control over their search functionality.

[Apache Solr](https://solr.apache.org/) builds on top of Lucene, offering additional features like distributed indexing, replication, and load-balanced querying.
Solr includes powerful capabilities such as faceting and highlighting, which can greatly enhance the search experience.

If you're already using [PostgreSQL](https://www.postgresql.org/), the built-in [full-text search](https://www.postgresql.org/docs/current/textsearch.html) is worth considering.
PostgreSQL uses its own text search engine, which supports multiple languages and custom dictionaries.
While not as feature-rich as dedicated search engines, it can be a convenient option for applications that want to keep their stack simple.

I've been tinkering with PostgreSQL [full-text search using EF Core](https://www.npgsql.org/efcore/mapping/full-text-search.html).
Here's what a full-text search query looks like:

```csharp
var blogs = context
    .BlogPosts
    .Where(b =>
        EF.Functions.ToTsVector("english", b.Title + " " + b.Excerpt + " " + b.Content)
            .Matches(EF.Functions.PhraseToTsQuery("english", searchTerm)))
    .Select(b => new
    {
        b.Slug,
        b.Title,
        b.Excerpt,
        b.Date,
        Rank = EF.Functions.ToTsVector("english", b.Title + " " + b.Excerpt + " " + b.Content)
            .Rank(EF.Functions.PhraseToTsQuery("english", searchTerm))
    })
    .OrderByDescending(b => b.Rank)
    .ToList();
```

But I'll have to write a dedicated article to cover this, so let's wrap it up.

## Wrapping Up

The old search functionality was, frankly, embarrassing.

It was slow, dumb as a rock, and about as useful as a chocolate teapot.

Now? It's actually not half bad.

- Searches are lightning-fast
- Results are more relevant
- The user experience is much smoother

I'm not gonna lie, there was a moment of frustration when I thought about giving up.
But I'm glad I stuck with it.
There's something satisfying about building it yourself, you know?

It might not be pretty, but it works.
And sometimes, that's all that matters.

The Brotli compression bit was a pleasant surprise.
I thought I'd end up with a massive index file that would make mobile users cry.
But nope, it all works smoothly.
Go figure.

Have you tried implementing search on your site? How'd it go?

That's all for today.

See you next week.

---



<Sponsorships>

Introducing [**IronSecureDoc**](https://ironsoftware.com/enterprise/securedoc/?utm_source=milan&utm_medium=email&utm_campaign=milan) -
the PDF security & compliance server.
Protect your PDFs with IronSecureDoc, the latest from Iron Software.
Redact sensitive data, add digital signatures, and stay compliant with PDF/A & PDF/UA standards.
Easy integration with our REST API.
[**Learn more here**](https://ironsoftware.com/enterprise/securedoc/?utm_source=milan&utm_medium=email&utm_campaign=milan).

Take Postman wherever you work with the [**Postman GA VS Code extension**](https://learning.postman.com/docs/getting-started/basics/about-vs-code-extension)!
The Postman VS Code extension enables you to develop and test your APIs in Postman directly from Visual Studio Code and Visual Studio Code Insiders.
[**Try it out**](https://learning.postman.com/docs/getting-started/basics/about-vs-code-extension)!

</Sponsorships>

I've been tinkering with .NET Aspire lately, and I've got some thoughts to share.
If you're curious about this new cloud-native development tool from Microsoft, stick around.
I'll break down what's great, what's not, and how you can start using it.

> .NET Aspire is an opinionated, cloud-ready stack for building observable, production-ready, distributed applications.

[**.NET Aspire**](https://learn.microsoft.com/en-us/dotnet/aspire/get-started/aspire-overview) is Microsoft's latest offering for cloud-native application development.
It aims to simplify the process of building, deploying, and managing distributed applications.

Distributed applications often consist of small applications that consume external services like databases, message brokers, and caching.
.NET Aspire gives you a set of tools to make building distributed applications easier.

## .NET Aspire Orchestration

How are you setting up a local development environment?
I often use **Docker Compose** to configure my applications and run external services.
It's a simple setup, but you need to manage environment variables and connection strings.
If you're not familiar with Docker, it can prove to be quite tricky sometimes.

Here's a `docker-compose.yml` file from a recent project:

```yml
services:
  contentplatform-api:
    image: ${DOCKER_REGISTRY-}contentplatform-api
    container_name: ContentPlatform.Api
    build:
      context: .
      dockerfile: ContentPlatform.Api/Dockerfile
    ports:
      - 5000:8080
      - 5001:8081

  contentplatform-reporting-api:
    image: ${DOCKER_REGISTRY-}contentplatform-reporting-api
    container_name: ContentPlatform.Reporting.Api
    build:
      context: .
      dockerfile: ContentPlatform.Reporting.Api/Dockerfile
    ports:
      - 6000:8080
      - 6001:8081

  contentplatform-presentation:
    image: contentplatform-ui:latest
    container_name: ContentPlatform.Presentation
    environment:
      - ASPNETCORE_ENVIRONMENT=Development
    ports:
      - 3000:80

  contentplatform-db:
    image: postgres:latest
    container_name: ContentPlatform.Db
    environment:
      - POSTGRES_DB=contentplatform
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
    volumes:
      - ./.containers/db:/var/lib/postgresql/data
    ports:
      - 5432:5432

  contentplatform-mq:
    image: rabbitmq:management
    container_name: ContentPlatform.RabbitMq
    hostname: contentplatform-mq
    volumes:
      - ./.containers/queue/data/:/var/lib/rabbitmq
      - ./.containers/queue/log/:/var/log/rabbitmq
    environment:
      RABBITMQ_DEFAULT_USER: guest
      RABBITMQ_DEFAULT_PASS: guest
```

This sets up two APIs, a client application, PostgreSQL, and RabbitMQ.
I also have to configure the connection strings manually to connect to these services.

So, I decided to migrate this application to .NET Aspire and documented the process.

You can right-click an existing project in Visual Studio and select `Add > .NET Aspire Orchestrator Support...`.

<figure className="figure-center">
  <Image
    src="blogs/mnw_107/aspire_orchestration.png"
    width={488 / 1.4}
    height={706 / 1.4}
    alt="Context menu with '.NET Aspire Orchestrator Support...' highlighted."
  />
  <figcaption>
    Source:
    [Microsoft](https://learn.microsoft.com/en-us/dotnet/aspire/get-started/add-aspire-existing-app)
  </figcaption>
</figure>

This will add an `AppHost` and `ServiceDefaults` project to your solution.
You will then repeat this for the remaining projects in your solution to enlist them all in Aspire orchestration.

The `AppHost` project is responsible for orchestration.
You can define your entire application stack in a single, readable file.
Running the `AppHost` project from Visual Studio will start the required applications and services.

Here's the setup for my application using Aspire:

```csharp
IDistributedApplicationBuilder builder = DistributedApplication.CreateBuilder(args);

var postgres = builder.AddPostgres("contentplatform-db")
    .WithPgAdmin();

var rabbitMq = builder.AddRabbitMQ("contentplatform-mq")
    .WithManagementPlugin();

builder.AddProject<Projects.ContentPlatform_Api>("contentplatform-api")
    .WithReference(postgres)
    .WithReference(rabbitMq);

builder.AddProject<Projects.ContentPlatform_Reporting_Api>("contentplatform-reporting-api")
    .WithReference(postgres)
    .WithReference(rabbitMq);

builder.AddProject<Projects.ContentPlatform_Presentation>("contentplatform-presentation");

builder.Build().Run();
```

The Aspire version is much more concise and readable.
Adding new services or changing configurations is straightforward.
You also get built-in observability.
Aspire includes tools for logging, metrics, and distributed tracing out of the box, making it easier to monitor and debug your applications.

When you run the application, you can see your applications and services on the Aspire dashboard:

<Image
  src="blogs/mnw_107/content_platfrom_resources.png"
  width={1919}
  height={583}
  alt="Aspire dashboard resource views showing the application services and containers running."
/>

### Orchestration - The Bad Parts

There are a few things I don't like with the current Aspire setup.

The `AppHost` project needs to reference all other projects to enlist them in orchestration.
If your services are all in one solution, this might be fine.
But what about large microservices systems?

We can go around this limitation by building a Docker image for an external service.
There's an `AddContainer` method that allows us to configure container resources.
However, we won't be able to debug these services.

The `ServiceDefaults` projects needs to be visible to all other applications.
Again, this works perfectly fine if everything is in one solution.
We can also distribute this project as a NuGet package for complex systems.

## .NET Aspire Integrations

If you're wondering how I configured PostgreSQL and RabbitMQ in the previous example, this is made available using Aspire Integrations.
These are NuGet packages that allow you to integrate with popular services, such as Redis or PostgreSQL.
Aspire integrations take care of many cloud-native concerns for you, like adding health checks and telemetry.

You can right-click on the `AppHost` project and select `Add > .NET Aspire package...` to see the list of available integrations:

<Image
  src="blogs/mnw_107/aspire_integrations.png"
  width={1088}
  height={917}
  alt="NuGet browser view showing a list of .NET Aspire integrations."
/>

If we want to add Redis to our project, we can install the `Aspire.Hosting.Redis` package.
Then, we would configure the Redis integration in the `AppHost` project:

```csharp {5,10}
var builder = DistributedApplication.CreateBuilder(args);

// Other service omitted for brevity

var redis = builder.AddRedis("contentplatform-cache");

builder.AddProject<Projects.ContentPlatform_Api>("contentplatform-api")
    .WithReference(postgres)
    .WithReference(rabbitMq)
    .WithReference(redis);

builder.Build().Run();
```

You can find a list of supported [Aspire integrations](https://learn.microsoft.com/en-us/dotnet/aspire/fundamentals/integrations-overview) in the documentation.

When you configure a resource with a known connection string format, Aspire automatically injects an environment variable.
The connection string name will have the same name as the respective resource.

- `WithReference(postgres)` produces `ConnectionStrings__contentplatform-db="<VALUE>"`
- `WithReference(rabbitMq)` produces `ConnectionStrings__contentplatform-mq="<VALUE>"`
- `WithReference(redis)` produces `ConnectionStrings__contentplatform-cache="<VALUE>"`

This lets you use logical connection string names when configuring your services:

```csharp
builder.Services.AddDbContext<ApplicationDbContext>(o =>
    o.UseNpgsql(builder.Configuration.GetConnectionString("contentplatform-db")));
```

## Service Defaults and OpenTelemetry

One of Aspire's killer features is its built-in observability stack.
It integrates [**OpenTelemetry**](introduction-to-distributed-tracing-with-opentelemetry-in-dotnet), providing distributed tracing, metrics, and logging out of the box.

When you enlist a project in .NET Aspire orchestration, there are some updates made to the `Program` file automatically:

- `AddServiceDefaults` is called to configure OpenTelemetry, health checks, and service discovery
- `MapDefaultEndpoints` is called to expose the health check endpoint

```csharp {3,9}
var builder = WebApplication.CreateBuilder(args);

builder.AddServiceDefaults();

// Other code omitted for brevity

var app = builder.Build();

app.MapDefaultEndpoints();

// Other code omitted for brevity

app.Run();
```

You can customize `AddServiceDefaults` according to your requirements.
For example, if you're using MassTransit, you can add the respective tracing configuration for this library.

Here's the distributed traces view on the Aspire dashboard.
You can see a `POST` request hitting the `contentplatform-api` service, publishing an `ArticleCreatedEvent`, and consuming that message in the `contentplatform-reporting-api` service.

<Image
  src="blogs/mnw_107/content_platfrom_traces.png"
  width={1920}
  height={550}
  alt="Aspire dashboard traces views showing one distributed trace."
/>

For local development, the .NET Aspire dashboard provides a UI for viewing telemetry data.
In a production environment, you can configure the OpenTelemetry server to receive telemetry data using the `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable.

## Deploying .NET Aspire Applications

.NET Aspire simplifies the deployment process for distributed applications, especially when targeting Azure.
To deploy an Aspire application, you first generate a **manifest file** using the `dotnet run` command with specific parameters.
This manifest is a JSON file that describes all the resources defined in your Aspire project, including services, databases, and other dependencies.

<figure>
  <Image
    src="blogs/mnw_107/aspire_manifest.png"
    width={3659}
    height={1821}
    alt=".NET Aspire manifest JSON file example."
  />
  <figcaption>
    Source:
    [Microsoft](https://learn.microsoft.com/en-us/dotnet/aspire/deployment/manifest-format)
  </figcaption>
</figure>

Deployment tools can use the manifest to set up the necessary infrastructure in your target environment.
Aspire generates the required configuration for Azure Container Apps or Kubernetes for Azure deployments.
It handles tasks like setting up networking, scaling services, and configuring monitoring automatically.

Here's a simple example of generating a manifest:

```
dotnet run --project ContentPlatform.AppHost\ContentPlatform.AppHost.csproj `
    -- --publisher manifest --output-path ../aspire-manifest.json
```

This command creates a JSON manifest file that deployment tools can use to set up your application in the cloud or on-premises infrastructure.

You can learn more about [Aspire deployment](https://learn.microsoft.com/en-us/dotnet/aspire/deployment/overview) in the documentation.

## Summary

I've used .NET Aspire a lot lately, and I'm genuinely impressed.

Aspire makes building complex systems much easier.
I can set up a distributed system with just a few lines of C# code.
This is much simpler than using Docker Compose.
The built-in observability and monitoring tools are also great.

While .NET Aspire is now production-ready, the ecosystem around it is still growing.
Developers, particularly those new to cloud-native concepts, might face a learning curve.

Should you adopt Aspire in your .NET projects?

If you're building distributed applications, especially for Azure, I'd say give it a try.
However, you might want to evaluate carefully if you work on simpler applications or use non-Azure cloud services.

That's all for today.

See you next week.

**P.S.** You can find the source code for this example in [**this repository**](https://github.com/m-jovanovic/aspire-orchestration).

---


Previous issues of The .NET Weekly