WaitForHttp not working for CosmosDb emulator container and no timeout

[diegosasw]

I am using the library to spin up a CosmosDb emulator.
The problem with this CosmosDb emulator is that it takes a while until it's ready to be used, so I need a good mechanism to wait.

I have tried the following because I know that it shows that message at the end. But, although it waits for it, it's not enough, as there is still a gap between the log message is added until the emulator is fully ready.

var cosmosDbService =
			new Builder()
				.UseContainer()
				.WithName(Constants.ContainerNames.CosmosDb)
				.UseImage("mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest")
				.ExposePort(8081, 8081)
				.ExposePortRange(10250, 10255)
				.WithEnvironment(
					"AZURE_COSMOS_EMULATOR_IP_ADDRESS_OVERRIDE=127.0.0.1")
				.DeleteIfExists()
				.RemoveVolumesOnDispose()
				.UseNetwork(network)
				.WaitForMessageInLog("Started 11/11 partitions")
				.Build();

I think the best way is to wait for an Https file to be available such as the https://127.0.0.1:8081/_explorer/emulator.pem but I suspect the https may be causing problems, because it keeps waiting indefinitely without any errors when I run the service.

var cosmosDbService =
	new Builder()
		.UseContainer()
		.WithName(Constants.ContainerNames.CosmosDb)
		.UseImage("mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest")
		.ExposePort(8081, 8081)
		.ExposePortRange(10250, 10255)
		.WithEnvironment(
			"AZURE_COSMOS_EMULATOR_IP_ADDRESS_OVERRIDE=127.0.0.1")
		.DeleteIfExists()
		.RemoveVolumesOnDispose()
		.UseNetwork(network)
		.WaitForHttp(
			"https://localhost:8081/_explorer/emulator.pem", contentType: "text/plain")
		.Build();

However, I've added the cosmosDb certificate to the cert manager with elevated permissions

curl -k https://localhost:8081/_explorer/emulator.pem > emulatorcert.crt
certutil.exe -addstore root emulatorcert.crt

to ensure it is trusted. That didn't make any difference.
I've also tried

.WaitForHttp("https://localhost:8081")

which returns a 401 when the container is ready, but no luck either. It keeps waiting.

Is there any way I can see what's going on or any better approach for this scenario? The strange thing also is that if I set a small timeout, it timesout, but if I set something like 60000 it does not time out.

PS: The .WaitForHealthy() does not wait enough.

[diegosasw]

I had a look at how testcontainers for CosmosDb is waiting, and it follows the strategy of waiting for https://localhost/_explorer/emulator.pem until there is a successful response

https://github.com/testcontainers/testcontainers-dotnet/blob/a0f1f7694b4602aa2ba7da6f167ddc4a56670ecc/src/Testcontainers.CosmosDb/CosmosDbBuilder.cs#L70

Not sure about the magic of having the url without any port.

[mariotoffia]

Hi @diegosasw, could you try to do a "manual" wait instead to see if there's a bug or otherwise in FluentDocker.

You may add any "lambda" by using the .Wait("", (service, count) => "lambda") where the lambda function return 0, the wait is over and a positive integer is the number of milliseconds to wait before trying the lambda again.

A pseudo example:

    // Add the wait instead of WaitForHttp in the container builder
    fd.Wait("Wait for CosmosDb", (service, count) => MyWaitFunc(count, service))
    
    // The custom wait function
    private static int MyWaitFunc(int count, IContainerService service)
    {
      if (count > 10)
        throw new FluentDockerException("Failed to wait for cosmosDb");

      var ep = service.ToHostExposedEndpoint("80/tcp");
      
      // Either manual
      var response = await $"https://{ep}/my_file.pem".Wget();

     // or use the FluentDocker WaitForHttp to see why it is failing (and fix the bug)     
      service.WaitForHttp(...)
   }

The http wait function is pretty simple:

    public static void WaitForHttp(this IContainerService service, string url, long timeout = 60_000,
      Func<RequestResponse, int, long> continuation = null, HttpMethod method = null,
      string contentType = "application/json", string body = null)
    {
      var wait = null == continuation ? timeout : 0;
      var count = 0;
      do
      {
        var time = Millis;

        var request = url.DoRequest(method, contentType, body).Result;
        if (null != continuation)
        {
          wait = continuation.Invoke(request, count++);
        }
        else
        {
          time = Millis - time;
          wait = request.Code != HttpStatusCode.OK ? wait - time : -1;
        }

        if (wait > 0)
          Thread.Sleep((int)wait);

      } while (wait > 0);
    }

Cheers,
Mario

[diegosasw]

Thanks for your help, it seemed, indeed, a problem with the SSL, but thanks to your suggestion I managed to add a custom Wait logic that works well. Here are the extensions methods

public static class ContainerBuilderExtensions
{
	public static ContainerBuilder ExposePortRange(this ContainerBuilder containerBuilder, int start, int end)
	{
		for (var port = start; port <= end; port++)
		{
			containerBuilder.ExposePort(port);
		}

		return containerBuilder;
	}
	
	public static ContainerBuilder WaitForHttps(
		this ContainerBuilder builder, 
		string url, 
		bool ignoreSslErrors = false,
		int retries = 40, 
		int delayMilliseconds = 5000)
	{
		return builder.Wait("Wait for Https", (_, count) =>
		{
			if (count > retries)
			{
				var secondsWaited = count * (delayMilliseconds / 1000);
				throw new FluentDockerException($"Failed to wait for {url} after {secondsWaited} seconds");
			}

			var httpClientHandler =
				ignoreSslErrors
					? new HttpClientHandler { ServerCertificateCustomValidationCallback = (_, _, _, _) => true }
					: new HttpClientHandler();

			using var client = new HttpClient(httpClientHandler);
			try
			{
				var response = client.GetAsync(url).Result;
				if (response.IsSuccessStatusCode)
				{
					return 0;
				}
			}
			catch (Exception)
			{
				// ignored
			}

			Thread.Sleep(delayMilliseconds);
			return 1;
		});
	}
}

And it can be used with CosmosDB (or any other service which requires waiting for an https url) specifying that SSL errors should be ignored

var cosmosDbService =
	new Builder()
		.UseContainer()
		.WithName(Constants.ContainerNames.CosmosDb)
		.UseImage("mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest")
		.ExposePort(8081, 8081)
		.ExposePortRange(10250, 10255)
		.WithEnvironment(
			"AZURE_COSMOS_EMULATOR_IP_ADDRESS_OVERRIDE=127.0.0.1")
		.DeleteIfExists()
		.RemoveVolumesOnDispose()
		.WaitForHttps("https://localhost:8081/_explorer/emulator.pem", ignoreSslErrors: true)
		.UseNetwork(network)
		.Build();

cosmosDbService.Start();

[mariotoffia]

Thanks for publishing your solution, I'll be sure to add the verifySSL {true, false} flag to the WaitForHttp in future.

Cheers,
Mario