Minio and Azure

I love to run OSS stuff. It's fun, I make a few $ a month reselling some services, etc. Everything I do has to have storage though and every OSS platform out there that has a storage need supports AWS's S3 protocol. Me being, well, me, that's not gonna do, so I am a prolific user of Minio, which allows me to set up an S3 compatible storage layer on Azure.

For the longest time, I had a VERY slick setup - we're talking YEARS. I had https://minio.azurewebsites.net and it pointed to ME ME ME! It was pretty simple really, an Azure Web App Container instance running in Linux mode with the following command line and environment variables set:

gateway azure
ENV=MINIO_ROOT_USER
ENV=MINIO_ROOT_PASSWORD

Those two bottom env variables pointed to an Azure Storage Account Blob Container which did the trick. Ooodddles of TBs of data slowly trickled in (backups, etc). The docker container itself was pointing to Docker Hub and Minio's "latest" release, so every now and then it would update itself and I'd automatically have all the shiny bells and whistles. I also REALLY enjoyed using the Policy object on storage accounts so I could auto prune things more than X months old AND had it set to Cool tier. Everything was grand! Until it stopped working - like KABOOM.

Having six kids, I forgot about it cause it was easier to just set up another Minio server elsewhere and figure it out later. Until I realized: storage is cheap in Azure - like comically cheap, and, well, I'm cheap. So, this Thanksgiving, I decided I'd figure out what was wrong.

Problem 1: Minio isn't hosted on Github anymore. Hmph. It's over on Quay now. Okay, surely that's an easy change of the Container Type settings:

Hmmm - okay, Azure Web Apps only likes Docker. So, perhaps I can use a Private Registry. Hmmm - that won't work cause I don't want to deal with Red Hat (owner of Quay) and it's not my registry, it's Minio's, so that won't work. Oh, I know! I'll create an Azure Container Registry and IMPORT Minio from Quay into my registry:

Nice - and the command to import someone else's container is nice too:

az acr import --name <myregistry> --source quay.io/minio/minio --image minio:latest

Reference

Cool - so now I can stitch that up into an updated Azure Web Apps container setup and I'll be done right? Nope:

Problem 2: My startup file "gateway azure" needs my environment variables passed to it. For reasons I can't seem to figure out, Azure Web Apps seems to have stopped injecting my application settings into the environment variables. Always worked before, but no more. Docs say it should work. It doesn't. Poop.

New Idea! I'll use Azure Container Instances. This seemed pretty straight forward:

az container create --name minio --resource-group $RES_GROUP --image \ --environment-variables MINIO_ROOT_USER=me MINIO_ROOT_PASSWORD=really_secure

Problem 3: I need a WHOLE bunch more "stuff" to get az container create to grab my ACR image - it has to log in. When using the GUI, it just "happens" but in the CLI, gotta specify. So, just use the GUI? For reasons I'll explain in a bit, that wouldn't work, so, CLI it is!

To do this, I'd need to specify a metric ton more parameters. So, I referenced this shiny article to string things together.

Problem 4 and 5: I wanted to gussy this up, so I had experimented with getting Minio deployed in a private vnet and make it accessible to Azure App Gateway, but that meant lots more $ than I wanted to spend and make my commands way more complex. Also: "You can't pull images from Azure Container Registry deployed into an Azure Virtual Network at this time." So, that was annoying. Also, I needed to now specify the ACR username and password cause: "You can't authenticate to Azure Container Registry to pull images during container group deployment by using a managed identity configured in the same container group." Uh....okay.

After following a laundry list of above shiny article procedures, I had everything down:

1. A KeyVault to store my U/P to ACR
2. A Service Principal that had rights to the KeyVault to get said U/P
3. The real problem was that the MS Docs on the above shiny article are wrong on this line:

az ad sp show --id http://$ACR_NAME-pull --query appId --output tsv

Can't query by Id anymore, so I had to go hunt the damn SPN ID down. Not difficult, but I chased my tail for almost an hour sifting through az ad sp cli syntax.

Now, I had a better command:

az container create --name minio --resource-group $RES_GROUP \ --image $ACR_LOGIN_SERVER/minio:latest \ --registry-login-server $ACR_LOGIN_SERVER \ --registry-username $(az keyvault secret show --vault-name $AKV_NAME -n $ACR_NAME-pull-usr --query value -o tsv) \ --registry-password $(az keyvault secret show --vault-name $AKV_NAME -n $ACR_NAME-pull-pwd --query value -o tsv) \ --environment-variables ...

Problem 6: after studying the output of the container logs for a while, I determined that even in this mode, "gateway azure" just wasn't working right. Examples abounded of using Azure Storage (with Azure Files), so I gave up on Gateway and thought, hey, I could just mount a drive in my container.

I needed a few more parameters to make this happen, but I figured it out thanks to THIS shiny article. Four more parameters to my CLI and shazam - Minio was alive!!!

Except not quite. My URL was http://<somedamnipaddress> Hmm, okay, well, I have to do this over HTTPS cause I'm legit sending data I don't want snooped out of the sky (backups, etc). So, how do I add HTTPS to my Azure Container Instance? Surely there's a button for that, right? RIGHT??? Of course not. Problem 7 is: in order to do HTTPS in ACI, one must add a sidecar, basically, an additional container. FFS. Okay, I know of one that can do the HTTPS and grab a Let's Encrypt cert for me - Caddy!

I knew that Azure Container Instances were actually (behind the scenes) called Azure Container Instance Groups, and that I could run more than one container within my deployment. So, I russled up another CLI command, thinking I just run a new az container create and it'd add it.

Wrong:

az container create --name minio --resource-group $RES_GROUP --image...minio

Followed by another:

az container create --name minio --resource-group $RES_GROUP --image...caddy

Replaces the running container of minio with a running container of caddy. So...using CLI, how do I get multiple containers going?

Problem 8: you cannot set up multiple containers in a single ACI group with multiple az container creates. In truthiness, I'm not shocked, but the resolution brought out tons of rage:

YAML deployment (or ARM/BICEP which is almost as bad). I don't HATE coded deployments, I just hate that my simple Azure Web App single container now has a flipping template to it. Okay, I can do this, I just need some help. Found a helpful blog and site.

Problem 9: YAML is the devil.

Problem 10: YAML must die.

Problem 11 (and two days of fighting later): YAML must still die...but I got it slung together with the "annoyed that I had to use this helper helper." Did I mention that YAML must die? TABS aren't allowed but spacing is critical. Oye.

https://gist.github.com/doodlemania2/ee73a20dd9b96b7093eca9b6561633f8

So, because Minio decided to move to Quay, Azure App Services decided to not send in my ENV variables anymore, and a host of other reasons, I'm now, FINALLY up and running with Minio again, and this time, in a neat shiny new, and potentially? cheaper setup with Azure Container Instances.

Problem 12: Now I have to remind myself to update my own ACR artifacts each time they release a patch. Good thing there's an automation script for that...I just have to go write it :)