Michael Mattsson

Tech Preview of Nimble Linux Toolkit 2.1: New Docker features!

Blog Post created by Michael Mattsson Employee on Feb 21, 2017

It has been a busy few weeks in Nimble engineering while polishing the next incremental update of our Docker Volume plug-in. We’re striving to abstract all our core features to the Docker API to make end-users self sufficient for all their persistent storage needs. NLT-2.1 and the next release of NimbleOS which NLT-2.1 depends on, will ship in a few weeks.

 

QoS limits

NimbleOS have very sophisticated IO scheduling to reduce contention and “noisy” neighbors. This is sufficient for single tenant type workloads but does not work well for our customers who wants to make money off their investments such as service providers. That's why NimbleOS 4.0 introduces QoS (Quality of Service) limits which allows administrators to manually govern performance between Nimble volumes and folders (a folder is a logical grouping of volumes). This allows for very intricate policing of IO where a tenant may purchase 5000 IOPS or 1Gbit of throughput assigned to a group of volumes and further pin IOPS and throughput within his family of volumes. The service provider may then resell different tiers of performance as well as capacity to meet the tenants needs in the ever emerging pay-only-for-what-you-need economy. Not having these governors in place gives tenant #1 a false sense of performance guarantee and will most likely start complaining when tenant #2 moves in on the same system.

 

We’re bringing these performance governors to the Docker API. Combining QoS limits with our current capabilities opens up a few interesting use cases.

 

Isolating Docker Swarm clusters to different performance tiers

For large deployments it makes a lot of sense to understand how much IO performance is available to household with and divide that among the different workloads. Production clusters may create ungoverned Docker Volumes but may run the image registry on governed Nimble volumes as pulling images is not desirable to impact transactional application workloads running of the Docker Volumes. Test/dev and QA clusters have its own set of default governors to allow a single Nimble array to cater a diverse population of IO needs.

 

In the below example there’s roughly 20K IOPS to household with for an entry level array using a synthetic transactional workload. In the ‘centos’ folder is where the production cluster Docker Volumes reside and in the ‘xenial’ folder is the test environment. To ensure the production cluster gets all the juice, the test environment is throttled significantly as the performance delivered there is not mission-critical.

centos.png

Fig. Folder 'centos' should be prioritizied, doubling performance by throttling folder 'xenial'

 

xenial.png

Fig. Performance observed from the 'xenial' folder, stealing IO from the important 'centos' folder.

 

In this particular example, performance doubled for the mission-critical workload when filtering out unimportant Docker Volumes.

 

Clone production volumes with performance caps for test/stage/qa

In data intense CI/CD build pipelines where tests run on clones of production there's always risk to impact production performance negatively. What we enable is the ability to set the limits on the volume during the clone operation which can easily be integrated into the build pipeline when preparing tests.

 

admin@nimble:~$ docker volume create -d nimble --name prod

prod

admin@nimble:~$ docker volume create -d nimble --name testing -o limitIOPS=500 -o cloneOf=prod
testing

admin@nimble:~$ docker volume inspect testing | json 0.Status.LimitIOPS
500

 

Being able to fine tune IO performance brings a whole new dimension to planning how data is being accessed. Cramming more work onto an array becomes less risky as it's possible to throttle undesired data access patterns and govern important workloads to roam freely on the available resources. Still using the default automatic QoS for fair balancing of IO between volumes.


Protection templates

Data without backups is not really data when disaster strikes. Local snapshots provide excellent RPO/RTO and asynchronous remote replication provide DR capabilities along with long-term archiving. This bread & butter capability is now available as an option for Docker Volumes where it’s possible to associate a pre-defined protection template during volume creation.


admin@source:~$ docker volume create -d nimble --name important-data -o protectionTemplate=Retain-30Daily

important_data

admin@source:~$ docker volume inspect important-data | json 0.Status.VolumeCollection

{

  "Description": "Provides daily snapshots retained for 30 days",

  "Name": "important-data.docker",

  "Schedules": [

    {

      "Days": "all",

      "Repeat Until": "23:59",

      "Replicate To": "group-sjc-array869",

      "Snapshot Every": "1 minutes",

      "Snapshots To Retain": 30,

      "Starting At": "00:00",

      "Trigger": "regular"

    }

  ]

}

 

While this is pretty straight-forward, there are some neat use cases opening up for having Docker nodes accessing the downstream Nimble group. Volumes are offline at the replica destination and can’t directly be seen by the Docker Volume plug-in but since we provide the ability to clone any Nimble volume in any state directly from the Docker API, that data is indirectly available on the downstream array for various uses.


Offload data validation and dev/test/qa to replicas

In CI/CD build pipelines with transactional workloads using load tests it may not be practical to run those tests on the primary array and instead run those tests during office hours on a replica set. Another good use would be to do data validation, such as Oracle 'dbv' or similar.

 

Plan and test DR strategies

Since data and applications are abstracted from each other using containers it’s quite convenient to build DR solutions around containerized applications. When disaster strikes, containers could be pre-pulled and built on off-site Docker hosts and taken online with replicas of the primary data. Testing these procedures could be done in an automatic fashion where the only manual intervention at a cutover would be to redirect front-end traffic. The less manual steps to perform during a live DR scenario the better.

 

On the destination group (group-sjc-array869) in the example above there’s a separate Docker cluster connected. Replica destinations are always offline as mentioned and they won’t show up on the remote Docker Engines. Since we can figure out the actual Nimble volume name on the source, cloning on the destination becomes easier.

 

admin@source:~$ docker volume inspect important-data | json 0.Status.VolumeName

important-data.docker

 

Flipping over to a Docker Engine on the remote site:

admin@destination:~$ docker volume create -d nimble -o importVolAsClone=important-data.docker --name replica-data
replica-data

 

Comprehensive data protection capabilities is essential for any production workload, persistent storage for containers is no exception. This is a first pass to bring this functionality to Docker, more features and capabilities are in the works to simplify these workflows further.

 

Locally scoped volumes

Docker Swarm introduced the concept of a service. An abstraction layer to define an application for scaling and availability. Docker images with volume definitions spits out anonymous local volumes and attaching a named globally scoped volume to the service will cause conflicts unless using a fully clustered aware filesystem (and application for that matter). Having these anonymous volumes co-mingled with the docker runtime might be undesirable as it's difficult to govern use of /var/lib/docker and it's best used only for images and runtime configuration data.

 

The default Nimble Docker Volume plug-in is a globally scoped driver which allows any node in the Docker Swarm to attempt mounting it. In NLT-2.1 we introduce a ‘nimble-local’ driver which creates locally scoped volumes which may create identically named volumes in the Docker Swarm but only accessible on the node it was created.

 

The following example will deploy a single Minio instance as a single replica and two global services will be created, one service replicates to the nimble-local volume and one service serve the data read-only. This sure copies a lot of data around but it will be deduped on the Nimble array. It yields very trivial horizontal scale and excellent resiliency. The neat thing about a global service is that it will automatically scale as more nodes are added to the cluster and Docker Volumes are created on the fly.

 

version: "3"

services:

  # this is a single replica minio instance

  minio:

    image: minio/minio:latest

    volumes:

    - export:/export

    command: server /export

    environment:

      MINIO_ACCESS_KEY: ${MINIO_ACCESS_KEY:-minio}

      MINIO_SECRET_KEY: ${MINIO_SECRET_KEY:-miniostorage}

    ports:

    - "9000:9000"

    deploy:

      replicas: 1

 

  # this is a global minio client service that mirrors the minio instance

  mirror:

    image: minio/mc:latest

    volumes:

    - mirror:/mirror

    entrypoint: /bin/ash

    command: -c "/go/bin/mc config host add minio http://minio:9000 ${MINIO_ACCESS_KEY:-minio} ${MINIO_SECRET_KEY:-miniostorage} && /go/bin/mc mirror --quiet --force --remove --watch minio/docker /mirror"

    deploy:

      mode: global

    depends_on:

    - minio

 

  # this runs a read-only nginx copy of the mirrored data as a global service

  web:

    image: thegillis/nginx-as-root-hack

    volumes:

    - mirror:/usr/share/nginx/html:ro

    ports:

    - "80:80"

    deploy:

      mode: global

    depends_on:

    - mirror

 

volumes:

  # this is the minio source volume

  export:

    driver: nimble

    driver_opts:

      size: 768

      description: "Minio export"

 

  # this is the nimble-local volume that is mounted on each swarm host

  mirror:

    driver: nimble-local

    driver_opts:

      size: 768
      description: "Minio client and nginx frontends"

Fig. An example Docker Compose v3 YAML file - stack.yml

 

Bringing this application up with a single command:

admin@nimble:~$ docker stack deploy -c stack.yml migx


Observing the stack in Docker UCP:

migx_stack.png

Fig. Displaying an application stack in Docker UCP

 

So, where are you in your containerization project and how can we help you overcome your persistent storage challenges?

Outcomes