Shaken Fist uses artifacts as disk templates for new instances. You therefore need to have a least one artifact before you can create your first instance, although there is a shorthand notation to create that artifact during instance creation.
The following artifact types exist:
- images: these are generally downloaded from the Internet, although they can also be created via an upload mechanism (see the artifact uploads section below for a detailed introduction to uploading images to the cluster).
- snapshots: these are created by taking a copy of the disk(s) of a running instance.
- labels: labels are a bit like symbolic links, although they still have versioning like other forms of artifact.
- other: a catch all for artifacts which don't fall into any of the other categories. For example captured instance console output archived after an instance was deleted.
Behind the scenes, artifacts are references to blobs. You can think of them as
symlinks if you'd like. All types of artifact support versioning. This is implemented
by having each artifact have a list of blobs. You can see this in the output of a
sf-client artifact show ... command:
$ sf-client artifact show 3420f4ac-529a-4b34-b8d8-c05a838b9e0c uuid : 3420f4ac-529a-4b34-b8d8-c05a838b9e0c namespace : ci-images type : label state : created source url : sf://label/ci-images/debian-11 current version blob uuid: cc6a6a96-8182-474a-ab31-45f1f9310b44 number of versions : 3 maximum versions : 3 shared : False Versions: 4 : blob f6090574-321c-4dec-b381-0caf001eeba5 is 2964.1MB 5 : blob 77b9032a-0d3e-4cc2-bb73-1730ad3c3cb0 is 2960.1MB 6 : blob cc6a6a96-8182-474a-ab31-45f1f9310b44 is 2950.4MB in use by instances 78d566f1-c664-46d9-aa46-cf717aa63743
Here we can see a few things:
- The artifact is a label in the ci-images namespace.
- There is a source URL, which is how you would reference this artifact when starting an instance.
- There are three versions of the artifact currently stored (
number of versions), which is the maximum (
maximum versions). However, there have been six versions of this artifact ever (as shown by the indices of the versions being 4, 5, and 6).
- The most recent version is currently in use by an instance.
Creating an image artifact
Image artifacts are created by specifying the URL of an image to download. This
can be either in the form of an explicit request to cache a URL with a
sf-client artifact cache command, or implied by specifying the URL of the
image in the specification for an instance.
There is also a shorthand form of image URLs where you are using an image from
https://images.shakenfist.com -- in that case you can use urls like
so specify the latest version of a Debian 11 base image. The valid shorthands
are those listed in the top level directory listing of that site. At the time of writing this is:
- centos (shorthand for centos:8-stream)
- debian (shorthand for debian:11)
- ubuntu (shorthand for ubuntu:20.04)
These images are updated nightly by an automated job from https://github.com/shakenfist/images.
Whenever you specify a URL for an image (either a new
cache command or at
instance start), the URL is checked. If the image has changed a new version is
downloaded, otherwise the already cached version is used.
You can also create an image artifact by uploading it, but that's complicated enough that its covered separately in the developer guide section on artifacts.
Creating a snapshot artifact
These are created by the
sf-client instance snapshot command. You can specify
which disk to snapshot on a multi-disk instance with the
--device flag. Repeated
snapshots of the same instance will result in multiple versions of the one
artifact being created. Note that there is an artifact per device snapshotted, so
a single snapshot of a multi-disk instance will create multiple artifacts.
Creating label artifacts
So what's a label? Well downloading new versions of images automatically is great,
but what if I want to ensure the version from two weeks ago that I tested is the
one I use? Or what if I want to refer to my favourite instance snapshot by
something more convenient than a snapshot URL like
sf://instance/78d566f1-c664-46d9-aa46-cf717aa63743/vda? Well, labels are the
answer to those questions.
Labels are artifacts where you specify what the new version should be. So in the download example you'd test an image version and when you decide that its right, you'd add that version's blob UUID to your label of known tested versions.
Blobs are reference counted, so even if the image artifact ages out a version, having that version referred to in a label artifact protects it from deletion.
An example of labelling a known good version of an artifact would be something like this:
$ sf-client artifact show 3420f4ac-529a-4b34-b8d8-c05a838b9e0c uuid : 3420f4ac-529a-4b34-b8d8-c05a838b9e0c namespace : ci-images type : label state : created source url : sf://label/ci-images/debian-11 current version blob uuid: cc6a6a96-8182-474a-ab31-45f1f9310b44 number of versions : 3 maximum versions : 3 shared : False Versions: 4 : blob f6090574-321c-4dec-b381-0caf001eeba5 is 2964.1MB 5 : blob 77b9032a-0d3e-4cc2-bb73-1730ad3c3cb0 is 2960.1MB 6 : blob cc6a6a96-8182-474a-ab31-45f1f9310b44 is 2950.4MB in use by instances 78d566f1-c664-46d9-aa46-cf717aa63743 ...test version 6 with blob UUID cc6a6a96-8182-474a-ab31-45f1f9310b44... $ sf-client label update my-tested-thing cc6a6a96-8182-474a-ab31-45f1f9310b44
If the label
my-tested-thing does not exist, it will be created the first
time you update it.
Listing and deleting artifacts
Artifacts follow the same user interface patterns as other objects. That is, you can list artifacts with this command:
sf-client artifact list
And you can delete artifacts with a command like this:
sf-client artifact delete ...name.or.uuid...
Note that deleting an artifact does not necessarily imply deleting the associated blobs. If those blobs are in use by other objects (artifacts, instances, and so on) then they remain stored by the cluster until there are no remaining references.
Additionally, you can also delete all artifacts in a given namespace by making
a HTTP DELETE request to /artifacts REST API endpoint, which is also provided by
delete_all_artifacts() method in the Python API client. This functionality
is not currently exposed in the command line client.
Finally, deleting a namespace implies deleting all artifacts within that namespace, so show care when deleting namespaces to ensure they no longer contain any data you are fond of.
Controlling the number of versions
You can also control the number of versions stored by an artifact with the
sf-client artifact max-versions command.
You can control the number of copies of a given blob are stored in the cluster
as well. This protects against machine or disk failures causing data loss. The
default number of replicas is 2, but this is not configurable per-blob. It is
configured with the
BLOB_REPLICATION_FACTOR configuration variable.
Artifact uploads and downloads
Artifacts may also be uploaded and downloaded. This means you can extract a snapshot from your cluster for offline backup (or movement to another cloud), or upload an image built with a tool like Hashicorp Packer.
To upload an artifact, use the
sf-client artifact upload command. To download
an artifact, use the
sf-client artifact download command.
Shaken Fist will calculate a checksum for the new blob created by an upload, and if it already has a blob matching that checksum it will only store the data once. This makes uploading a given artifact more than once effectively free apart from a small amount of etcd storage.