Skaffold is powerful and flexible command line tool to automate the workflow for fast Source to Kubernetes development and continuous deployment to Kubernetes Cluster.
Its great strength is its pluggable architecture. It allows a very simple entry into the automation as shown in this article but allows to add to the deployment workflow as projects gain in complexity.
Matching the project requirements, Skaffold can be run on demand to build and deploy once off or put in the mode to monitor for changes in code and automatically build and deploy once it’s detected. The configuration options are highly adaptable to the specific requirements – it can for example be configured to deploy multiple microservices to multiple cluster every time a change is detected.
The following diagram shows the architecture that forms the basis of this article.
A development environment is hosted on an Oracle Cloud Infrastructure Virtual Machine. It is used to develop code and a local single node Kubernetes Cluster (Minikube) is used for testing. Changes to the local source code will be deployed directly on the local Kubernetes cluster. On demand the current state can be pushed to a Cluster based on Oracle Cloud Infrastructure Container Engine for Kubernetes (OKE). This is an end-to-end walkthrough of a basic, but powerful setup using Skaffold to allow rapid development and deployment. It can easily be extended to allow production deployment workflows.
Environment Setup - VCN and OKE Setup
For this example a simple Oracle Cloud Infrastructure Container Engine for Kubernetes (OKE) Cluster is a great choice to get started as requires next to no configuration for a production ready deployment. The workflows take the configuration of VCNs etc off your hands, as this is not the focus of this exercise.
As this is a fresh environment we will create everything from scratch including VCNs etc.
For this example a private cluster is the best choice as we don’t want to expose the workers unnecessarily to increase security. You can choose to include the Add On Tiller (Helm) for this deployment, even though it is not used in this example - as a basis to expand the Skaffold workflows at a later stage. The Kubernetes Dashboard can be enable for easier troubleshooting.
Environment Setup – Development VM Setup
Avoiding additional configuration the basis for the Development VM is an Oracle Marketplace Image (https://cloudmarketplace.oracle.com/marketplace/oci). It can be found in the Marketplace under the Operating System Section. Make sure to place this host in the same subnet of the VCN as the Kubernetes Workers in your OKE Cluster, if you are following the example and choose a private Kubernetes OKE cluster for improved security - https://www.oracle.com/a/ocom/docs/bastion-hosts.pdf
Additionally we need to install docker and git – this is easily done following the commands below – for a detailed description see https://blogs.oracle.com/blogbypuneeth/a-simple-guide-to-docker-installation-on-oracle-linux-75
sudo yum install git docker-engine
systemctl enable docker
systemctl start docker
sudo usermod -aG docker opc
The kubernetes tools for this example are kubectl and minikube – the installation is straight forward as shown below – the Kubernetes documentation is a great source to find additional details in case of issues here https://kubernetes.io/docs/tasks/tools/install-minikube/
curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl
curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 \
> && chmod +x minikube
sudo mkdir -p /usr/local/bin/
sudo install minikube /usr/local/bin/
minikube start --vm-driver=kvm2
Make sure minikube and kubectl start up correctly by executing:
The installation of Skaffold is also very straight forward – for further details the manual is a great place to start https://skaffold.dev/docs/install/
curl -Lo skaffold https://storage.googleapis.com/skaffold/releases/latest/skaffold-linux-amd64
chmod +x skaffold
sudo mv skaffold /usr/local/bin
As this article is focusing on the setup and configuration part and not the actual development part, we will simply use one of the excellent provided examples on github.com.
git clone https://github.com/GoogleContainerTools/skaffold
We will be using the getting started example, which is a very simple Go based example. The key takeaway here is that the application is simply printing out “Hello World” to the console. This allows us to easily see the changes to the application as it is being deployed.
The other key component to take note of is the skaffold.yaml file. It tells Skaffold exactly what to execute as the deployment workflow. Here the artifacts statement instructs Skaffold to build the given docker image and process the manifests starting with k8s-pod.yaml in the same folder. Skaffold can generate the skaffold.yaml file using the command
From the folder where the Skaffold examples where cloned to we start Skaffold in Dev mode and observe the output until the Console output starts. We can see the image being pulled and build.
With the build successful Skaffold automatically runs the deployment and we can see the output.
We open up a separate console window and edit the main.go file to say something like:
In the original window we can see Skaffold getting to work. Because of the code change that was detected Skaffold goes ahead and execute the workflow again with the new code base and deploys the container to the local Kubernetes Cluster – minikube. Once deployed the console output changes to “Hello Oracle”.
The Skaffold dev run can be quit using ctrl-c and the Skaffold cleans up after itself.
Kubectl Config and Context
Working efficiently with the OKE Cluster requires the OCI CLI to be setup. Follow the steps below for detailed input see https://docs.cloud.oracle.com/iaas/Content/API/SDKDocs/cliinstall.htm
bash -c "$(curl -L https://raw.githubusercontent.com/oracle/oci-cli/master/scripts/install/install.sh)"
oci setup config
With the OCI CLI setup completed go to the OKE Cluster Homepage and hit the “Access Kubeconfig” button.
This will help to create the correct kubectl configuration. Please note the cluster_id is unique to each cluster – so you will have to repeat this step, if you recreate your cluster along with the configuration.
The OCI CLI will configure the Kubernetes with multiple contexts – keeping the configuration for the local minikube intact while merging the OKE Cluster configuration as shown below, there should be no need to manually work on the Kubernetes configuration for this example.
Confirm the contexts using the command:
kubectl config view
As highlighted in the screenshot there are now two contexts that can be used to switch between the local minikube and the remote OKE cluster. For the minikube use:
kubectl config use-context minikube
For the OKE Cluster use – the context id is unique for every environment:
kubectl config use-context context-cqtmnjymi4d
kubectl config use-context context-cqtmnjymi4d
Following security best practices we don’t want to expose as little as possible that includes avoiding public registries for this example we use Oracle Cloud Infrastructure Registry (OCIR).
Hence we create a Token for the OKE Cluster to pull the images from our Private Registry. Head to the OCI User management page and locate the user that is used for this example and generate an auth token.
Carefully note down the generated token as we will need it in the next step.
Now we have to login to the registry, to allow Skaffold to place the images here. The location URL is the shortname of the region you want the registry to be in (https://docs.cloud.oracle.com/iaas/Content/Registry/Concepts/registryprerequisites.htm#Availab) followed by .ocir.io with the username of the OCI user and the Token from the step above – note this is not the normal OCI Password.
Now we have to add this secret to our deployment – please note this is again the Auth Token and not the password.
kubectl create secret docker-registry ocirsecret --docker-server=iad.ocir.io --docker-username=' ateamsaas/rkoenn’ --docker-password='_H LRVSxQ] eo9U' --email@example.com'
The secret has to be added to the k8s-pod.yaml manifest file. Simply add:
- name: ocirsecret
We can either change the skaffold.yaml file to match our new registry or set a new default repository as shown:
skaffold config set default-repo iad.ocir.io/ateamsaas/rkoenn-skaffold/scaffold
To visualize the success of the deployment on OKE I have changed the main.go file to read:
fmt.Println("Hello Oracle Cloud Infrastructure!")
As the Remote Deployment to the OKE Cluster only is supposed to show once off deployments we use skaffold in run mode. The parameter --tail allows to redirect the logs to the current console to validate the success of the deployment easily.
Skaffold can of course also be run in DEV mode, if continuous monitoring for code changes and deployment is required.
Kubectl can also show the pod that was created for this deployment:
Conclusion & Next Steps
Skaffold is a very simple and flexible tool to automate the deployment process to kubernetes. It works nicely with the Oracle Cloud Infrastructure Container Engine for Kubernetes Clusters allowing a production grade cluster deployment with minimal configuration changes.
It’s recommended to see this setup as a starting point for further improvement. As a next step looking at more complex deployments with for example Helm and Skaffold is a great starting point.
I also highly recommend my colleagues John Featherlys deployment articles about the application SageMath on OCI.