Using CockroachDB with an Operator
The deployment of Keycloak can use a multi-node CockroachDB setup using the CockroachDB operator.
As an alternative CockroachDB setup, there is also Using CockroachDB single node storage which uses fewer resources.
| Please see Known issues for CockroachDB operator-based setup below. |
Enabling CockroachDB storage
Using a CockroachDB operator installation is available in Keycloak’s new JPA map storage.
It can be enabled via the following settings in the .env file in the provision/kubernetes folder:
KC_DATABASE=cockroach-operator
KC_STORAGE=jpaIn the default configuration, the CockroachDB with an Operator brings up 3 CockroachDB pods.
See Customizing the deployment for a list of all Keycloak configuration options.
After every change, re-run the task command to apply the changes to the minikube instance.
Task will trigger the necessary scripts and will use the minimal steps to update the minikube.
Running rebuild.sh is not required: It will destroy the complete instance and will take a lot longer to complete.
The deployment adds a CockroachDB operator which then spins up three CockroachDB nodes to the minikube setup and removes all other storage pods that are no longer necessary. All connections to the database are TLS encrypted, and Keycloak uses a private key and a certificate to authenticate towards the database.
Keycloak connects to the CockroachDB database pod inside minikube and populates the DB schema. The SQL Pad allows accessing the database via the browser.
Learning more about the CockroachDB operator
The documentation is available online in the CockroachDB docs and the Operator’s GitHub repository.
Verifying the setup
The automated script verifies that Keycloak starts up, which requires that CockroachDB start up first. A user can test the setup by accessing CockroachDB’s console.
Accessing the CockroachDB console
Assuming the command minikube ip returned 192.168.39.39, the console is available on http://cockroach.192.168.39.39.nip.io.
The username is keycloak, password is keycloak.
| minikube’s IP address changes every time the minikube instance is re-created. |
Connecting a local application to CockroachDB
CockroachDB’s database port is also available as a node port so that it can be accessed from a local application.
Assuming the command minikube ip returned 192.168.39.39, the JDBC URL is jdbc:postgres://192.168.39.39:30013/keycloak?sslmode=disable.
| minikube’s IP address changes every time the minikube instance is re-created. |
It can be used, for example, to connect the developer’s IDE to the database.
The connection details: Port is always 30013, username is root, database name is keycloak.
All ports are specified without the port offset.
If you are using the configuration option KC_PORT_OFFSET you need to add the offset to port numbers.
For the port offset XX (00-26) and the port number 300YY the resulting port will be 3XXYY.
|
Authentication is handled via TLS.
The necessary files are located in privision/minikube/crdb as ca.crt, tls.crt and tls.key.
To connect on the commandline to the CockroachDB, use the following command, which will connect as the root user and will authenticate using the certificate stores in minikube.
kubectl -n keycloak exec -it cockroach-client-secure-0 -- ./cockroach sql --certs-dir=/cockroach/cockroach-certs --host=cockroach-publicMetrics and Dashboards for CockroachDB
Metrics from the CockroachDB are available in Prometheus, and dashboards are available in Grafana. They are named “CDRB Console: SQL”, “CDRB Console: Runtime” and “CDRB Console: Storage”. The dashboards have been taken from the CockroachDB manual and adapted for the Job name.
To see all metrics, query Prometheus for {job="keycloak/cockroach-metrics"}.
Known issues for CockroachDB operator-based setup
Users of this setup should be aware of the following known issues.
Transactions are rolled back with the message RETRY_SERIALIZABLE
As the database will always run in isolation level serializable, there are situations where this can’t be avoided. On the other hand, the system could be optimized to retry those failures, or to use better query paths.
See the docs about this error message on how to collect more information on the Keycloak message RETRY_SERIALIZABLE.
Long shutdown times for CockroachDB nodes
When shutting down the nodes after deleting the CockroachDB custom resource with a command like this …
kubectl -n keycloak delete crdbclusters.crdb.cockroachlabs.com/cockroach
... then the shutdown of the last node might take up to five minutes.
Until that shutdown is complete, no new CockroachDB instance with the same name can be instantiated via the task command.
CockroachDB Operator will not clean persistent storage
When removing a CockroachDB cluster that has been created by the CockroachDB operator, the persistence volume claims (PVCs) and the persistent volumes (PVs) are retained, so that a recreated cluster will have its database restored.
To reset the database, there’s a script init-database.sh that is called when running task reset-keycloak.
To remove the PVCs and PVs and also the secrets connected to it, this needs to be done manually either via the command line of the minikube dashboard.
CockroachDB Operator needs to run to modify CockroachDB custom resource
When the CockroachDB operator is not running, no CockroachDB custom resource can be modified, as the callback that’s registered in Kubernetes is not available.
When this happens, have a look at the pod running in the namespace keycloak and ensure that it is running.
If it is in BackOff or Error state, consider killing it and waiting for it to restart.
Status in the CockroachDB custom resource is not updated
After the cluster has been initialized, the status in the resource is still clusterStatus: Failed, although the database is fully usable.
Due to that, the status of the resource can’t be used to monitor the cluster initialization.
See cockroach-operator/issues#937 for the status and updates.
Automated tests for CockroachDB Operator setup don’t run on GitHub actions
It seems that the memory, CPU or disk space constraints that exist on GitHub runners don’t allow the full stack of CockroachDB, Keycloak and monitoring tool to spin up for a test run.
Therefore, there is no provisioning test for this on GitHub actions. Any change to this setup needs to be tested manually.