John's Technical Blog

When setting up Readiness and Liveness checks in Kubernetes for Kafka connectors, the use of the httpGet described in my previous blog post (Kubernetes Readiness and Liveness with Spring Boot Actuator) is not an option because there is no endpoint to reference. These can be deployed with the Apache Kafka REST Proxy, which gets us on the right path, but doesn't quite work how we want in this respect.

The Kafka REST Proxy provides endpoints that allow one to get some basic status info about connectors. However, the standard Kubernetes httpGet calls use status code >= 200 and < 400 to determine the status, and since the Kafka REST status endpoint always provides a 200 status code, it is not possible to use this methodology to determine if a connector is down.

What we would like to do is check the content of the status call, and do a string comparison. For example, when the service is up, the status endpoint indicates that the state is "RUNNING":

# curl http://10.30.128.1:8083/connectors/mysql-kafka-connector/status

{"name":"mysql-kafka-connector","connector":{"state":"RUNNING","worker_id":"10.30.128.1:8083"},"tasks":[{"state":"RUNNING","id":0,"worker_id":"10.30.128.1:8083"}]}

We can pause the connector using this endpoint:

# curl -i -X PUT http://10.30.128.1:8083/connectors/mysql-kafka-connector/pause

HTTP/1.1 202 Accepted

And then the state is changed to PAUSED:

# curl http://10.30.128.1:8083/connectors/mysql-kafka-connector/status

{"name":"mysql-kafka-connector","connector":{"state":"PAUSED","worker_id":"10.30.128.1:8083"},"tasks":[{"state":"PAUSED","id":0,"worker_id":"10.30.128.1:8083"}]}


To accomplish this check, we can leverage the exec command probe:

readinessProbe: 

  exec: 

    command:

      - /bin/sh 

      - -c 

      - curl -s http://127.0.0.1:8083/connectors/mysql-kafka-connector/status | grep "RUNNING"

  initialDelaySeconds: 240

  periodSeconds: 5

  timeoutSeconds: 5 

  successThreshold: 1

  failureThreshold: 10 

livenessProbe:

  exec: 

    command: 

      - /bin/sh

      - -c 

      - curl -s http://127.0.0.1:8083/connectors/mysql-kafka-connector/status | grep "RUNNING"

  initialDelaySeconds: 300 

  periodSeconds: 60

  timeoutSeconds: 10 

  successThreshold: 1

  failureThreshold: 3

The the exec command allows us to execute a shell command. In this case:

• running the shell (/bin/sh)
• telling it to run a single command (-c)
• with the command being a cURL call to the specific connector status endpoint, and grepping for the string "RUNNING"

When the grep is successful, Kubernetes interprets this as a success. If the grep comes back empty (i.e., "RUNNING" is not found), then it gets viewed as a failure. You can then test this on the server by pausing the service in question as described above.

To get the service running again and start passing readiness and liveness again, then you will want to use the RESUME endpoint.

# curl -i -X PUT http://10.30.128.1:8083/connectors/mysql-kafka-connector/pause

HTTP/1.1 202 Accepted

In Kubernetes, "readiness" is the indicator that the service is ready to accept traffic, and is only performed at the beginning of a pod's life cycle. "Liveness" is a periodic health check that should indicate that the service is still functional within the pod.
More details can be found in the documentation at
https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/

Spring Boot is a stand-alone Spring environment well suited for micro services.
https://projects.spring.io/spring-boot/

Spring Boot has a bundle available called "Actuator" that exposes several helpful endpoints, one of which is a "health" endpoint. The simple version of this endpoint returns a simple JSON with a status UP or DOWN.
https://github.com/spring-projects/spring-boot/tree/master/spring-boot-actuator

One particular feature of the health endpoint that is useful is that besides the text indicator in the JSON response, it also signals up and down through the status code. UP = 200, and DOWN = 503 (Service Unavailable).

Putting this all together, the readiness and liveness configuration in the Kubernetes deployment YAML can look something like this:

readinessProbe:
httpGet:
  scheme: HTTP
  path: /health
  port: 8080
initialDelaySeconds: 240
periodSeconds: 5
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 10
livenessProbe:
httpGet:
  scheme: HTTP
  path: /health
  port: 8080
initialDelaySeconds: 300
periodSeconds: 60
timeoutSeconds: 10
successThreshold: 1
failureThreshold: 3

In this example, we instruct Kubernetes to wait 240 seconds to allow the application to start before performing the check. It will retry up to 10 times with a five second pause between each try. It will wait for a maximum of 5 seconds for a result to be returned. After 10 failures, the pod will be restarted.

For the liveness check, we instruct Kubernetes to wait 300 seconds, and to check every 60 seconds. If three failures occur in a row, the pod will be restarted.

I recently ran into a problem where I wanted to add objects attained by reflection to a TreeSet. In this specific case, I had a problem when a Locale object was retrieved. It could not be added to the TreeSet because it is not "Comparable." It did, however, have toString values that could be sorted intelligibly.

I decided to create a Comparator that would handle arbitrary objects, and allow them to be added to a TreeSet of type Object:

The caveat is that this works best when the object in question overrides the default toString method in a meaningful way. If the base Object.toString() method is inherited by the object, then the sorting will most likely appear random, since the method looks like this:

    public String toString() {
        return getClass().getName() + "@" + Integer.toHexString(hashCode());
    }

After updating an application to Hibernate 5.1.1 from 5.1.0, I started seeing the following warning:

HibernateJpaDialect - JDBC Connection to reset not identical to originally prepared Connection

I did not find an "explicit" solution on the Internet, but looking through some of the code fixes in the ngrinder project, I figured out that I could fix my problem the same way by adding <prop key="hibernate.connection.release_mode">on_close</prop> to applicationContext.xml:

Thanks for this solution, JunHo!

Quick Answer

Manually add the JMeter path at

Preferences » Other Settings » JMeter » JMeter home directory

Long Answer

I wanted to be able to launch JMeter scripts from within IntelliJ IDEA. This is the process I went through to make this happen:

Install JMeter

Having already installed Homebrew, JMeter can be easily installed with the command

brew install jmeter

Install JMeter Plugin in IntelliJ IDEA

Preferences » PlugIns » Search for JMeter » Select JMeter plugin » install » restart IntelliJ

Configure JMETER_HOME

This is where the troubleshooting needed to start. For the quick solution, see the first section. Otherwise, this is the process I went through:

Ideally, one would want things to "just work" after installing JMeter and the plugin. Unfortunately, when I created my first JMeter Run Configuration in IntelliJ, I got the message in the dialog box:

Run Configuration Error: JMeter not found

I eventually found a dialogue box under

Preferences » Other Settings » JMeter

with a helpful indicator that

JMETER_HOME is used by default

Sure enough, when I ran the env command in a Terminal, JMETER_HOME was not there.

To find out what my JMeter home directory was, I did the following in the Terminal

$ which jmeter
/usr/local/bin/jmeter
$ cat /usr/local/bin/jmeter
#!/bin/bash
exec "/usr/local/Cellar/jmeter/3.0/libexec/bin/jmeter" "$@"

So, based on some other information I gathered, the home directory was just above the "bin" directory.

My first attempt was to add the following line to my .bash_profile:

export JMETER_HOME=/usr/local/Cellar/jmeter/3.0/libexec

Sadly, I found this didn't work after rebooting and confirming that it showed up when the env command was executed in the Terminal.

I eventually ran across Setting global environment variables in IntelliJ IDEA and other test config goodies, from which I learned that environment variables are not automatically passed to GUI applications in Mac OS X. Nonetheless, Update 2 in the article looked promising.

I then added this line to .bash_profile:

launchctl setenv JMETER_HOME /usr/local/Cellar/jmeter/3.0/libexec

Again, to my frustration, I found that after rebooting and confirming that launchctl getenv JMETER_HOME displayed the correct value in the terminal, that this only worked in IntelliJ when I started IntelliJ, quit completely, and then started it again. I have no idea why the environment variable only seems to get read on the second startup.

So, for my final answer, I just had to go with the manual solution of adding the JMeter home path as an Override in the dialogue box found at:

Preferences » Other Settings » JMeter » JMeter home directory