« Back
in Java GCP read.

Debugging Java Apps in Production.

Debugging Applications in Production with Google Cloud Debugger

This example shows how a Java Spring Boot application can be debugged using Google's Cloud Debugger even if the executable is running on a different environment e.g. AWS. The appeal of this is the ability to debug in production and without sharing all the source code with another cloud provider.

Reasoning

This is my exploration into using Google Cloud Debugger as I just took over an existing monolith Java application. As I was receiving handover from the current Tech Lead, we ran into a production issue. The logging was not great and there were many downstream services being called. So the problem was not knowing the exact state the customer was in.

Now the reason why I say state is because the way the application has been coded in such a way that the REST models and Domain models were intertwined. Because the JSON serialization required empty constructors and all the setters and getters on properties, classes ended up being nearly identical to POJOs with all the business logic in some kind of service layer. This is not by any means the fault of the current team which are all pretty new to it themselves.

So, due to the logic being in some kind of middle layer, there are objects storing state and a lot of setting/getting happening. Objects had to reference other objects to determine their state. This can leave things in funny states and just gets more complicated over time.

Alas I just wanted to find another tool for the tool belt to help debugging if it really came down to this point. Ideally there should be no surprises in prod but ... shit happens. The debugger doesn't just have to be used for prod though. It can be any environment especially if you're unsure if the networking might not be done right etc.

High Level Steps

At a high level what we're going to do is package the Java application into a fat jar and run it on a different machine. That machine will also have the debugger agent on it which hooks into the JVM. This is how the agent works, it will take snapshots of what's running in order for us to get debugging output. Google says that at most it will have a 10ms response time impact on the environment. Also, we want to be able to do this by only sharing a portion of our code with Google not the whole thing. Although in the end it's just byte code so it can be reversed but let's not get into that.

Requirements

  • At least Java 7 for the agent. This example will run Java 8.
  • Linux - the agent only installs on Linux at the moment
  • Java application (GCP supports Python, Go and Java)
  • Maven
  • A Google Account registered with the free trial

Here's a link to the source code we'll be using:
https://github.com/serinth/stackdriver-debugger-example.git

1. Compile And Package This Application

git clone https://github.com/serinth/stackdriver-debugger-example.git  
cd stackdriver-debugger-example  
mvn clean install spring-boot:repackage  

This should produce target/stackdriverExample-1.0-SNAPSHOT.jar which will have all dependencies in it as a fat jar.

Make sure that it runs:

java -jar target/stackdriverExample-1.0-SNAPSHOT.jar  
curl localhost:8080/profile  

You should get a response.

Now copy this jar file over to a different machine that will execute it. e.g. An EC2 instance, a VM etc.

Don't run it quite yet. We'll do that once the agent is set up.

2. Create A Debugger Agent Service Account

After creating a new project on GCP, type in "service account" in the search bar and go to the IAM option for it.

Create a service account like so:
Cloud Debugger Service Account

Ensure that Furnish a new private key is ticked and make sure it's a JSON file. Save that file in a safe place. That's the authentication that's going to be used by the debugging agent.

3. Run Agent and App

  • Copy the Jar file over to the destination machine that will run it. Do not include source code
  • Copy the service account json file to the machine as well
  • Copy and extract the agent files into the same folder

The agent can be found here: https://storage.googleapis.com/cloud-debugger/compute-java/debian-wheezy/cdbgjavaagentserviceaccount.tar.gz.

That one in particular has service accounts enabled. Usually the debugger agent would just go fetch the meta-data from the instance running on GCP. But since we're not running on GCP we'll need to use the service account for auth.

On my fresh virtualbox machine, it looks like this:
directory tree of app for debugging

Now we run it with the following parameters and replace PATH with your path.

java -agentpath=PATH/cdbg_java_agent.so -Dcom.google.cdbg.auth.serviceaccount.enabled=true -Dcom.google.cdbg.auth.serviceaccount.jsonfile=PATH/service-account.json -jar stackdriverExample-1.0-SNAPSHOT.jar  

The arguments are pretty self explanatory.

A few things to note: I ran this on an Ubuntu Gnome machine so the logging for the agent was actually in /tmp in case you needed to debug authentication issues.

4. Get Snapshots from Cloud Debugger

Navigate back to GCP console and select source code from your dev machine. It can be one file or the whole directory. Doesn't matter. These are the possible source code locations:

GCP cloud debugger source file options

Then click on a couple of line numbers on the left side. You should get an indicator on the right panel to indicate that it's waiting to make a snapshot of what you want. This is the ToDo.java file:

GCP cloud debugger snapshot pending

So if you read the code, you can guess that the data coming in will have hours. It just gets multiplied by 2. We'll take a snapshot at line 19 to ensure that we're getting our original value. Then a snapshot at the end to see the transformed value. I put the second snapshot on line 22 but it should actually be 24. You'll see that in the next couple of screen caps.

GCP cloud debugger first snapshot state

So we see that the original value coming in was 2. The request was made using curl on the VM box with:

curl localhost:8080/todo/list/ -X POST -H "Content-Type: application/json" -d '{"itemList": [{"task":"task1", "hours": 2}]}'  

Now we click on the second break point and check out what state we're in:

GCP cloud debugger after modification state

The value of the hours has been doubled in the after state.

I think this was pretty neat but hopefully nobody has to use it too much.

comments powered by Disqus