Debugging Multi-Node Cassandra Cluster on Windows

First : Run a single node cluster, and set breakpoints.

  1. Download latest version of cassandra from https://github.com/apache/cassandra. Click “Clone in Desktop” button on the right side of the page for quickest download experience.
    1. You can also setup “Egit” so that you can get source control in Eclipse (Just like Team window in VS). Follow: Eclipse GIT Integration
    2. If you want to edit sources, it’s better to clone the repo (From command prompt, type “git clone -b <branch name> http://git-wip-us.apache.org/repos/asf/cassandra.git <branch name>”. This will create a <branch name> folder in current folder, and download sources in there.)
  2. Download latest version of Eclipse – https://eclipse.org/downloads/download.php?file=/technology/epp/downloads/release/mars/R/eclipse-java-mars-R-win32-x86_64.zip
  3. Download Java 1.8 – http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
  4. Download latest version of Ant (1.9.6)- https://ant.apache.org/bindownload.cgi
  5. Download latest version of Maven (3.3.3) – https://maven.apache.org/
  6. Download Python version 2.7 (this is important: Don’t download the latest) – https://www.python.org/download/releases/2.7/
  7. For items downloaded above from 2 through 6, be sure to set their executables in your PATH environment variable. It’s also a good idea to create JAVA_HOME environment variable and point it to your JDK installation.
  8. Follow instructions from here – https://wiki.apache.org/cassandra/RunningCassandraInEclipse to build Cassandra. If you run into issues, make sure your JDK installation is good (such as “javac -version” should show you 1.8)
  9. Once Eclipse is fired up, you may see errors in building the stress tool. Remove that from project build settings.

To do so, right click Project -> Select Properties, java Build Path. Ignore what the link in (8) says for Main class, instead choose “org.apache.cassandra.service.CassandraDaemon” as the Main class.

    1. In Debug -> Debug Configurations menu, you need to double click on “Java Application” to select a new configuration.
    2. Provide following VM arguments

For Cassandra 2.1.9 (current version in DSE 4.7.3 cluster)

-Dcassandra.config=file:E:\DI\cassandra\conf\cassandra.yaml -Dcassandra-foreground -ea -Xmx1G -Dlog4j.configuration=file:E:\DI\cassandra\conf\log4j-server.properties -Djava.rmi.server.hostname=127.0.0.1 -Dcom.sun.management.jmxremote.port=10020 -com.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcassandra.storagedir=F:\data

For Cassandra 3.x

-Dcassandra.config=file:E:\DI\cassandra\conf\cassandra.yaml -Dcassandra-foreground -ea -Xmx1G -Dlog4j.configuration=file:E:\DI\cassandra\conf\log4j-server.properties -Djava.rmi.server.hostname=127.0.0.1 -Dcom.sun.management.jmxremote.ssl=false -Dcassandra.jmx.local.port=7199 -Dcassandra.storagedir=F:\data

Replace E:\DI\Cassandra with your location. F:\data can be any directory. For JMX pick a port that’s not busy

[You can use wintellect Process Explorer for this]

  1. If you have Cassandra running as Datastax services, stop them from services.msc
  2. Click “Debug”. At this point, the view will change and you should see bunch of output in Console window at the bottom.
  3. Note the logs are written to build\test\logs directory by default.

Run cqlsh.bat from E:\DI\Cassandra\bin. It should successfully connect and you should be able to run commands. You can also connect from Dev Center if you like.

Running a command like below

cqlsh> CREATE KEYSPACE Test1 WITH REPLICATION = { ‘class’ : ‘SimpleStrategy’, ‘replication_factor’ :  3 };

Should print below in eclipse output.

INFO  20:05:45 Create new Keyspace: KeyspaceMetadata{name=test1, params=KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=3}}, tables=[], functions=[], types=[]}

INFO  20:05:45 Enqueuing flush of keyspaces: 1492 (0%) on-heap, 0 (0%) off-heap

 

To stop cassandra, click on “Stop sign” icon on Console window on the right.

 

You’re all set to put breakpoints and be a C* wizard now.

Now you are ready to debug a multi-node cluster

1. Create two copies of Cassandra (say E:\cassandra-bkup and E:\cassandra). These will be our two nodes in a cluster.

2. We will use the fact that 127.0.0.1 through 127.0.0.8 map to localhost. The first node will run as 127.0.0.1 and second as 127.0.0.2

In E:\cassandra-bkup

Edit cassandra.yaml from E:\cassandra-bkup\conf

Replace “localhost” with “127.0.0.1”

Replace seeds: “127.0.0.1” with  seeds: “127.0.0.2”

Use the following VM args to Eclipse.

-Dcassandra.config=file:E:\DI\cassandra-bkup\conf\cassandra.yaml -Dcassandra-foreground -ea -Xmx1G -Dlog4j.configuration=file:E:\DI\cassandra-bkup\conf\log4j-server.properties -Djava.rmi.server.hostname=127.0.0.1 -Dcassandra.jmx.local.port=7199 -Dcassandra.storagedir=E:\DI\cassandra-bkup\data

Start Cassandra in Eclipse and make sure its running fine.

In E:\cassandra

Edit cassandra.yaml from E:\cassandra\conf

Replace “localhost” with “127.0.0.2”

Use the following VM args to Eclipse.

-Dcassandra.config=file:E:\DI\cassandra\conf\cassandra.yaml -Dcassandra-foreground -ea -Xmx1G -Dlog4j.configuration=file:E:\DI\cassandra\conf\log4j-server.properties -Djava.rmi.server.hostname=127.0.0.2 -Dcassandra.jmx.local.port=7200 -Dcassandra.storagedir=E:\DI\cassandra\data

Run “nodetool status” from both E:\cassandra-bkup\bin and E:\cassandra\bin and you will see this =>

Datacenter: datacenter1

========================

Status=Up/Down

|/ State=Normal/Leaving/Joining/Moving

—  Address    Load       Tokens  Owns    Host ID                               Rack

UN  127.0.0.1  139.12 KB  256     ?       d24dfee8-05b8-4ba0-b200-2e432d648b6d  rack1

UN  127.0.0.2  123.92 KB  256     ?       9e7fe5dc-a3e8-4695-88e4-b2edbac4a113  rack1

Now you can put breakpoints and bring nodes up / down and see how it all works.

FAQ:

What if I get an error that another process is listening on port 10020 (or some other port that Cassandra complains about) ?

Follow this and delete that process: http://stackoverflow.com/questions/48198/how-can-you-find-out-which-process-is-listening-on-a-port-on-windows