The Problem

The Java application was behaving strangely in 4 out of 10 runs. It did not process all data available and assumed that the data input already ended. The application features several producer-consumer patterns, where one thread offers preprocessed data to the next one, passing it into a buffer where the next thread reads it from.

The consumer or producer fall into a wait state in case no data is available or the buffer is full. In case of a state change, the active threads notifies all waiting threads about the new data or the fact that all data is consumed.

On 2-core and 8-core machines, the application was running fine but when we moved it to 24-cores, it suddenly started to act in an unpredictable manner.

The Cause

After a lot of debugging I found out that threads wake up without having been notified by their partner thread. In this case the consumer was woken up despite the fact that data was unavailable aka the producer has not delivered and therefore not notified anyone. But the consumer was awake…

The debugging nightmare finally revealed a rare behavior of any POSIX based application. This is the quote from the JDK6 doc:

A thread can also wake up without being notified, interrupted, or timing out, a so-called spurious wakeup. While this will rarely occur in practice, applications must guard against it by testing for the condition that should have caused the thread to be awakened, and continuing to wait if the condition is not satisfied. In other words, waits should always occur in loops.

The Verdict

So never trust your notification chains. Threads might wake up even though nobody directly notified it. Additionally you should never exclude the possibility that the move from a small box to a big box does not influence the application behavior. More cores mean more trouble.

• Parallel and concurrent garbage collectors
• How does garbage collection work?

Enjoy reading.

Most modern programming languages are virtual machine based and these machines have knobs you can turn to adjust their behavior according to your requirements. XLT runs on Java and so all the things you might have already learnt from tuning your Java-based servers apply to XLT as well. If you do not have experience in tuning your Java-based servers, you will learn a lot that can be applied to your servers and help you to increase performance.

So, let’s take a look at the upcoming XLT feature that provides you with more details about garbage collection (GC) without the need to use any external tool to monitor or analyze logs or JVMs runtime behavior.

Perfect GC behavior

This is a chart displays nice memory statistics that indicate absolutely no problems with the garbage collection. As you can see the memory usage (first chart in the image) is alternating but never touching the upper limit. The second chart gives you the times when full garbage collection occurred, basically these are the bad events. Events that block the virtual machine from continuing to run. We should avoid these at any cost… nearly, because a full GC is costly. There was only one full GC right at the beginning at the test. This is a normal JVM behavior. So we seem to have done a pretty good job. No major collection of garbage. Sweat!

There are still the minor collections. Minor collections occur frequently and they reclaim small portions of memory with short living objects. The third charts shows us, that minor collection took about 10 to 30 ms, some spikes up to 70 ms. This is acceptable.

GC with Issues

Ok, let’s have a bad example in comparison. It runs with almost default VM settings. This means just min and max heap were set (Xms256MB/Xmx512MB). You can see clearly that a lot of major collections were running for up to 800 ms each. Also the minor collections were longer.

Clearly, the new charts help you see and fix misbehavior quickly. “What about the settings?” you might ask now. Yes, this example is not very helpful without revealing the difference between both test runs. So, we will give you the settings of the good one. The bad run had only Xms and Xmx set, as mentioned before.

## Set minimum memory to use
-Xms512m

## Set maximum permitted memory
-Xmx512m

## Enable concurrent old GC to avoid sudden long pauses
-XX:+UseConcMarkSweepGC

## When to start GC for the tenured/old area of the memory.
## This has to be low enough to avoid that threads need
## memory and can not get any before the GC has finished.
## This will lower the wait time.
-XX:CMSInitiatingOccupancyFraction=70

That’s really all. The use of the Concurrent-Mark-Sweep-Collector (CMS) is the most important setting. The CMS does not block the main VM worker threads, but collects the garbage concurrently. Because there might be situations were the CMS collector is not fast enough in cleaning up, this means the old space runs out of free memory, we give the CMS a hint with the last line. This line explicitly tells the CMS to start early with the clean up. In this case at a 70% fill level. The default is 96% and way too high.

One last thing. Setting Xms and Xmx to the same number helps the VM to relax because it will not try to get back to the minimal memory limit (Xms) and so it cleans less aggressively.

If you want to lean more about Java Garbage Collection, take a look at this Oracle documents: GC Tuning.

So, the G1 is not yet ready for production, of course nobody stated that it is ready for production. If I read the release notes of JDK6u21 correctly, it delivers plenty of G1 changes, so I might try that soon.

Das Ganze ist ein bekanntes Problem seit Ubuntu 9.10 und sollte mit Eclipse 3.5.1 weg sein. Wenn das aber keine Lösung ist, dann muss man seine Umgebung mit diesem Parameter anpassen:

GDK_NATIVE_WINDOWS=true

Danach funktioniert es wieder. Die Lösung habe ich hier gefunden: Widdix – Eclipse unter Ubuntu 9.10 und hier gibt es mehr dazu in Englisch.

-verbose:gc
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
-XX:+PrintGCApplicationStoppedTime
-XX:+PrintReferenceGC

Details zu den Optionen und zur Auswertung kann man unter GC Tuning oder in der Liste der JDK-Optionen finden.

-Dsun.java2d.noddraw=true

Mehr dazu findet sich bei Sun unter Graphics Performance Improvements.

In software engineering, the Initialization on Demand Holder idiom (design pattern) is a lazy-loaded singleton. The idiom can be implemented in both single-threaded/serial and concurrent environments, but care must be taken to correctly implement the idiom under concurrent conditions.

Ganz besondern wichtig ist die Erklärung, warum Lazy in diesem Fall so und nicht anders funktioniert:

The implementation relies on the well-specified initialization phase of execution within the Java Virtual Machine (JVM); see section 12.4 of Java Language Specification (JLS) for details.

When the class Something is loaded by the JVM, the class goes through initialization. Since the class does not have any static variables to initialize, the initialization completes trivially. The static class definition LazyHolder within it is not initialized until the JVM determines that LazyHolder must be executed. The static class LazyHolder is only executed when the static method getInstance is invoked on the class Something, and the first time this happens the JVM will load and initialize the LazyHolder class.

Im JDK 6v14 könnten wir ihn vielleicht sehen. Ich bin schon ganz aufgeregt, da ich mittlerweile oft und viel mit GC-Internas hantiere… keine Wunder bei Lasttests gegen 120 CPUs mit Java drauf.

Java Virtual Machines and Linux virtual Servers do not play well with each other all the time. Some tweaking and configuration will be necessary to get it working and optimize the use of the (memory) resources.

Die VPS / Virtuellen Server haben leider die Angewohnheit, den Speicher als zu gross zu melden. Sie melden den Gesamtspeicher des Hosts und nicht den Speicher des Virtuellen Servers. Deswegen schlägt Java gnadenlos zu, und versucht sich reinzuoptimieren. Die einzige Lösung ist die Limitierung des Speichers schon beim Start von Java mit -Xms und -Xmx, um die Selbstoptimierung von Java zu umgehen.

Soft references are kept alive longer in the server virtual machine than in the client. The rate of clearing can be controlled with the command line option -XX:SoftRefLRUPolicyMSPerMB=<N>, which specifies the number of milliseconds a soft reference will be kept alive (once it is no longer strongly reachable) for each megabyte of free space in the heap.

The default value is 1000 ms per megabyte, which means that a soft reference will survive (after the last strong reference to the object has been collected) for 1 second for each megabyte of free space in the heap. Note that this is an approximate figure since soft references are cleared only during garbage collection, which may occur sporadically.

Quelle: JDK 6 Garbage Collection Tuning Guide

Nachtrag: Erwähnenswert ist auch dieser Blogeintrag von Jeremy Manson.

Einen kleinen Trick für unnötige Objekt-Erzeugung und damit für weniger Müll im System, gerade wenn immer wieder Strings gebaut werden müssen, ist die korrekte initiale Größe des StringBuilders. Ein new StringBuilder() reserviert zunächst nur ein Array mit 16 Characters. Wenn wir also recht viel zusammenbauen, muss mehrmals eine neues grösseres Array (2 * ggw. Grösse + 1) erzeugt und der Inhalt umkopiert werden.

Wer also weiss, dass er ca. 100 Zeichen aneinanderhängt, der lässt sich gleich einen StringBuilder mit Grösse 120 geben – new StringBuilder(120) – und spart damit das Anlegen von drei Arrays und drei Copy-Operation ein. Gerade bei intensiven Operationen mit Strings macht das eine Menge aus.

Wer würde schon bei einem Umzug zuerst mit dem kleinsten Karton anfangen und dann wegwerfen, weil nicht alles reinpasst? Man schätzt doch am Anfang schon vernünftig ab, wie groß der Karton sein sollte.

Java Concurrency – A Tutorial

Java was one of the first languages to make multithreading easily available to developers. Java had multithreading capabilities from the very beginning. Therefore, Java developers often face the problems described above. That is the reason I am writing this trail on Java concurrency. As notes to myself, and any fellow Java developer whom may benefit from it.

Einzig die Behauptung

Garbage Collection pauses were killing them. [LinkedIn said they were using advanced GC's, but GC's have improved since 2003; is this still a problem today?]

kann ich nicht glauben. Deswegen wurde die Graph-Engine in C geschrieben. Ich denke mal, da war komplett etwas falsch, wenn der GC austickt.