brunoborges 1 day ago

Lots of details that teams miss when tuning the JVM on Kubernetes:

1. Default heap size varies: it can be 50% of the available memory for any amount of available memory up to 256mb; ~128mb for anywhere between 256 and 512; and then 25% for anything above 512mb;

2. The JVM will still do parallel threads even when CPU limit is 1,000 or below , which triggers CPU throttling much faster. Threads such as GC (even when it is SerialGC), ForkJoin pools, Database connection pools, HTTP server, and so on.

3. The JVM behaves MUCH better with 2,000 or more milicores, and a well sized heap. Having very small containers and dozens of replicas is worse than having slightly bigger containers (cpu/memory limits) and fewer replicas, even with a different ratio (e.g. 6 replicas will perform worse than 3 replicas, even when aggregated CPU and memory limits are the same).

I've been advocating on this for years [1].

[1] https://www.youtube.com/watch?v=wApqCjHWF8Q

  • gavinray 1 day ago

    Modern JVM's (JDK +10) have runtime detection for whether they're operating in a container environment, and apply sensible defaults based dynamically on resources if so.

    Google term: "-XX:+UseContainerSupport"

    • gavinray 1 day ago

      I just realized the username of the comment I replied to.

      The OP undoubtedly knows this, whoops.

      • froh 1 day ago

        I (and presumably some others) didn't. TIL and thx for sharing.

        and I also read it not as contradiction but simply as useful additional information.

    • hibikir 1 day ago

      While you are replying to someone that definitely knows this, you'd be surprised by the number of production containers out there, in big companies, that haven't turned on container support, or hand tuned their memory well, and thus end up throwing away large parts of the memory they are allocated.

      Having used Java since the days where you needed to do very heavy tuning to get it to run properly, it all seems alien, as understanding the old garbage collectors was basically mandatory 20 year ago. But even if you reply to someone that does understand the new features, you might be helping a bunch of people that don't. In my unit of about 100 programmers I am the only one that even bothered looking, given how every single pod we deployed was either completely unconfigured, or so badly misconfigured, using settings that haven't been ideal since Java 6.

      In one of the pods, running a batch job that was using way too much memory, about a third of the memory in the old generation were millions of copies of the very same string. Interning of strings loaded like that has been a JVM option since 2014.

      • gavinray 23 hours ago

        On that note, here's my favorite little-known tip for running Docker/Container JVM apps:

        The Redhat image distros, like UBI and the JDK ones, have a really handy bash script, "run-java.sh". It's a wrapper script for best-practice and intelligent JVM runtime args inside of container envs:

          /opt/jboss/container/java/run/run-java.sh
        

        It started life as fabric8's run-java-sh, and it's what the Dockerfile.jvm generated by Quarkus apps uses as its entrypoint.

        The script handles, among other things:

        - Heap sized as a percentage of the container's memory limit via -XX:MaxRAMPercentage (default 80%, compared with JVM's default of 25%). Tunalbe with JAVA_MAX_MEM_RATIO env.

        - -XX:+ExitOnOutOfMemoryError, so an OOM kills the container and your orchestrator restarts it

        - Defaults for GC and metaspace

        - JAVA_DEBUG=true / JAVA_DEBUG_PORT=*:5005 turns on JDWP without you having to edit the Dockerfile

        - http_proxy/https_proxy/no_proxy env vars get translated into -Dhttp.proxyHost etc

    • vips7L 18 hours ago

      Isn’t that on by default?

      • brunoborges 14 hours ago

        Pretty much, yes.

        • vips7L 11 hours ago

          So is there nuance in pretty much? My understanding for a while is that hotspot automatically detected containers.

    • brunoborges 17 hours ago

      When cgroups / container awareness was added to OpenJDK, it was not on by default, and indeed there were versions that needed this flag. However, all modern JDKs 11 and later, have container awareness support on by default.

      That said, this only makes the HotSpot JVM use cgroups data to interpret CPU and Memory limits. It does not make any special consideration of default ergonomics (thread pools, GC selection, heap sizing, and so on). Before the container awareness support, the JDK would read the VM/bare metal data; in other words, pre-container awareness support would result in this: if the VM has 32GB of RAM and a container is set for 4GB of memory limit, the JVM reads the 32GB as available memory.

      All the defaults I mentioned still apply even inside a container.

      Edit: no worries! thanks for sharing!

  • joosd 23 hours ago

    Hey man just wanted to let you know years ago our team used your Devoxx talk as a starting point for tuning our Java apps that are running on K8s and it has been of tremendous help. I still refer back to it every now and then!

    • brunoborges 17 hours ago

      Thanks a lot mate!

      It's been an amazing journey exploring JVM on Kubernetes: all that research is being turned into tools and products, if you are interested! One is https://aka.ms/jaz (not open source at the moment, but it is free), and https://brewlet.sh (MIT).

  • madduci 14 hours ago

    Absolutely! Passing fine-tuned parameters to JVM makes a big difference and everyone should profile their applications and find the correct balance

Fizzadar 1 day ago

I remember switching on G1GC for Elasticsearch back in 2014 and it was a total game changer for cluster stability (ES had plenty of other warts back then to counter the improvement). Cool to see it land as the default everywhere over a decade later.

  • cogman10 1 day ago

    It's definitely gotten a LOT better since it was introduced. Java 17, in particular, really made G1 great.

    But if you've not already looked at it for your usages, I suggest people check out both the parallel collector and ZGC. ZGC in particular has been getting a lot of very nice features. Semi recently getting fast heap freeing (to the OS). The parallel collector for small heaps can generally beat G1 pause times.

    Where G1 really shines over the parallel collector is the fact that it regularly collects old gen. If your app has a lot of caches, then G1 is very good at keeping the VM from ending up with very long pauses.

  • pjmlp 1 day ago

    It is not everywhere, because there are other JVM implementations, with their own GC algorithms.

ahmedfromtunis 1 day ago

I love that they also have learn.java and inside.java.

They should make get.java point to the download page. Oh and docs.java would be very cool!

  • 0x54MUR41 14 hours ago

    They also have ops.java, https://ops.java, which I just knew a few weeks ago. This web page is pretty useful for Java administrators. An administrator usually has activities, like handling configurations, troubleshooting, and keeping things up to date.

    The website seems to address how things are moving fast now in the age of AI. I mean using AI to find security vulnerabilities.

Sid_Lmao 4 days ago

Easy-to-miss detail: before JDK 27, the JVM only picked G1 with 2+ CPUs and ~1.8 GB of memory; otherwise, it silently used SerialGC. So a lot of 1- CPU Kubernetes pods have been on serial all along. Worth checking with -XX:+PrintCommandLineFlags before upgrading, since GC behaviour will change.

  • citrin_ru 1 day ago

    Isn't the reason for this that on a single core SerialGC is better (lower CPU overhead at least)?

    • ch4s3 1 day ago

      I'd imagine so. The kind of bookkeeping they're talking about here would negatively effect performance on a single core.

    • antonhag 1 day ago

      Lower overhead yes - but G1 should have shorter pauses. And shorter pauses is probably worth it unless you run only background jobs in your jvms.

      • cogman10 1 day ago

        Lower CPU and Memory overhead.

        Serial will be your best choice if your app is constantly using 100% of that core or if you need to minimize the amount of memory overhead.

        G1 is a better choice if you can be flexible with your memory and if you have a lot of periods in your app where it's below 100% CPU usage (maybe a lot of IO).

    • brunoborges 1 day ago

      A 1,000 millicore CPU limit on Kubernetes does not mean the JVM can only use a single core. But that is indeed how the JVM interprets the limit: as a single core being available.

  • seabrookmx 1 day ago

    > So a lot of 1- CPU Kubernetes pods have been on serial all along

    I don't think this is true.

    Setting your CPU limit to 1 in Kubernetes does not mean the container sees a single logical processor. It only means the kernel allows it to use 1 core worth of CPU _time_.

    If your k8s node is an 8 core node, Java will still see 8 logical processors. It just means you can only use 1/8th of each, or all of one, or some combination in-between.

    I've never seen a k8s node running on a single core VM, as most of the cloud providers only use these for their free tier or shared core VM's for the lightest of light workloads.

    • rileymichael 1 day ago

      > Setting your CPU limit to 1 in Kubernetes does not mean the container sees a single logical processor

      that's incorrect, the jvm has been container aware for nearly a decade (https://bugs.openjdk.org/browse/JDK-8146115)

      you are correct about the underlying scheduling. limits don't pin you to a core, you get quotas across all available

    • jeffbee 21 hours ago

      Actually, setting your limit and request to exactly 1 does mean that, in some common kubernetes configs. There will be exactly 1 visible CPU.

  • brunoborges 1 day ago

    This is correct. I have been advocating about this for years now [1].

    I have also seen customers wasting resources across Kubernetes because their developers would eventually tune JVMs with lowercase `-xmx` flag, which the JVM ignores, and thus by default, the JVM only sets heap size to 25% by default (for => 512mb of available memory).

    [1] https://www.youtube.com/watch?v=wApqCjHWF8Q

  • pjmlp 1 day ago

    Open JDK, not The JVM.

    There are other implementations.

    • hashmash 1 day ago

      Not really, other than OpenJ9. In my experience, it's way behind OpenJDK in terms of performance and stability.

      • pjmlp 1 day ago

        Then you never embedded development with real time JVMs, for example.

        • hashmash 1 day ago

          Nope. What kind of collectors do they use to be real time? I've never understood how this is possible.

          • pron 22 hours ago

            Even ZGC now offers better predictablity than malloc/free allocators (which are a long way from predictable as far latency goes). But some hard-realtime software uses (or at least used to) RTSJ, a hard-realtime version of Java, that separates the heap to a part that uses a GC and a part that uses nested arenas. It was a big deal twenty years ago, when I worked on such software. I don't know how much it's still used. Probably not much. C is still king in that world, and Ada too, to some extent.

            It's important to know that for hard realtime, even C alone isn't enough, though, and you need special kernels. Ordinary OS schedulers aren't predictable enough for hard realtime.

cyberax 1 day ago

And still no automatic heap sizing... Maybe next year?

  • karussell 1 day ago

    What do you mean? G1 definitely has automatic heap sizing, i.e. Xmx just defines the maximum it is allowed to use.

    edit: looks like you probably mean sth like this: https://openjdk.org/jeps/8359211

    • cyberax 1 day ago

      Indeed.

      One great feature in Go (or Python, or Node JS) is that you don't have to care about setting the -Xmx. The runtime will just use more RAM as needed, and you can use normal mechanisms like cgroups to limit the maximum size.

      • nlitened 13 hours ago

        Just set it to a very large number if you want that behavior. If I remember correctly, JVM may use some sweet optimizations like use 32-bit pointers if you limit your max memory below 32 GB. It wouldn’t be able to do that if it does not know what the theoretical maximum for your app will be.

_verandaguy 1 day ago

Tangential, but I really dislike that the `.java` gTLD was allowed to come about given it's also a reasonably popular file extension.

The `.zip` gTLD set a really bad precedent.

  • lmz 1 day ago

    .zip? Not .com?

    • dbt00 1 day ago

      The potential for confusion in 1985 wasn't really on anyone's radar.

pancakeguy 1 day ago

garbage collection will almost certainly be a relic of the past surprisingly soon. Fundamentally it exchanges efficiency for readability & developer experience, to things that do not matter when using AI to write code.

  • ogogmad 1 day ago

    If true, it's sad. So much work and so much accumulated wisdom. It's like seeing a cathedral built by thousands of hands over dozens of years collapse in an earthquake.

    • jitix 1 day ago

      Disagree, we need to move ahead with progress.

      Hobby and domain specific languages may still use GC but getting rid of the concept of GC is the right direction, along with removing different ways of doing things in a programming language. Go was right when it forced developers to use only pattern for loops, one pattern for error handling, etc.

      • titzer 1 day ago

        Memory management is a resource issue that should be orthogonal to the logic of code. Moving away from GC is not the right direction for writing smaller, more concise, and more correct programs.

        I don't think that making programs a lot more complicated because they have to prove a very specific memory management discipline is the right way to go.

        • jitix 1 day ago

          Programming languages are just an intermediate representation of the solution, just like bytecode or assembly.

          With AI we don't need expressive programming languages for humans to read and understand at line-level details. Instead, we need stronger guardrails and fewer building blocks so that AI generates consistent solutions.

          GC became poplar only because it was hard for programmers to keep track of memory management in languages like C and C++. And back then I was a huge proponent of GC based languages.

          This is just a step along the same path that Go introduced. Many programmers switching from other languages complain about Go's restrictive and seemingly verbose syntax but once the benefits are realized the tradeoffs become worth it.

          • titzer 1 day ago

            > GC became poplar only because it was hard for programmers to keep track of memory management in languages like C and C++. And back then I was a huge proponent of GC based languages.

            As I said elsewhere in the thread, GC is not inherently less efficient than manual memory management, so I don't really agree with this. It's better for programmers and makes for more ergonomic and evolvable APIs.

            GC is a pervasive, system-wide service that makes ownership and management orthogonal to program logic. It does this by being a runtime service that comes with some runtime cost. Even manual memory management comes with runtime cost--different ones.

            > With AI we don't need expressive programming languages for humans

            Actually I think it's completely the opposite. PL is for humans to understand algorithms. We need AIs to reduce their thinking to algorithms and then present their solutions as algorithms to us. When we want to communicate precisely, reason and understand precisely, we don't write prompts, we write and understand programs.

            That's not really a popular opinion right now because so many people are excited to be freed from thinking and staring at shitty code, but that isn't how it's going to go down. Programs are proofs that something has been reduced to a deterministic algorithm instead of a sloppy, ill-defined prompt or skill.

            And to that, I for one do not want to have the communication medium be littered with memory management nonsense.

  • IncreasePosts 1 day ago

    Doesn't this assume infinite abundance of compute? For most problems, using a GC is probably easier for AI than to track everything manually(just like with humans). Similarly, we could probably ask AI to output everything in optimized assembler code, but it will almost certainly burn more tokens getting to a solution than if we allow it to write in python or some other HLL

  • xxpor 1 day ago

    Disagree. Maybe the naive business app use case will, but there are plenty of places where you want to implement effective GC regardless. Anywhere you batch frees to avoid calling it in a hot path is a GC. So it's not _always_ an efficiency sacrifice. Or, a different perspective is you always have to ask "efficient with what resource?" CPU time efficiency vs memory efficiency for example (an interesting case where the _money efficiency_ has obviously moved severely in the past year!).

    • MaxBarraclough 8 hours ago

      > Anywhere you batch frees to avoid calling it in a hot path is a GC

      I wouldn't say so. An implementation of the free function could do batching of this kind internally.

  • sealWithIt 1 day ago

    On the one hand, I agree with you; on the other, I also find reasonable that lower cognitive load may keep justifying the existence of GC-based languages even for AI.

  • titzer 1 day ago

    No, manual memory management, with or without static ownership, is not inherently more efficient than GC. Bump pointer allocation is crazy fast. GC work is proportional to live data, not dead data. GC can do compaction and locality optimizations. Manual deallocation is proportional to dead data, unless you are doing bulk deallocation with regions. Stack allocation is crazy fast. It's a little beyond what state-of-the-art compilers do by default, but it's just pointer analysis that will also get better with AI optimizing and improving compilers.

    • paulddraper 1 day ago

      GC is fast.

      GC is not memory efficient.

      If a primary concern is optimizing memory usage, you will avoid GC languages.

      • pron 1 day ago

        That is not so simple. First, the differences among different GCs are bigger than between some GCs and no GC. Second, moving GCs - of the kind used in HotSpot (as well as V8 and .NET), but not Go or Python (which are as different from each other as they are from moving GCs) - do, indeed have a higher memory utilisation, but it's not necessarily inefficient, because most of it comes into effect when CPU is under higher pressure than RAM (that's when you get a larger young-gen, as the heap overhead is a function of the allocation rate, not of the live set which, in turn, means a high CPU usage). RAM and CPU are not independent resources; the most basic way to think about this is that your machine is exhausted when the first of CPU and RAM is exhausted. For example, I can take your car battery or I can take your petrol, but if I take your car battery when you don't have petrol, I'm not impacting your use of the car, so I'm not really creating any "inefficiency".

        You are right, though, that in a very memory-constrained environment, usually embedded devices, you may want to lower memory consumption regardless of anything else, and in such environments moving GCs are probably not what you want (although other kinds of GC could be).

        More here: https://youtu.be/xr73mR7ii9M

      • mike_hearn 10 hours ago

        Are you assuming here that AI will never, ever code a memory leak? Because I don't think that's the case.

        Rust is not the silver bullet its community is trying to meme it into being. Its approach to memory allocation only works in some cases, the rest of the time you're back to refcounting and the potential for leaks.

        For AI it's especially bad because they never memorize full codebase context in the way an experienced dev would, so it all becomes very dependent on how your codebase is structured and how you prompt it. GC just eliminates all those bugs and the associated token cost of trying to find and fix them.

        • steveklabnik 7 hours ago

          Rust has never claimed to prevent memory leaks.

    • KMag 17 hours ago

      > GC work is proportional to live data, not dead data.

      This is true of your basic moving collectors. It's generally not true of reference-counting collectors (including Bacon-style cycle detecting collectors). It's also not true of tracing collectors that have a step of sweeping all live and dead objects, such as mark-sweep or mark-compact.

  • jeffbee 1 day ago

    Can't agree. GC language platforms are more efficient in many ways. Look at the hoops that C allocators try to jump through just to get decent THP coverage which a compacting GC gets essentially for free.

  • quotemstr 1 day ago

    Why would you think that malloc/free style memory management is more efficient? Both functions can be quite expensive, and they work synchronously. GCes give you compaction, compressed pointers, and bump pointer young generation operation as well as offloading allocation machinery to threads other than your business logic one. It's manual memory management that's going the way of the dinosaur, and the current interest in manual memory management is a fad.

    • solyin 21 hours ago

      But there's the GC pause. Fundamentally there's no way around that, though I'm sure there are lots of modern techniques to make it not a problem.

      • quotemstr 21 hours ago

        The GC pause hasn't been a problem for years. Myth that just won't die. Have you tried a modern latency-optimized JVM GC? No, it doesn't pause in any way you'll care about.

        Now, free/Drop/etc. the root a tree of 100k nodes. That's a pause.

        • solyin 21 hours ago

          Golang still got that issue.

  • hashmash 1 day ago

    Implementing efficient concurrent algorithms without a tracing collector is quite difficult, and it often requires the use of hazard pointers or epochs. Both act like mini collectors, and they also require special tuning.

    Unless you want AI generated code to be single threaded and not take advantage of multiple cores, some form of collector is still required. If you want to avoid heap fragmentation issues, a compacting collector is generally recommended.

  • pron 1 day ago

    That is just wrong. Moving GCs were invented because they're more efficient than manual memory management. In fact, in his famous 1986 paper, Garbage Collection Can Be Faster Than Stack Allocation, Andrew Appel wrote: "It is easy to believe that one must pay a price in efficiency for this ease in programming… But this is simply not true."

    Moving GCs free you from the high overhead of malloc/free runtimes, something that us old C and C++ programmers have had decades of experience with, and why our colleagues developed large, complex runtimes such as TCMalloc and jemalloc to remove some of that overhead, even though not nearly as much as moving GCs do.

    The reason low-level languages don't use moving collectors despite their superior efficiency is because they must give up on some performance optimisations for the abillity to interact with the hardware or OS directly, and that means that pointers must be stable machine addresses (this is why you need a richer FFI layer in Java). Moving collectors work in many ways like arenas. They don't need to maintain free lists whenever an object is allocated or freed (in fact, they never free any object at all), and memory is freed in bulk. This is not the only performance that low-level languages leave on the table because above everything else they must be low level.

    Every single language that can use moving collectors does (it's also a matter of resources; moving collectors are hard to implement, especially the modern pauseless ones).

    Sadly, people who are not compiler or memory allocator engineers don't know that, so they believe "popular wisdom" disseminated by other non-experts.

    P.S.

    Re AI: It's certainly possible that AI will be able to write good, efficient, large software some day, perhaps soon, but if it's able to perform the global program transformations required to replicate the optimisations of a moving collector without one, it won't need the simpler and more local optimisations done by a compiler, and will be able to work better without one, directly in machine code, in which case all compilers and programming languages are also done. In other words, moving GCs will not be a relic of the past before compilers are (or maybe roughly around the same time).

    • KMag 17 hours ago

      Though, if your compiler generates stack maps to facilitate pointer moving at runtime, one could have a system that used a bump allocator and compaction of a GC without running the trace portion of GC. It could also use a per-thread list of freed objects to batch up the freeing. Then we get into a semantic argument as to whether compile-time GC is really GC.

      • pron 27 minutes ago

        I'm not sure what you mean exactly because there can be pointers in the heap (although work on compile-time GC based on arena-like "regions" has been around for decades), but there are plenty more optimisations that generational moving collectors can do, and we're working on them. But no matter what you do, once you can move objects you need an FFI layer between the "movable stack" and the hardware/OS, which is not something you want in a low-level language (because once you need an FFI layer, you might as well use a more high level language for the parts above it).

    • preg_match 17 hours ago

      Moving GC’s can be more efficient due to their much cheaper bump allocations and cache usage by putting objects together.

      But, this is only assuming we use only the most naive allocation algorithms when managing memory. Yes, it’s true allocating is faster with a GC than malloc. But then you have to remember - most GC languages force way more allocations than necessary. Java is perhaps the worst about this, where almost everything goes on the heap.

      You can use arena allocators to put objects close together and get allocations almost for free. And, you can put more objects on the stack next to each other, avoiding allocation cost altogether. In practice, this will greatly outperform a GC, even if you don’t take into account collection cost! What will not outperform a GC is doing a million tiny allocations.

      But, in languages like C and C++, you don’t typically do that. A million tiny allocations is a Java thing. That’s why when people compare runtimes they sometimes mess up. Yes, if you implement a Java algorithm verbatim in C++… it will perform poorly. So, don’t.

      Also, moving GCs only perform adequately when memory pressure is low. The more memory usage and churn you have, the more moving, compacting, and collecting you have to do. The same is not true of other allocation techniques. A lot of comparisons are not well thought out because they’ll be running at 10% heap utilization. Of course that performs well, it’s likely the GC isn’t even performing many collections. But that’s not true at 90% utilization. Hence why collectors like the G1GC were historically disabled on memory constrained environments.

      • pron 8 hours ago

        It is true that if the size of your RAM is close to the size of your live set, a moving GC isn't the best choice, but this doesn't happen for most software on machines that we would call computers these days.

        As for performance differences in practice, it is easy to encounter Java programs, typically either large ones or very concurrent ones, that easily outperform C++ programs by a factor of two or more, just as it's easy to see examples of the opposite, typically in smaller or less concurrent programs. This is what we would expect.

        As experienced low-level programmers know, it is quite easy to avoid the operations that are slow in C++ or other low-level languages - heap allocations and dynamic dispatch - when programs are relatively small and/or young. As programs grow larger, it gets more and more expensive to do this because these operations offer high abstraction, the ability to keep changes local. And it is these two operations that the HotSpot JVM was built to optimise.

        I gave a talk about exactly that at the last Devoxx: https://youtu.be/vhDwy1eCXbs

    • adrian_b 10 hours ago

      It is true that some implementations of garbage collectors can be faster than some implementations of mallloc/free.

      Nonetheless, the title of the paper by Andrew Appel (Garbage Collection Can Be Faster Than Stack Allocation) was just a false statement, which today would be called a clickbait title.

      No method of dynamic memory allocation can be faster than stack allocation, if the latter is done correctly.

      In Appel's paper, he claimed that if you have tons of memory, than the overhead of freeing memory by the garbage collector can become low, because of its laziness, i.e. the garbage collector will be invoked only when there is a huge amount of memory that must be freed, so the overhead per freed byte will be low, even lower than when stack-allocated memory is freed word-by-word.

      However, this comparison is a classic example of a straw man. When memory is abundant, many compilers implement an optimization where stack deallocation is delayed as much as possible, in order to free with a single instruction large amounts of memory. So the same lazy deallocation strategy is also available for stacks, and the difference between a single instruction needed for stack deallocation and a long procedure needed by the garbage collector is enough to ensure that for any realistic amount of available memory, stack allocation is the most efficient.

      Obviously, stack allocation is not applicable to everything, which is why other techniques, like reference counts and garbage collectors, are also necessary.

      • pron 10 hours ago

        Yes, I'm aware that the title of that paper isn't actually a true statement.