This issue report feels like it ought to be accompanied by a fix. If you think you can do better than journald's existing format, propose a new one with tests to prove it. GenAI makes this much easier than it used to be.
>This issue report feels like it ought to be accompanied by a fix.
systemd is a stewarded FOSS, which means there's a team behind it, who are getting paid, and develop this software with release cycles, backwards compatibility guarantees, architectural decisions, and such.
These people know better. I usually only prepare fixes for FOSS one-man-show which have little to no maintenance, otherwise I prefer professionals to handle it. Sometimes "suggestion" PR is worse than a triaged issue IMO.
In this case, it's also a very ego-driven project.
Honestly, the few times I went into systemd source (to deal with how they didn't document some critical information without which I couldn't ensure coexistence of other software, software needed for functionality systemd didn't expose), I found it a total mess - combined with very loud and explicit ways the decision of the "stewards" were defended by the team, I would be frankly wary of trying to contribute anything non-trivial.
Designing a new on-disk format seems like a pretty far reaching architectural decision... I don't think that's an appropriate target for a drive-by fix from a new contributor
Or you talk with the others first to see what kind of setup everyone thinks is good. I'd find it strange if someone barges into my project with a pull request that fundamentally changes the design of a major component
Because you become responsible for feeding and taking said care of horse and dealing with any technical debt associated with it. If someone submits code to a project that I maintain that's going to make my life difficult in the future, I'm not going to accept it.
It's basically comparing an append-only fixed-format text file with a queryable database. Of course the former is going to be more performant on writes.
> syslog doesn't have nearly the functionality that journald + journalctl does.
A huge feature list doesn't matter much if the software is bad. Given that journald still irrecoverably corrupts its logs even after all these years and -apparently- suffers from substantial write amplification, I'm gonna stick with my ordinary syslog implementations, thanks.
Also, in regards to your original comment:
> This issue report feels like it ought to be accompanied by a fix.
This smells a lot like the "Don't come to me with problems, come to me with fixes." order that a lot of mid-level and director-level management really loved to make five, ten years back. [0] While this sounds like a hard-charging order and gives the impression that it's bringing much-needed discipline to lazy-ass subordinates, the truth of the matter is that its actual effect [1] is to get people to shut the fuck up about the company's problems. The job of most mid-level and nearly all director-level management is to do inter-organization coordination. Most low-level folks don't come to mid- or director-level management with problems they can solve. After all, if they could solve them, they would... talking to folks in that layer of management is usually a huge drag. Most low-level folks only come to these sorts of folks with issues that require inter-organization coordination!
So, yeah... the only obligation of someone who's reporting a bug is to provide a reasonably well-written bug report accompanied with reproduction instructions and diagnostics that are as clearly written as is reasonably possible. Reporters of performance bugs are under no obligation to suggest how to eliminate the bug... especially not if the project they're reporting the bug against has both paid maintainers and claims it's the infrastructure on top of which all Linux systems should be built. Corporate-backed projects that make such grand claims put themselves in a radically different class than the one that covers hobby or small-time projects.
[0] AIUI, it came out of Google, but my understanding might be incorrect.
[1] ...regardless of whether or not that effect is intentional...
Nobody's talking about an obligation here. It's open source, and the maintainer owes non-paying users nothing. So if you want something fixed, it's now easier than ever to get involved in the fix. Nothing more, nothing less.
(Also, I'm not entirely sure this is a bug so much as an inefficiency report. Consumption of storage space isn't a documented or promised behavior, nor is the behavior technically incorrect. It's just wasteful.)
If you think you can do better than journald's existing format, propose a new one with tests to prove it.
The fact that you're personally powerless to enforce an obligation doesn't change the fact that you're talking about creating an obligation.
> I'm not entirely sure this is a bug so much as an inefficiency report.
Performance bugs absolutely are bugs... especially when they're in a long-running corporation-backed project that presents itself as the project atop which all Linux systems should be built.
Well, I didn't intend to suggest an obligation, more of a best practice or challenge. Please put more faith in my intentions over whatever interpretation of my words you want to make.
Per our Guidelines:
> Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith.
Please explain, because it's not coming across that way. It's coming across as needlessly picky and combative, especially after I told you what I meant (or, at least, didn't mean) and you continued to argue with me.
I'm neither required nor strongly obligated to do so, nor do I see significant personal benefit to doing so. So, I will not.
However, these days it's quick and easy to command an LLM-based system to generate most any text. Before one demands an explanation from a human, perhaps one should machine-generate a plausible-sounding explanation and present that along with one's demand for a human-synthesized one?
> Way to double down on the “needlessly picky and combative” angle, dude.
Sit and consider the points of similarity between my refusal-shaped reply and the entire conversation we had prior to it and you might find enlightenment, in the style of those classic Zen tales. Perhaps an LLM-based tool might be able to assist you in this, or maybe it will be distracting and misleading.
"Don't come to me with problems, come to me with fixes." has always been a thing. You shouldn't take it too literally or personally. For "fixes" read alternatives or suggestions. e.g. hook in a senior engineer that you know is intimate with the system.
Indeed. It also means "don't come to me with problems alone." Yes, one can come with a problem, but the exhortation is to come with possible solutions as well and seek guidance on which one is best.
Yes, a lot of awful practices have always been a thing.
> e.g. hook in a senior engineer that you know is intimate with the system.
unless, of course, you don't know said engineer because you don't even work in the same company, you're a just a user seeing a problem in an app you use
> unless, of course, you don't know said engineer because you don't even work in the same company...
Or they work in a different part of the fairly-large company that you both work for.
I guess emmelaich either missed the part of my commentary where I talked about handling inter-organization communication, and/or has never worked at a company where it's simply impossible to know everyone who could reasonably be relevant to the stuff that the company works on.
I would argue the difference is that upstream is well known for being defensive about their ideas and pushing back.
On LKML you might get cursed out, but if your fix is solid fix, it has high chances of getting through. Regardless of how true it would be in reality, the atmosphere created by upstream is that I do not expect the same with journald unless you convince redhat management
What is the real-world use for these features? Who is this built for?
Modern drives will read data at 500MB/s, sometimes even more. Your log files are approaching tens if not hundreds of gigabytes before a sequential read stops being a viable option. Tinies modicum of partitioning by date and source basically makes it a complete nothingburger.
It feels like you didn't read the design rationale, because the use cases and issues are listed therein. Maybe you don't see the value, but that doesn't mean it's not there for others. I certainly find its query features useful.
The point is that ripgrep will give you basically all the same query features just by being fast. A more structured format makes sense, but the indexing is not obviously adding value in most cases.
The difference between grep/ripgrep and querying by field is the difference between a full table scan and an index query. Query performance is a very good reason to have databases. ripgrep is certainly fast, but it's still O(N). Doing complete file scans also trashes the OS's buffer cache.
I really want to love journald (it sounds like it's aiming for a good system) but I share the other commenter(s)' frustration here about journald being slower than just pulling out ripgrep on regular files.
We have some services at work that log to text files and some to journald.
The log volume to file is >> the log volume to journald.
Yet `rg query myservice.2026-08-01.log` seems to always wind up being faster and better than something like `journalctl -u myservice.service --since '2026-08-01' --until '2026-08-02' -g 'query'`. (The tab completion and discoverability is also better, I guess)
I have been complaining about journald abysmal performances for almost as long as I can remember. Here is my latest documented benchmark from 2023, which was slightly better than the one I ran in 2020.
I'm no fan of journald, but I have some methodological issues with this test.
The reads from /tmp/all.log are almost certainly cached since you just wrote the file, and will basically boil down to a memcpy call, rather than actual disk I/O. Speed difference isn't as big as you would think on a modern SSD, but it isn't nothing either.
Running this between calls should flush the changes to disk and then drop the page cache, making for a fairer test.
I did, and I still don't get why you would want this monstrosity over a structured append-only log file. If you want to index the data, you can do that when you roll over the file. That way you get the exact same robustness guarantees, without the insane architecture.
Like ultimately it isn't even fast, journalctl is so bad at rendering text that it's approximately still as slow as seeking in a 400 MB .log-file using less.
Anyone with any sort of scale where you actually need indexing immediately drops journald and uses loki or elasticsearch instead. Journald is not even remotely a contender in that space.
> Anyone with any sort of scale where you actually need indexing immediately drops journald and uses loki or elasticsearch instead. Journald is not even remotely a contender in that space.
That I agree with. I don't personally use journalctl much these days, particularly now that practically everything's a container and all their logs are getting shipped off-host for indexing. But I get why, 14 years ago, it was considered a good idea.
>This issue report feels like it ought to be accompanied by a fix.
systemd is a stewarded FOSS, which means there's a team behind it, who are getting paid, and develop this software with release cycles, backwards compatibility guarantees, architectural decisions, and such.
These people know better. I usually only prepare fixes for FOSS one-man-show which have little to no maintenance, otherwise I prefer professionals to handle it. Sometimes "suggestion" PR is worse than a triaged issue IMO.
In this case, it's also a very ego-driven project.
Honestly, the few times I went into systemd source (to deal with how they didn't document some critical information without which I couldn't ensure coexistence of other software, software needed for functionality systemd didn't expose), I found it a total mess - combined with very loud and explicit ways the decision of the "stewards" were defended by the team, I would be frankly wary of trying to contribute anything non-trivial.
Designing a new on-disk format seems like a pretty far reaching architectural decision... I don't think that's an appropriate target for a drive-by fix from a new contributor
You could however choose to maintain a fork. If you had time. Few people do.
Or you talk with the others first to see what kind of setup everyone thinks is good. I'd find it strange if someone barges into my project with a pull request that fundamentally changes the design of a major component
Sure, a concrete proposal first would be a good idea.
That said, would you look a gift horse in the mouth?
Because you become responsible for feeding and taking said care of horse and dealing with any technical debt associated with it. If someone submits code to a project that I maintain that's going to make my life difficult in the future, I'm not going to accept it.
It doesn't matter how free a turd sandwich is.
There's no proposal that we can evaluate to determine whether it's a turd sandwich or not.
Well, they are comparing in comments with syslog, and it does better, as you asked. Syslog was there for 46 years.
syslog doesn't have nearly the functionality that journald + journalctl does. Take a look at journalctl's man page: https://www.freedesktop.org/software/systemd/man/latest/jour...
See also the design rationale: https://docs.google.com/document/u/0/d/1IC9yOXj7j6cdLLxWEBAG...
It's basically comparing an append-only fixed-format text file with a queryable database. Of course the former is going to be more performant on writes.
> syslog doesn't have nearly the functionality that journald + journalctl does.
A huge feature list doesn't matter much if the software is bad. Given that journald still irrecoverably corrupts its logs even after all these years and -apparently- suffers from substantial write amplification, I'm gonna stick with my ordinary syslog implementations, thanks.
Also, in regards to your original comment:
> This issue report feels like it ought to be accompanied by a fix.
This smells a lot like the "Don't come to me with problems, come to me with fixes." order that a lot of mid-level and director-level management really loved to make five, ten years back. [0] While this sounds like a hard-charging order and gives the impression that it's bringing much-needed discipline to lazy-ass subordinates, the truth of the matter is that its actual effect [1] is to get people to shut the fuck up about the company's problems. The job of most mid-level and nearly all director-level management is to do inter-organization coordination. Most low-level folks don't come to mid- or director-level management with problems they can solve. After all, if they could solve them, they would... talking to folks in that layer of management is usually a huge drag. Most low-level folks only come to these sorts of folks with issues that require inter-organization coordination!
So, yeah... the only obligation of someone who's reporting a bug is to provide a reasonably well-written bug report accompanied with reproduction instructions and diagnostics that are as clearly written as is reasonably possible. Reporters of performance bugs are under no obligation to suggest how to eliminate the bug... especially not if the project they're reporting the bug against has both paid maintainers and claims it's the infrastructure on top of which all Linux systems should be built. Corporate-backed projects that make such grand claims put themselves in a radically different class than the one that covers hobby or small-time projects.
[0] AIUI, it came out of Google, but my understanding might be incorrect.
[1] ...regardless of whether or not that effect is intentional...
Nobody's talking about an obligation here. It's open source, and the maintainer owes non-paying users nothing. So if you want something fixed, it's now easier than ever to get involved in the fix. Nothing more, nothing less.
(Also, I'm not entirely sure this is a bug so much as an inefficiency report. Consumption of storage space isn't a documented or promised behavior, nor is the behavior technically incorrect. It's just wasteful.)
You: [0]
Also you: [1]
The fact that you're personally powerless to enforce an obligation doesn't change the fact that you're talking about creating an obligation.
> I'm not entirely sure this is a bug so much as an inefficiency report.
Performance bugs absolutely are bugs... especially when they're in a long-running corporation-backed project that presents itself as the project atop which all Linux systems should be built.
[0] <https://news.ycombinator.com/item?id=49292944>
[1] <https://news.ycombinator.com/item?id=49291746>
Well, I didn't intend to suggest an obligation, more of a best practice or challenge. Please put more faith in my intentions over whatever interpretation of my words you want to make.
Per our Guidelines:
> Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith.
> Please respond to the strongest plausible interpretation... Assume good faith.
That's what I did. So, right back at you.
> That's what I did
Please explain, because it's not coming across that way. It's coming across as needlessly picky and combative, especially after I told you what I meant (or, at least, didn't mean) and you continued to argue with me.
> Please explain...
I'm neither required nor strongly obligated to do so, nor do I see significant personal benefit to doing so. So, I will not.
However, these days it's quick and easy to command an LLM-based system to generate most any text. Before one demands an explanation from a human, perhaps one should machine-generate a plausible-sounding explanation and present that along with one's demand for a human-synthesized one?
facepalm
Way to double down on the “needlessly picky and combative” angle, dude.
> Way to double down on the “needlessly picky and combative” angle, dude.
Sit and consider the points of similarity between my refusal-shaped reply and the entire conversation we had prior to it and you might find enlightenment, in the style of those classic Zen tales. Perhaps an LLM-based tool might be able to assist you in this, or maybe it will be distracting and misleading.
GLHF and all that.
> you might find enlightenment, in the style of those classic Zen tales
Doctor, heal thyself.
"Don't come to me with problems, come to me with fixes." has always been a thing. You shouldn't take it too literally or personally. For "fixes" read alternatives or suggestions. e.g. hook in a senior engineer that you know is intimate with the system.
Indeed. It also means "don't come to me with problems alone." Yes, one can come with a problem, but the exhortation is to come with possible solutions as well and seek guidance on which one is best.
And it's always been cope. Only the worst kind of person would rather not know about a problem if it's not already solved for them.
Yes, a lot of awful practices have always been a thing.
> e.g. hook in a senior engineer that you know is intimate with the system.
unless, of course, you don't know said engineer because you don't even work in the same company, you're a just a user seeing a problem in an app you use
> unless, of course, you don't know said engineer because you don't even work in the same company...
Or they work in a different part of the fairly-large company that you both work for.
I guess emmelaich either missed the part of my commentary where I talked about handling inter-organization communication, and/or has never worked at a company where it's simply impossible to know everyone who could reasonably be relevant to the stuff that the company works on.
I would argue the difference is that upstream is well known for being defensive about their ideas and pushing back.
On LKML you might get cursed out, but if your fix is solid fix, it has high chances of getting through. Regardless of how true it would be in reality, the atmosphere created by upstream is that I do not expect the same with journald unless you convince redhat management
What is the real-world use for these features? Who is this built for?
Modern drives will read data at 500MB/s, sometimes even more. Your log files are approaching tens if not hundreds of gigabytes before a sequential read stops being a viable option. Tinies modicum of partitioning by date and source basically makes it a complete nothingburger.
It feels like you didn't read the design rationale, because the use cases and issues are listed therein. Maybe you don't see the value, but that doesn't mean it's not there for others. I certainly find its query features useful.
The point is that ripgrep will give you basically all the same query features just by being fast. A more structured format makes sense, but the indexing is not obviously adding value in most cases.
The difference between grep/ripgrep and querying by field is the difference between a full table scan and an index query. Query performance is a very good reason to have databases. ripgrep is certainly fast, but it's still O(N). Doing complete file scans also trashes the OS's buffer cache.
Writing multiple pages for a few hundred actual bytes also thrashes the OS cache...
Not if you use the correct cache hint flags on the write.
I really want to love journald (it sounds like it's aiming for a good system) but I share the other commenter(s)' frustration here about journald being slower than just pulling out ripgrep on regular files.
We have some services at work that log to text files and some to journald.
The log volume to file is >> the log volume to journald. Yet `rg query myservice.2026-08-01.log` seems to always wind up being faster and better than something like `journalctl -u myservice.service --since '2026-08-01' --until '2026-08-02' -g 'query'`. (The tab completion and discoverability is also better, I guess)
What do the metrics look like in practice? seems is a bit too handwavy. I get that impressions matter, but data is actionable.
I have been complaining about journald abysmal performances for almost as long as I can remember. Here is my latest documented benchmark from 2023, which was slightly better than the one I ran in 2020.
$ time journalctl > /tmp/all.log
real 1m11.364s user 0m52.299s sys 0m6.540s
$ time wc -l /tmp/all.log 3659597 /tmp/all.log
real 0m0.152s user 0m0.056s sys 0m0.096s
$ time journalctl | grep sshd | wc -l
12944
real 0m53.973s user 0m49.535s sys 0m5.210s
$ time grep sshd /tmp/all.log | wc -l 12944
real 0m0.429s user 0m0.332s sys 0m0.100s
https://github.com/systemd/systemd/issues/2460#issuecomment-...
I'm no fan of journald, but I have some methodological issues with this test.
The reads from /tmp/all.log are almost certainly cached since you just wrote the file, and will basically boil down to a memcpy call, rather than actual disk I/O. Speed difference isn't as big as you would think on a modern SSD, but it isn't nothing either.
Running this between calls should flush the changes to disk and then drop the page cache, making for a fairer test.
$ sudo sync
$ echo 3 | sudo tee /proc/sys/vm/drop_caches
I did, and I still don't get why you would want this monstrosity over a structured append-only log file. If you want to index the data, you can do that when you roll over the file. That way you get the exact same robustness guarantees, without the insane architecture.
Like ultimately it isn't even fast, journalctl is so bad at rendering text that it's approximately still as slow as seeking in a 400 MB .log-file using less.
Anyone with any sort of scale where you actually need indexing immediately drops journald and uses loki or elasticsearch instead. Journald is not even remotely a contender in that space.
> Anyone with any sort of scale where you actually need indexing immediately drops journald and uses loki or elasticsearch instead. Journald is not even remotely a contender in that space.
That I agree with. I don't personally use journalctl much these days, particularly now that practically everything's a container and all their logs are getting shipped off-host for indexing. But I get why, 14 years ago, it was considered a good idea.
Nothing says Linux like "information is hosted on a site owned by a user-hostile and privacy-mulching corporation."
You forgot to link to the tests that prove it