> Now, POSIX says that data blocks for a hard link “may be included” in pax archives, but the implications of this are not entirely clear.
How so? If the data blocks are included, then those are the intended content of the file behind that hard link. They should overwrite the previous content of the file.
> So the 512 bytes after the hard link are its data, and not the header of empty as we archived it?
Yes, precisely.
> How can a hard link have different contents from its previous counterpart file?
Why, very simply: while the archive was being created, the file behind the two different hard links changed; the archival utility noticed it, and included the new content of the file as the content for the second hard link instead of rewinding and overwriting the original file in the archive (which would be impossible to do on a tape, you know). At least I believe that's what was the rationale for this allowance, and the intended behavior on the extraction.
“Precisely” and “simply” are not words I would use to describe pax’s prescriptions around data-bearing hardlink members.
(I maintain several of the tar parsers mentioned in this post, which I appreciate for its clear treatment of a larger differential problem in the tar ecosystem.)
Fair enough, although in all honestly, "precisely" and "simply" are not words I would personally use to describe anything about POSIX's stipulations on the pax's behaviour either. But as you probably know way better than I do, the whole tar standardization effort was quite a train wreck (IIRC one of the major discussions was "shouldn't we just ditch tar entirely in favor of cpio"... yeah), with lots of different people with different interests and opinions, so what we got is what we got. It's a fine format for a single-system/installation archival and backup; for portable data interchange? Not so that much. Hardened against malicious actors? Absolutely not in the slightest.
Were you involved with this process? I'm the author of the blog post, and I'm looking for people to discuss a rewrite of that POSIX pax page. What bothers me most about that section is that it prescribes that if you don't support a (vendor) typeflag, you should treat it as a header for a regular file, extract it, and continue with the next header. The problem is that vendor extensions may change the framing of the archive, and POSIX sets a precedent for that with pax headers, where you can override size=... of the next header. It should just leave it unspecified what you should do when you encounter an unrecognized header, or say that the implementation may error. There are more such improvements to be made.
There's a prominent tar implementation missing. paxmirabilis is tar, cpio, and pax all in one, as a multiple-call binary. It's what one gets as pax in Debian, in fact.
>NET, Julia and uv's tar-codec - These implementations reject the tarball. I believe this is the best thing you can do, even if it's technically not POSIX-compliant.
While this is the safest choice, it is the least helpful, I would always prefer to get something out of an archive then nothing. Damage/irregularity to an archive shouldn't prevent data inspection, just a warning and a way to fall back with caveats for what can be extracted.
You really want two different modes. If you're parsing untrusted tarballs (eg. ones uploaded to your website) then you should err on the side of rejecting anything that's even slightly suspicious. But in another mode you are a user who needs to extract any data you can from your corrupted family photos archive.
Really what I think I'd want out of any archive utility is a safe default mode (for whatever "safe" means here), but the ability for a user to pull out everything in the archive one way or another. If an archive can be read as having multiple versions of a given file, the user should be able to pick to extract any or all of them at will, but only on purpose.
> While this is the safest choice, it is the least helpful, I would always prefer to get something out of an archive then nothing. Damage/irregularity to an archive shouldn't prevent data inspection, just a warning and a way to fall back with caveats for what can be extracted.
I would argue the correct answer can have both.
Fail by default, fail with a clear error that says the file is damaged, then offer an explicitly opt-in path to try your best anyways, preferably one where the command line option is something like --dangerous-try-extract-anyways and will itself fail on non-broken files so it can't just be shoved in a script as a way to "make sure it works".
I’ve been working on an open source tool for creating SBOMs from binaries/file systems. Automated extraction of archives like zip/tar files present interesting challenges - for a while I’ve known about OS filesystem formatting gotchas like archives from a Linux-based system with symlinks or case colliding names not extracting correctly on NTFS (and HFS+ for the latter case).
But the ambiguity of this tar files… I guess the “correct” extraction for what I’m doing would require figuring out what the system that is processing the tar files used and copy that behavior, which is non-trivial if you don’t know what an embedded device is using behind the scenes.
Astral blogged about their new tar parser for uv recently, see https://astral.sh/blog/tar-codec. I've included it in my work-in-progress test suite https://stoppels.ch/tar/ which shows it's by far the strictest tar library out there.
For your use case it would make sense to pick the strictest tar parser, such that you can error or warn on most ambiguities. In practice these ambiguities are rare.
A strict parser makes sense for flagging that there are issues, but that also can’t block extraction; it needs to be configurable enough that users can make a decision on how to handle the ambiguity.
There are many scenarios (such as fetching package distributions over the internet) where the user lacks sufficient information to make an informed and secure decision about how to safely parse and extract a malformed tar (or any other archive format).
(Another way of saying that is that we live in a post-Postel world[1].)
That’s true, but also much more general than our SBOM generator is targeting. We’re typically looking at filesystems pulled from embedded systems, or their update files. Ultimately we want to accurately represent what the state of the file system in the device looks like, which can require reverse engineering the update process to determine how to handle ambiguous cases (“correct” is matching what the device firmware update does).
I feel like the most faithful extraction would dictate that any field in the original header should take precedence over the PaxHeader's iff the PaxHeader field's value would've been representable in the original field. Anyone have thoughts?
Right, it just feels wrong to me for pax to mandate that. My argument is that the whole point of the extension is to handle the cases that the original can't handle while still preserving existing behavior for what is already supported - hence making it an extension. If you're fully overriding the original then you just have a different format altogether and aren't respecting the original at all, and therefore shouldn't be allowing decoders for the original format to read the new one.
I think this is true in a sense. But the other major benefit to pax is that it normalizes metadata handling away from the mess that is fixed octal fields in a V7 tar header. pax is its own mess, but at least it’s extremely clear about how spaces, NULs, etc. interact with values.
(On a more basic level, I can imagine it would be even more differential prone for the precedence rule to be “pax, but not always.” pax still makes it complicated but that would be another whole circle of hell.)
Hit up Wikipedia for tarball format and you may agree with me. It's a classic example of a standard where you look at it and go, "??? Who asked for that?". I'm not surprised the article focuses on pax when finding tar inconsistencies. Original tar is just fine.
I don’t really think pax is the issue here, although pax is a bad specification in absolute terms. But “classic” (V7) tar just isn’t specified at all, and has plenty of its own differentials.
(Users also want things like filenames that are longer than 100 characters, for obvious reasons.)
Pax has an interesting use in POSIX: combined with od, pax is the only way (in POSIX) to get a Unix timestamp from the command line. The exact command line is left as an exercise for the interested reader.
> Now, POSIX says that data blocks for a hard link “may be included” in pax archives, but the implications of this are not entirely clear.
How so? If the data blocks are included, then those are the intended content of the file behind that hard link. They should overwrite the previous content of the file.
> So the 512 bytes after the hard link are its data, and not the header of empty as we archived it?
Yes, precisely.
> How can a hard link have different contents from its previous counterpart file?
Why, very simply: while the archive was being created, the file behind the two different hard links changed; the archival utility noticed it, and included the new content of the file as the content for the second hard link instead of rewinding and overwriting the original file in the archive (which would be impossible to do on a tape, you know). At least I believe that's what was the rationale for this allowance, and the intended behavior on the extraction.
“Precisely” and “simply” are not words I would use to describe pax’s prescriptions around data-bearing hardlink members.
(I maintain several of the tar parsers mentioned in this post, which I appreciate for its clear treatment of a larger differential problem in the tar ecosystem.)
Fair enough, although in all honestly, "precisely" and "simply" are not words I would personally use to describe anything about POSIX's stipulations on the pax's behaviour either. But as you probably know way better than I do, the whole tar standardization effort was quite a train wreck (IIRC one of the major discussions was "shouldn't we just ditch tar entirely in favor of cpio"... yeah), with lots of different people with different interests and opinions, so what we got is what we got. It's a fine format for a single-system/installation archival and backup; for portable data interchange? Not so that much. Hardened against malicious actors? Absolutely not in the slightest.
Were you involved with this process? I'm the author of the blog post, and I'm looking for people to discuss a rewrite of that POSIX pax page. What bothers me most about that section is that it prescribes that if you don't support a (vendor) typeflag, you should treat it as a header for a regular file, extract it, and continue with the next header. The problem is that vendor extensions may change the framing of the archive, and POSIX sets a precedent for that with pax headers, where you can override size=... of the next header. It should just leave it unspecified what you should do when you encounter an unrecognized header, or say that the implementation may error. There are more such improvements to be made.
I think that as of last Sunday you discovered the best place to be for that. (-:
Although now comes the hard part: filing defect reports and convincing other people.
Ah, excellent, so my email did come through ;)
There's a prominent tar implementation missing. paxmirabilis is tar, cpio, and pax all in one, as a multiple-call binary. It's what one gets as pax in Debian, in fact.
* https://mbsd.evolvis.org/pax.htm
* https://manpages.debian.org/trixie/pax/index.html
No, it's not libarchive's bsdtar. There's more than 1 tar in the BSD world.
This guy, what a hero. He even included a little graphic to show how this "hardest tarball" looks like.
At the bottom, there's a link to much more edge cases.
It reminds me of this beauty: "Parsing JSON is a Minefield" https://news.ycombinator.com/item?id=16897061
TIL there are different hardness for tarballs...
>NET, Julia and uv's tar-codec - These implementations reject the tarball. I believe this is the best thing you can do, even if it's technically not POSIX-compliant.
While this is the safest choice, it is the least helpful, I would always prefer to get something out of an archive then nothing. Damage/irregularity to an archive shouldn't prevent data inspection, just a warning and a way to fall back with caveats for what can be extracted.
It depends where. Inside a server application, fail fast and fail safe.
Now if I'm a user trying to extract the file then I'd want to get as much as possible.
You really want two different modes. If you're parsing untrusted tarballs (eg. ones uploaded to your website) then you should err on the side of rejecting anything that's even slightly suspicious. But in another mode you are a user who needs to extract any data you can from your corrupted family photos archive.
It is somewhat unfortunate that tar has two main purposes with very different threat models, namely backup and (software) distribution.
The POSIX standard promotes best-effort extraction, which is reasonable for backups, but a liability for distribution.
Julia's Tar.jl was developed for Pkg.jl and tar-codec for uv, so these are focused on the security aspect of software distribution.
Really what I think I'd want out of any archive utility is a safe default mode (for whatever "safe" means here), but the ability for a user to pull out everything in the archive one way or another. If an archive can be read as having multiple versions of a given file, the user should be able to pick to extract any or all of them at will, but only on purpose.
> While this is the safest choice, it is the least helpful, I would always prefer to get something out of an archive then nothing. Damage/irregularity to an archive shouldn't prevent data inspection, just a warning and a way to fall back with caveats for what can be extracted.
I would argue the correct answer can have both.
Fail by default, fail with a clear error that says the file is damaged, then offer an explicitly opt-in path to try your best anyways, preferably one where the command line option is something like --dangerous-try-extract-anyways and will itself fail on non-broken files so it can't just be shoved in a script as a way to "make sure it works".
> This tarball is one of 134 tricky tarballs that I'm collecting and categorizing in my test suite haampie/tar-minefield
Surely "tarpit" is a more natural name?
I don’t know if ‘stabbles didn’t pick it for this reason, but `tarpit` is the low level tar inspection CLI that tar-codec provides.
I’ve been working on an open source tool for creating SBOMs from binaries/file systems. Automated extraction of archives like zip/tar files present interesting challenges - for a while I’ve known about OS filesystem formatting gotchas like archives from a Linux-based system with symlinks or case colliding names not extracting correctly on NTFS (and HFS+ for the latter case).
But the ambiguity of this tar files… I guess the “correct” extraction for what I’m doing would require figuring out what the system that is processing the tar files used and copy that behavior, which is non-trivial if you don’t know what an embedded device is using behind the scenes.
Astral blogged about their new tar parser for uv recently, see https://astral.sh/blog/tar-codec. I've included it in my work-in-progress test suite https://stoppels.ch/tar/ which shows it's by far the strictest tar library out there.
For your use case it would make sense to pick the strictest tar parser, such that you can error or warn on most ambiguities. In practice these ambiguities are rare.
A strict parser makes sense for flagging that there are issues, but that also can’t block extraction; it needs to be configurable enough that users can make a decision on how to handle the ambiguity.
There are many scenarios (such as fetching package distributions over the internet) where the user lacks sufficient information to make an informed and secure decision about how to safely parse and extract a malformed tar (or any other archive format).
(Another way of saying that is that we live in a post-Postel world[1].)
[1]: https://alexgaynor.net/2025/mar/25/postels-law-and-the-three...
That’s true, but also much more general than our SBOM generator is targeting. We’re typically looking at filesystems pulled from embedded systems, or their update files. Ultimately we want to accurately represent what the state of the file system in the device looks like, which can require reverse engineering the update process to determine how to handle ambiguous cases (“correct” is matching what the device firmware update does).
Disregarding erroring as an option-
I feel like the most faithful extraction would dictate that any field in the original header should take precedence over the PaxHeader's iff the PaxHeader field's value would've been representable in the original field. Anyone have thoughts?
pax says the opposite: the extended headers always take precedence, even when they produce invalid states (such as deleting the `path` state).
pax does not prescribe what to do in these kinds of invalid states, so there’s a long tail of people making educated guesses.
Right, it just feels wrong to me for pax to mandate that. My argument is that the whole point of the extension is to handle the cases that the original can't handle while still preserving existing behavior for what is already supported - hence making it an extension. If you're fully overriding the original then you just have a different format altogether and aren't respecting the original at all, and therefore shouldn't be allowing decoders for the original format to read the new one.
I think this is true in a sense. But the other major benefit to pax is that it normalizes metadata handling away from the mess that is fixed octal fields in a V7 tar header. pax is its own mess, but at least it’s extremely clear about how spaces, NULs, etc. interact with values.
(On a more basic level, I can imagine it would be even more differential prone for the precedence rule to be “pax, but not always.” pax still makes it complicated but that would be another whole circle of hell.)
Tl;dr don't use pax.
Hit up Wikipedia for tarball format and you may agree with me. It's a classic example of a standard where you look at it and go, "??? Who asked for that?". I'm not surprised the article focuses on pax when finding tar inconsistencies. Original tar is just fine.
I don’t really think pax is the issue here, although pax is a bad specification in absolute terms. But “classic” (V7) tar just isn’t specified at all, and has plenty of its own differentials.
(Users also want things like filenames that are longer than 100 characters, for obvious reasons.)
Pax has an interesting use in POSIX: combined with od, pax is the only way (in POSIX) to get a Unix timestamp from the command line. The exact command line is left as an exercise for the interested reader.
As of POSIX.1-2024 you can use %s to get the number of seconds since the epoch. But that was indeed not specified in earlier POSIX versions, AFAIK.