"The choice of Haskell has also led to a high quality and low volume of contributors"
I feel this influence of choosing a tech stack and its impact on self selected and auto-reenforced culture is most often underestimated.
From my own experience, at a time I was (involuntarily) working in Java, and when .Net was released, from a pure technical point of view it was like a breath of fresh air. Java was suffering from overengineering, archtecture astronauts galore and no sensible UX framework. .Net, the new kid, came in lean and clean with a UX library that 'just worked'.
Problem later was that for all its flaws and being overly 'academic', in teams (the real thing, not the awfull app), you could have indepth discussions about non trivial aspects of SWE topics in the Java world, whereas for all its technical prowess, in .Net land you were mostly dwelling amongst the 2 week CRUD app bootcamp folks. This ofc is a gross oversimplication.
You had brilliant engineers and challanged codemonkeys on both sides. But the skew was more than a little biased.
Maybe try F#. Those "2 week CRUD bootcamp" people all write C# but the people writing F# are more likely to have some actual engineering because it's an ML but just .NET flavoured.
"few applicants, most pretty good" was true for us as well when hiring Haskell developers at Freckle (https://tech.freckle.com/) - in some ways it was scary: we're away from the herd, the numbers aren't on our side, what if we ever need to hire large volumes of people?
Instead of getting 200 JavaScript devs in the pipeline, you would get maybe 10-20 Haskellers. What if the person you're looking for is not in that pool? Intuitively, you assume that more is always better.
In other ways it made filtering through candidates quite simple: half of the people who applied were really good, the other either language astronauts or folks fascinated with the tooling who didn't genuinely want to move fast to produce commercially-viable software, they wanted to tinker. You just needed to figure out which bucket the person was in.
Most systems are not large enough to need 200 developers. Even when they are, you hire those 200 over time. If you have 200 developers (or 200 anything) you need to have a good training and mentoring program.
Language is not the problem. If someone is a senior developer (not just has the title because of years of service), you can teach them Haskell on the job for little cost. Sure it will take them a few years to be an expert in the language, but most problems they need to solve don't need language experts, just someone good enough. And Haskell is a different language, most often you are hiring for a language that is only slightly different from ones they already know.
> by writing N parsers (“readers”) and M renderers (“writers”), one could support N × M conversions.
Beautiful writeup for a wonderful project. In an age of vibe-coding hype it's also so nice to see how things can be extended and snowball in usefulness when things are built correctly, by hand, from basic principles.
> Perhaps, then, in the future, people will no longer have a need for tools like pandoc.
I think we will need wonderful things like pandoc more and more. As mentioned there is a huge ecological and practical difference. Even if LLMs could get infintisamally close to deterministic-level reliability, it's still so many more orders of magnitude better in efficiency, especially with big batch jobs etc.
I think the art is about figuring out and defining what the underlying technical principles you want to leverage in order to provide a relevant software solution are. LLMs used naively might do that, but they probably won't. The work is still in figuring out what a useful domain representation of the problem is - and then you can unleash the llms for the boilerplate.
His interests are in early analytic philosophy, philosophy of language and formal logic. This niche is what created the foundations for both theoretical computer science and formal programming languages. This is kind of a reason why quite a few of big names in tech are from a philosophy background - Paul Graham, Peter Thiel, Reid Hoffman, etc.
Oh hey, I'm sure his Stanford philosophy teachers are very impressed by how Thiel took their teaching to hearth and these days offer compelling arguments for why Greta Thunberg is a servant of the Antichrist. /s
I do this too, but I use cmake to switch the backend between pandoc and lowdown [0]. Lowdown is not a drop-in replacement for pandoc flavored markdown, but it's between 10-100X faster.
It's very handy to switch when tinkering with template changes because I can convert ~ 1000 markdown files instantaneously on my mid 2010s desktop.
which strips out styling, wrapper divs, spans, inline attributes, etc from (for instance) HTML copied from a google or word doc. Just the semantic goodness!
Pandoc is awesome. My fav usage is configuring git to use it to normalize binary docs to markdown (like a .docx) so they can be diffed. Works amazing for redlining contacts.
I do this too. If you use e.g. GitHub you can build this into your PR checks so that any PRs containing docx file changes get a CI bot comment with the diff from the docx.
To top it all off Pandoc has a great experience for contributors. Over the past few years I've opened several bug reports related to Typst and docx and all of them got responses that were kind and helpful. I even had a few PRs merged in despite knowing almost nothing of Haskell.
Pandoc is a fantastic piece of software. I have never had any issues with it, and it's the tool I reach for anytime I need to covert documents. I'm super grateful to John MacFarlane for creating it and maintaining it for all these years!
Thank you for pandoc. I made the (at the time perhaps not transparently wise) choice to go all in on it when I started my PhD, and that decision has paid nothing but dividends since. I owe my career as a scientist to it.
How do you usually collaborate with other authors? I don't know your area, but it's common to use Overleaf to collaborate on papers, and you're stuck with LaTeX there. I wrote the first draft of my thesis in Typst, but then converted to LaTeX to get feedback on Overleaf from other people.
Yes, in my field (astronomy), Overleaf is also ubiquitous, especially for observational papers with long author lists. However, by the same token, this means that most of my collaborators are distant ones, and from experience many are happy (and occasionally prefer) to conduct business by way of emailed PDFs. My close collaborators have been happy to co-edit a Markdown file by way of GitHub pull requests or directly committing to a shared repository.
I have, for a while, been toying with the idea of maintaining a fork of (or plugin for?) Overleaf CE that runs pandoc-powered build scripts instead of using latexmk directly. Perhaps this might be something vibe-codeable.
I remember noting "djot" when John M. created it but hadn't really looked into it before now. After reading through the design goals and differences from Markdown, I can see we'd have been better off if somehow djot had been able to come first. Gruber has said he thinks Markdown being underspecified was a feature, but as a user trying to mix and match different tools it drove me nuts.
A few years ago, I gobbled together some bash scripts around pandoc to build a site generator for my personal website. Works great. I use html templates, markdown for the content, etc. Mostly the bash scripts just serve to list files and process them one by one. I actually process them concurrently by forking processes so it's reasonably fast. A bit wonky but it works fine for my use case.
As for Haskell, I guess tree transformations and parsing are the perfect use case for functional programming. I studied in Utrecht in the nineties when Erik Meijer was still teaching there (later went to work at Microsoft Research where he contributed to things like F# and Linq). In short, my compiler course was taught using functional programming. We were toying around with writing our own parser generators to implement a subset of Modula 3 or our own toy languages. Lots of monads and other esoteric abstractions.
I haven't really done much professionally with any of that since except having a really easy time when languages like Kotlin, Javascript, etc. started borrowing liberally from functional programming. These days, if you have a list, calling map or forEach on it with another function is perfectly normal in many languages. Very nice alternative to a for or while loop.
I can imagine that it works great, but at the same time I have also used plain Rust to generate HTML. It’s not that hard to generate valid HTML. Parsing however is a different thing. That’s hard. For Markdown source, there are libraries to parse the Markdown. That’s a bit more work to get right, but at the same time Pandoc also requires some tweaks and configuration to get right, plus getting the dependency installed in your environment(s). I personally would go for the Rust libraries way for a new project for flexibility sake but wouldn’t say the other way is wrong per se.
Thank you for Pandoc! I used it for many things over the years, and it was always there as a good option. I mostly use it for Markdown->Typst these days and it does this job very well.
Pandoc is amazing. I have HTML and want a PDF? One command and it just works painlessly. That was just my last use of pandoc this weekend. I do not use it that often, but when I do its perfect.
I've always taken particular note of how wisely scope-limited Pandoc is. Markup that aligns with natural language convention[1], is tightly converted, but the further from natural language, the less fidelity Pandoc can promise. Until, at the DITA or S1000D stage of "this ain't natlang, brah", Pandoc says "forget it" and just won't even pretend that such markup is even convertible.
Because, spoiler, it's not.
Constructs like tables and bibliographies challenge natural language markup - resulting in an explosion of different formalisms - but component content system artifacts for transclusion and conditionals shatter any pretense that these file types are "documents" at all. Both of those artifacts must draw formal structure from outside of language, i.e., from their own product / domain. They're parts of a system that make documents, but are not themselves documents or natural language. They are meaningless - or, worse, full of wrong meaning - outside of their runtime environment inside an explicit knowledge domain. Something that newer component content formats like Typst recognize explicitly.
The proof of all this is, as they say, in the pudding. What do people write documents in today? Well, they stick to natural language formats, sometimes they let the document system handle tables in some bespoke way, but conditionals are viewed with justified suspicion. DITA and S1000D projects, and the cursed migrations that lead to them, are sparse and driven almost exclusively by regulatory requirements, or, more often, program offices misreading regulatory requirements[0].
And here we all are in the LLM age, where natural language is being vindicated in ways both awe-inspiring and devastating. While component content systems force an LLM to expand its context window to the entire repository to make sense of any single sentence.
The crap of all this is, this is stuff that computer / information science has known since at least the 1980s. There are papers written about it. But high-complexity component content systems are sellable to non-technical writer groups because they don't see the tripwires in the fundamentals, or they think[2] that their product domain is so structured that the tripwires can be rigged as structure.
[0] No, converting to a pile of S1000D 040As doesn't magically fix your MTAs or your ILS or anything else.
[1] I do realize that proximity to natural language is correlate, not cause. Markdown converts well as a low-power notation whose instances denote values; it reads like natural language because that's what low-power does. The operative variable is whether the artifact denotes a document or a function from configuration to documents. AsciiDoc with `ifdef::[]` and `include::[]` converts every bit as badly as DITA, although without the fundamental nonsense of XSD and Horn's Information Mapping.
"The choice of Haskell has also led to a high quality and low volume of contributors"
I feel this influence of choosing a tech stack and its impact on self selected and auto-reenforced culture is most often underestimated.
From my own experience, at a time I was (involuntarily) working in Java, and when .Net was released, from a pure technical point of view it was like a breath of fresh air. Java was suffering from overengineering, archtecture astronauts galore and no sensible UX framework. .Net, the new kid, came in lean and clean with a UX library that 'just worked'.
Problem later was that for all its flaws and being overly 'academic', in teams (the real thing, not the awfull app), you could have indepth discussions about non trivial aspects of SWE topics in the Java world, whereas for all its technical prowess, in .Net land you were mostly dwelling amongst the 2 week CRUD app bootcamp folks. This ofc is a gross oversimplication. You had brilliant engineers and challanged codemonkeys on both sides. But the skew was more than a little biased.
Maybe try F#. Those "2 week CRUD bootcamp" people all write C# but the people writing F# are more likely to have some actual engineering because it's an ML but just .NET flavoured.
> "The choice of Haskell has also led to a high quality and low volume of contributors"
I find this mindset very refreshing in the era of availability often implying noise over signal.
"few applicants, most pretty good" was true for us as well when hiring Haskell developers at Freckle (https://tech.freckle.com/) - in some ways it was scary: we're away from the herd, the numbers aren't on our side, what if we ever need to hire large volumes of people? Instead of getting 200 JavaScript devs in the pipeline, you would get maybe 10-20 Haskellers. What if the person you're looking for is not in that pool? Intuitively, you assume that more is always better.
In other ways it made filtering through candidates quite simple: half of the people who applied were really good, the other either language astronauts or folks fascinated with the tooling who didn't genuinely want to move fast to produce commercially-viable software, they wanted to tinker. You just needed to figure out which bucket the person was in.
Most systems are not large enough to need 200 developers. Even when they are, you hire those 200 over time. If you have 200 developers (or 200 anything) you need to have a good training and mentoring program.
Language is not the problem. If someone is a senior developer (not just has the title because of years of service), you can teach them Haskell on the job for little cost. Sure it will take them a few years to be an expert in the language, but most problems they need to solve don't need language experts, just someone good enough. And Haskell is a different language, most often you are hiring for a language that is only slightly different from ones they already know.
> by writing N parsers (“readers”) and M renderers (“writers”), one could support N × M conversions.
Beautiful writeup for a wonderful project. In an age of vibe-coding hype it's also so nice to see how things can be extended and snowball in usefulness when things are built correctly, by hand, from basic principles.
> Perhaps, then, in the future, people will no longer have a need for tools like pandoc.
I think we will need wonderful things like pandoc more and more. As mentioned there is a huge ecological and practical difference. Even if LLMs could get infintisamally close to deterministic-level reliability, it's still so many more orders of magnitude better in efficiency, especially with big batch jobs etc.
Well the difficult part is finding a good intermediate representation, which they did
You can build things this simple and elegant with LLM’s you just need to instruct them to do so.
I think the art is about figuring out and defining what the underlying technical principles you want to leverage in order to provide a relevant software solution are. LLMs used naively might do that, but they probably won't. The work is still in figuring out what a useful domain representation of the problem is - and then you can unleash the llms for the boilerplate.
That a professor of philosophy [1] made tools [2] that are used by millions around the world... that's just mind boggling. In a good (great!) way.
Pandoc is my go-to tool. Thank you, Sir!
[1] https://johnmacfarlane.net/index.html
[2] https://johnmacfarlane.net/tools.html
His interests are in early analytic philosophy, philosophy of language and formal logic. This niche is what created the foundations for both theoretical computer science and formal programming languages. This is kind of a reason why quite a few of big names in tech are from a philosophy background - Paul Graham, Peter Thiel, Reid Hoffman, etc.
Oh hey, I'm sure his Stanford philosophy teachers are very impressed by how Thiel took their teaching to hearth and these days offer compelling arguments for why Greta Thunberg is a servant of the Antichrist. /s
Pandoc is great. I use it multiple times per day to move content between Outlook emails and a coding harness:
https://gist.github.com/rahimnathwani/210b1f9cb6ce731a304322...
Love pandoc, I use it for all sorts of stuff…here is my minimal static site generator:
find . -name '*.md' -type f -exec sh -c '
' sh {} +
I do this too, but I use cmake to switch the backend between pandoc and lowdown [0]. Lowdown is not a drop-in replacement for pandoc flavored markdown, but it's between 10-100X faster.
It's very handy to switch when tinkering with template changes because I can convert ~ 1000 markdown files instantaneously on my mid 2010s desktop.
[0]: https://kristaps.bsd.lv/lowdown/
My most used pandoc snippet:
tidyhtml () { pandoc -f html-native_divs-native_spans -t markdown-raw_html-raw_attribute | pandoc -f markdown -t html }
which strips out styling, wrapper divs, spans, inline attributes, etc from (for instance) HTML copied from a google or word doc. Just the semantic goodness!
Pandoc is awesome. My fav usage is configuring git to use it to normalize binary docs to markdown (like a .docx) so they can be diffed. Works amazing for redlining contacts.
Wow. Thanks. This is an awesome idea. Will blatantly steal. Thanks for sharing.
I do this too. If you use e.g. GitHub you can build this into your PR checks so that any PRs containing docx file changes get a CI bot comment with the diff from the docx.
To top it all off Pandoc has a great experience for contributors. Over the past few years I've opened several bug reports related to Typst and docx and all of them got responses that were kind and helpful. I even had a few PRs merged in despite knowing almost nothing of Haskell.
Thanks so much for pandoc!
I never heard about djot [0], is anyone using it?
[0]: https://djot.net/
Djot is great, I wrote a web server that uses it natively.[0]
[0] https://github.com/gn0/nvim-web-server
Pandoc is a fantastic piece of software. I have never had any issues with it, and it's the tool I reach for anytime I need to covert documents. I'm super grateful to John MacFarlane for creating it and maintaining it for all these years!
Twenty years of quiet, consistent maintenance. No hype, no VC money, just solid work. Rare and precious
Thank you for pandoc. I made the (at the time perhaps not transparently wise) choice to go all in on it when I started my PhD, and that decision has paid nothing but dividends since. I owe my career as a scientist to it.
How do you usually collaborate with other authors? I don't know your area, but it's common to use Overleaf to collaborate on papers, and you're stuck with LaTeX there. I wrote the first draft of my thesis in Typst, but then converted to LaTeX to get feedback on Overleaf from other people.
Yes, in my field (astronomy), Overleaf is also ubiquitous, especially for observational papers with long author lists. However, by the same token, this means that most of my collaborators are distant ones, and from experience many are happy (and occasionally prefer) to conduct business by way of emailed PDFs. My close collaborators have been happy to co-edit a Markdown file by way of GitHub pull requests or directly committing to a shared repository.
I have, for a while, been toying with the idea of maintaining a fork of (or plugin for?) Overleaf CE that runs pandoc-powered build scripts instead of using latexmk directly. Perhaps this might be something vibe-codeable.
I don't need Pandoc often, but when I do there's nothing like it.
One thing I like with pandoc is that it produces clean html or latex code (it can also produce clean markdown with the right option).
If you compare this with the HTML produced by typst or hevea, this is super useful, as we can then roll out our own styles.
I remember noting "djot" when John M. created it but hadn't really looked into it before now. After reading through the design goals and differences from Markdown, I can see we'd have been better off if somehow djot had been able to come first. Gruber has said he thinks Markdown being underspecified was a feature, but as a user trying to mix and match different tools it drove me nuts.
I have at times mused that writing the internal state of pandoc to disk (yes 13th standard etc) would be the most interoperable file format
Since your document is a haskell value, syntax highlighting gives added value to the terminal drug trip
I don’t think it has backward compatibility / stability guarantees though, does it?
A few years ago, I gobbled together some bash scripts around pandoc to build a site generator for my personal website. Works great. I use html templates, markdown for the content, etc. Mostly the bash scripts just serve to list files and process them one by one. I actually process them concurrently by forking processes so it's reasonably fast. A bit wonky but it works fine for my use case.
As for Haskell, I guess tree transformations and parsing are the perfect use case for functional programming. I studied in Utrecht in the nineties when Erik Meijer was still teaching there (later went to work at Microsoft Research where he contributed to things like F# and Linq). In short, my compiler course was taught using functional programming. We were toying around with writing our own parser generators to implement a subset of Modula 3 or our own toy languages. Lots of monads and other esoteric abstractions.
I haven't really done much professionally with any of that since except having a really easy time when languages like Kotlin, Javascript, etc. started borrowing liberally from functional programming. These days, if you have a list, calling map or forEach on it with another function is perfectly normal in many languages. Very nice alternative to a for or while loop.
I can imagine that it works great, but at the same time I have also used plain Rust to generate HTML. It’s not that hard to generate valid HTML. Parsing however is a different thing. That’s hard. For Markdown source, there are libraries to parse the Markdown. That’s a bit more work to get right, but at the same time Pandoc also requires some tweaks and configuration to get right, plus getting the dependency installed in your environment(s). I personally would go for the Rust libraries way for a new project for flexibility sake but wouldn’t say the other way is wrong per se.
Thank you for Pandoc! I used it for many things over the years, and it was always there as a good option. I mostly use it for Markdown->Typst these days and it does this job very well.
Thanks also that new reader/writer are also integrated like typst.
I use it to create pdfs for my blog posts with that code :-)
Pandoc is amazing. I have HTML and want a PDF? One command and it just works painlessly. That was just my last use of pandoc this weekend. I do not use it that often, but when I do its perfect.
TIL pandoc for the people, courtesy of pandoc compiled to WASM. cool
https://pandoc.org/app/
It's not easy to maintain so many parsers and renderers, and at the same time to keep the qualities of them.
TIL of this wonderful tool, how did I never used this before, boggles the mind
unending thanks, as much for pandoc as for the example of a clear design+implementation that lasts.
pandoc was (and still is) critical to some of my flows. I hope it never dies!
I can only imagine the hell it must be when the AST is changed to update all inbound and outbound code.
Effort worth admiration. Thank you for creating pandoc! It allowed me to write my CV in more readable markdown, but also get nice pdf with latex.
Thank you for pandoc!
Happy birthday and thank you so much!
I've always taken particular note of how wisely scope-limited Pandoc is. Markup that aligns with natural language convention[1], is tightly converted, but the further from natural language, the less fidelity Pandoc can promise. Until, at the DITA or S1000D stage of "this ain't natlang, brah", Pandoc says "forget it" and just won't even pretend that such markup is even convertible.
Because, spoiler, it's not.
Constructs like tables and bibliographies challenge natural language markup - resulting in an explosion of different formalisms - but component content system artifacts for transclusion and conditionals shatter any pretense that these file types are "documents" at all. Both of those artifacts must draw formal structure from outside of language, i.e., from their own product / domain. They're parts of a system that make documents, but are not themselves documents or natural language. They are meaningless - or, worse, full of wrong meaning - outside of their runtime environment inside an explicit knowledge domain. Something that newer component content formats like Typst recognize explicitly.
The proof of all this is, as they say, in the pudding. What do people write documents in today? Well, they stick to natural language formats, sometimes they let the document system handle tables in some bespoke way, but conditionals are viewed with justified suspicion. DITA and S1000D projects, and the cursed migrations that lead to them, are sparse and driven almost exclusively by regulatory requirements, or, more often, program offices misreading regulatory requirements[0].
And here we all are in the LLM age, where natural language is being vindicated in ways both awe-inspiring and devastating. While component content systems force an LLM to expand its context window to the entire repository to make sense of any single sentence.
The crap of all this is, this is stuff that computer / information science has known since at least the 1980s. There are papers written about it. But high-complexity component content systems are sellable to non-technical writer groups because they don't see the tripwires in the fundamentals, or they think[2] that their product domain is so structured that the tripwires can be rigged as structure.
[0] No, converting to a pile of S1000D 040As doesn't magically fix your MTAs or your ILS or anything else.
[1] I do realize that proximity to natural language is correlate, not cause. Markdown converts well as a low-power notation whose instances denote values; it reads like natural language because that's what low-power does. The operative variable is whether the artifact denotes a document or a function from configuration to documents. AsciiDoc with `ifdef::[]` and `include::[]` converts every bit as badly as DITA, although without the fundamental nonsense of XSD and Horn's Information Mapping.
[2] Almost always wrongly