Hi,
Original propolis architect here, and primary engineer for much of the effort porting bhyve to illumos.
Quite a bit was changed between "illumos bhyve" and FreeBSD upstream. We in illumos decided to take a slightly different approach in how we approached the machinery for saving/restoring the state of devices emulated in-kernel, which is a necessary part of doing such a live migration. FreeBSD was first to add that functionality, but we (I) felt it exposed too much in the way of implementation details from that device emulation, rather than a more abstract model of the emulated hardware. It was at that point that illumos "soft-forked" the kernel VMM component so we could take a different approach. While the two kernel VMM components are very similar, and much of the bhyve userspace emulation (a counterpart to propolis) will run on either OS, there are some key differences around instruction emulation and those kernel-emulated devices which prevent them from being drop-in replacements for each other.
All that said, I'm happy to see FreeBSD making progress on this front. They've been a friendly peer in this space, and both projects benefit from efforts in this area.
Not really familiar with bhyve or even illumos in the slightest. But reading this comment makes me a bit sad in a way. These types of conversations and comments used to be common where I worked just few years ago. Now no one even cares how something is implemented let alone a "I felt it exposed too much in the way of implementation details from that device emulation, rather than a more abstract model of the emulated hardware". The "good" ones today are looking exclusively at external properties rather than internal ones. Thank you for the comment.
I feel you. Good programmers still care about those things, but it turns out there are a whole lot of people who never cared about the craft and were just in the profession because it was lucrative. It's a shame.
> Now no one even cares how something is implemented ...
> The "good" ones today are looking exclusively at external properties rather than internal ones.
To provide a little additional context here: This effectively was an external property of the system (just to the userspace VMM). One of the significant challenges for those implementing a feature live migration is how you'll deal with multiple version of the software (older and newer) interacting in the future. A primary use case for live migration is moving workloads around to permit software updates, making that multiple version issue even "worse".
Abstracting the implementation details of those in-kernel emulated devices was helpful in two ways. Firstly, it reduced (hopefully) the chances that a later change to the internal implementation would be visible in the exported/imported state. Secondly, that the act itself of doing that state abstraction required a much more intentional and thorough examination of the emulation logic itself, resulting in a cleaner and simpler artifact.
So while it may seem like it was driven by a desire to take pride in the craft (undeniably _a_ factor), the main motivation was avoiding a versioning mess in the future.
I love FreeBSD. But I reluctantly had to migrate one of my home servers to Ubuntu Server because I needed Docker support as well as ZFS. Ubuntu Server, compared to FreeBSD, is a complete clusterfuck of an OS. And I regret the effort I spent migrating.
I used to use Ubuntu a lot about 7 years ago. I saw the writing on the wall. Once they started forcing snap down people’s throats, especially. I’ve been on Debian ever since and have been very happy. I also have run many other OS’s and I feel that Debian is still a good blend of polished and still friendly to the hacker.
Yeah Debian is nice. But at the time of switching, only Ubuntu offered ZFS precompiled in their official repos.
I did, for a period, run some other distro with DKMS but that was a massive pain to keep up-to-date due to ZoL’s piss poor documentation and messy build tools.
This was quite a few years ago though. Long before even FreeBSD synced with ZoL for OpenZFS. So the problems I encountered have likely matured tenfold since.
But if I’m going to the hassle of installing a new OS then I might as well install FreeBSD again because I just prefer it to Linux (nothing against Linux in general, my laptops and PCs all run Arch and I’ve made a career from supporting a variety Linux platforms)
I have FreeBSD directly on metal, with just ZFS datastore and vm-bhyve [1] directly on it. All my services are in separate bhyve VMs. Most of those are also FreeBSD-based but some that require docker are Linux-based (my radio [2] is azuracast [3] in docker inside Ubuntu as a bhyve vm on FreeBSD). This way I can setup zrepl [4] on host to periodically snapshot whole VMs and send them to another box or to external USB disk, and I'm backed up.
If you need Linux compatibility layer AND ZFS without a hypervisor, unfortunately it's hard to get close to a Linux host. I've never used FreeBSD or SmartOS in production (I did use Solaris and zones in the past) and I know ZFS support is first class in those. But I've been using both zfs-dkms and kmod-zfs in RockyLinux and the experience has been pretty pleasant (ended up chosing kmod because we only follow OS releases anyway and install time is better there). I'm sure the story is similar for OSes like AlmaLinux (which also get upgrades sooner than Rocky, and I would prefer to use it instead of Rocky, but it is a requirement for us).
On my desktop I also use ZFS with kmod on cachy and updates have been seamless so far.
I did consider the bhyve approach myself, before I made the switch. But it was still pretty experimental at the time (iirc) and wanted something reliable otherwise I’d have a wife and two kids moaning at me that their stuff isn’t working haha!
But maybe you’re right that it’s worth reconsidering now.
There's experimental support for Podman and OCI containers on FreeBSD 14.3 and later, with some additional system configuration required [0], [1]. I've been playing with it off and on with FreeBSD 15.0 and 15.1, but I'm not using it for anything I depend on.
Out of curiosity, what required docker that wasn't better suited by jails? I know such things exist (e.g. software projects that are only distributed as dockerfiles), but not sure how common they are.
Laziness. I ran Jails for years and was largely happy with it. But I didn’t want to handle the jails upgrades manually. I just wanted to run “docker pull”.
Looks like this could really become a game changer for the FreeBSD folks using bhyve. The author often complained about this missing feature and he was also contributing to the Sylve project. Looks like he's crafting something.
I am using FreeBSD, not DragonflyBSD, but I happen to be on a DragonflyBSD developer mailing list, which is normally low traffic, except that I have noticed recently a lot of messages with various patches, mentioning both bug fixes and newly implemented features.
So the project is alive and it seems more active than previously.
There is some uptick in interest in various BSDs as a linux alternative given the rough few months it's been having. I know 2 companies that are trying to see if they can switch some edge nginx proxies to run on OpenBSD. A RCE in a TCP handshake is up there tbf
Interesting to hear / read this. I know OpenBSD has made a lot of improvement in the past 2 - 3 years on performance. But I would have thought for heavy traffic it is still not anywhere near FreeBSD or Linux.
But I guess we are at the point security is important and hardware ( CPUs ) is relatively cheap?
It was a brief conversation in a social gathering for smallish engineer-led startups in my area. The one guy who went into details mentioned they are evaluating OpenBSD because of its presumed focus on security. I doubt they are super high throughput, but they might land on something else too. They run their own hardware at least, so the hardware compat issues of OpenBSD didn't seem to be a problem for them. FreeBSD was mentioned as well fwiw. I don't know how far along they got in performance testing or actual switch.
Just general angst some people were expressing for publicly exposed resources. The common wisdom just a few years ago was that throwing an internet facing nginx or haproxy on an up-to-date LTS linux with some firewall debouncer is as rock solid as you could get. And instead, you should worry about security in your own upstream app, not Linux's TCP sockets or nginx's regex path matching.
This brings back memories of watching someone do a live demo of bhyve at nycbug. I think that just have been almost a decade ago now, but the live demo was amazing.
Even Solaris in Kernel Zones uses virtualization for the kernel part to have 'live migration' - regular Solaris Zones Containers does not have 'live migration' - you can just start/stop them.
But ... OpenVZ in the past had 'live migration' - so it would be nice to implement that feature into FreeBSD Jails to have 'high availability'.
Also related is Propolis from Oxide who worked on live migration with Bhyve https://github.com/oxidecomputer/propolis
Hi, Original propolis architect here, and primary engineer for much of the effort porting bhyve to illumos.
Quite a bit was changed between "illumos bhyve" and FreeBSD upstream. We in illumos decided to take a slightly different approach in how we approached the machinery for saving/restoring the state of devices emulated in-kernel, which is a necessary part of doing such a live migration. FreeBSD was first to add that functionality, but we (I) felt it exposed too much in the way of implementation details from that device emulation, rather than a more abstract model of the emulated hardware. It was at that point that illumos "soft-forked" the kernel VMM component so we could take a different approach. While the two kernel VMM components are very similar, and much of the bhyve userspace emulation (a counterpart to propolis) will run on either OS, there are some key differences around instruction emulation and those kernel-emulated devices which prevent them from being drop-in replacements for each other.
All that said, I'm happy to see FreeBSD making progress on this front. They've been a friendly peer in this space, and both projects benefit from efforts in this area.
Not really familiar with bhyve or even illumos in the slightest. But reading this comment makes me a bit sad in a way. These types of conversations and comments used to be common where I worked just few years ago. Now no one even cares how something is implemented let alone a "I felt it exposed too much in the way of implementation details from that device emulation, rather than a more abstract model of the emulated hardware". The "good" ones today are looking exclusively at external properties rather than internal ones. Thank you for the comment.
I feel you. Good programmers still care about those things, but it turns out there are a whole lot of people who never cared about the craft and were just in the profession because it was lucrative. It's a shame.
> Now no one even cares how something is implemented ...
> The "good" ones today are looking exclusively at external properties rather than internal ones.
To provide a little additional context here: This effectively was an external property of the system (just to the userspace VMM). One of the significant challenges for those implementing a feature live migration is how you'll deal with multiple version of the software (older and newer) interacting in the future. A primary use case for live migration is moving workloads around to permit software updates, making that multiple version issue even "worse".
Abstracting the implementation details of those in-kernel emulated devices was helpful in two ways. Firstly, it reduced (hopefully) the chances that a later change to the internal implementation would be visible in the exported/imported state. Secondly, that the act itself of doing that state abstraction required a much more intentional and thorough examination of the emulation logic itself, resulting in a cleaner and simpler artifact.
So while it may seem like it was driven by a desire to take pride in the craft (undeniably _a_ factor), the main motivation was avoiding a versioning mess in the future.
> Thank you for the comment.
My pleasure!
By your storing of a more abstract representation of the emulated device state, could you potentially live migrate to tape or object storage?
What’s FreeBSDs story like for Docker these days?
I love FreeBSD. But I reluctantly had to migrate one of my home servers to Ubuntu Server because I needed Docker support as well as ZFS. Ubuntu Server, compared to FreeBSD, is a complete clusterfuck of an OS. And I regret the effort I spent migrating.
I used to use Ubuntu a lot about 7 years ago. I saw the writing on the wall. Once they started forcing snap down people’s throats, especially. I’ve been on Debian ever since and have been very happy. I also have run many other OS’s and I feel that Debian is still a good blend of polished and still friendly to the hacker.
Yeah Debian is nice. But at the time of switching, only Ubuntu offered ZFS precompiled in their official repos.
I did, for a period, run some other distro with DKMS but that was a massive pain to keep up-to-date due to ZoL’s piss poor documentation and messy build tools.
This was quite a few years ago though. Long before even FreeBSD synced with ZoL for OpenZFS. So the problems I encountered have likely matured tenfold since.
But if I’m going to the hassle of installing a new OS then I might as well install FreeBSD again because I just prefer it to Linux (nothing against Linux in general, my laptops and PCs all run Arch and I’ve made a career from supporting a variety Linux platforms)
I have FreeBSD directly on metal, with just ZFS datastore and vm-bhyve [1] directly on it. All my services are in separate bhyve VMs. Most of those are also FreeBSD-based but some that require docker are Linux-based (my radio [2] is azuracast [3] in docker inside Ubuntu as a bhyve vm on FreeBSD). This way I can setup zrepl [4] on host to periodically snapshot whole VMs and send them to another box or to external USB disk, and I'm backed up.
[1] https://github.com/freebsd/vm-bhyve
[2] https://www.dno.rs/radio
[3] https://github.com/AzuraCast/AzuraCast
[4] https://github.com/zrepl/zrepl
Nice. I’ll have a read of those links. Thank you
If you need Linux compatibility layer AND ZFS without a hypervisor, unfortunately it's hard to get close to a Linux host. I've never used FreeBSD or SmartOS in production (I did use Solaris and zones in the past) and I know ZFS support is first class in those. But I've been using both zfs-dkms and kmod-zfs in RockyLinux and the experience has been pretty pleasant (ended up chosing kmod because we only follow OS releases anyway and install time is better there). I'm sure the story is similar for OSes like AlmaLinux (which also get upgrades sooner than Rocky, and I would prefer to use it instead of Rocky, but it is a requirement for us).
On my desktop I also use ZFS with kmod on cachy and updates have been seamless so far.
was that like 5 years ago or more ? Debian "just have packages" for a while
Now I think about it, it might have been 5 jobs ago!! So possibly 10 years.
Wow it’s scary how quickly time goes by!
> What’s FreeBSDs story like for Docker these days?
My current setup for Docker (Podman really in my case):
My plan:
So basically just ditching Proxmox for FreeBSD/bhyve. Then at long last I'll be, again, systemd-free.
I did consider the bhyve approach myself, before I made the switch. But it was still pretty experimental at the time (iirc) and wanted something reliable otherwise I’d have a wife and two kids moaning at me that their stuff isn’t working haha!
But maybe you’re right that it’s worth reconsidering now.
There's experimental support for Podman and OCI containers on FreeBSD 14.3 and later, with some additional system configuration required [0], [1]. I've been playing with it off and on with FreeBSD 15.0 and 15.1, but I'm not using it for anything I depend on.
[0]: https://docs.freebsd.org/en/books/handbook/containers/
[1]: https://podman.io/docs/installation#installing-on-freebsd
Nice. I’ll have a read of that later. Thank you
What a shame.
There is also this which was not mentioned by other commenters.
https://daemonless.io/
Looks interesting. Thanks for sharing.
Out of curiosity, what required docker that wasn't better suited by jails? I know such things exist (e.g. software projects that are only distributed as dockerfiles), but not sure how common they are.
Laziness. I ran Jails for years and was largely happy with it. But I didn’t want to handle the jails upgrades manually. I just wanted to run “docker pull”.
We run Linux in a bhyve VM for podman support but backed by FreeBSD ZFS. Migrate back!
Use Debian. Ubuntu is a clusterfuck but it's not all related to the software they use
Debian is in a downward spiral after Buster.
Why’s that?
Looks like this could really become a game changer for the FreeBSD folks using bhyve. The author often complained about this missing feature and he was also contributing to the Sylve project. Looks like he's crafting something.
Anyone heard anything from dragonflyBSD lately? This sounds like something that might go well with HammerFS?
I am using FreeBSD, not DragonflyBSD, but I happen to be on a DragonflyBSD developer mailing list, which is normally low traffic, except that I have noticed recently a lot of messages with various patches, mentioning both bug fixes and newly implemented features.
So the project is alive and it seems more active than previously.
There is some uptick in interest in various BSDs as a linux alternative given the rough few months it's been having. I know 2 companies that are trying to see if they can switch some edge nginx proxies to run on OpenBSD. A RCE in a TCP handshake is up there tbf
Interesting to hear / read this. I know OpenBSD has made a lot of improvement in the past 2 - 3 years on performance. But I would have thought for heavy traffic it is still not anywhere near FreeBSD or Linux.
But I guess we are at the point security is important and hardware ( CPUs ) is relatively cheap?
It was a brief conversation in a social gathering for smallish engineer-led startups in my area. The one guy who went into details mentioned they are evaluating OpenBSD because of its presumed focus on security. I doubt they are super high throughput, but they might land on something else too. They run their own hardware at least, so the hardware compat issues of OpenBSD didn't seem to be a problem for them. FreeBSD was mentioned as well fwiw. I don't know how far along they got in performance testing or actual switch.
Just general angst some people were expressing for publicly exposed resources. The common wisdom just a few years ago was that throwing an internet facing nginx or haproxy on an up-to-date LTS linux with some firewall debouncer is as rock solid as you could get. And instead, you should worry about security in your own upstream app, not Linux's TCP sockets or nginx's regex path matching.
> So the project is alive and it seems more active than previously.
It indeed is! Looking forward to the next release, which should happen "soon", as 6.4 is already 4 years old.
If this is comming I can ditch proxmox
This brings back memories of watching someone do a live demo of bhyve at nycbug. I think that just have been almost a decade ago now, but the live demo was amazing.
Nice! Can we have high availability / clustering for jails next?
That sounds interesting. Could you elaborate what that would mean?
Even Solaris in Kernel Zones uses virtualization for the kernel part to have 'live migration' - regular Solaris Zones Containers does not have 'live migration' - you can just start/stop them.
But ... OpenVZ in the past had 'live migration' - so it would be nice to implement that feature into FreeBSD Jails to have 'high availability'.
Nice!
Welcome to 19 years ago I guess (KVM got it about that time)
L4Linux on NomadBIOS got it in 2002, and Xen got it about a year later, after I had met with the Cambridge Xen team at SOSP in upstate New York.
Yes, Xen is several years older than KVM. The NSDI paper on live migration from the Xen folks is from 2025.
And I am one of the authors of that paper :-)
I know, ;) this was more for context.
Also neither of us spotted that I wrote 2025 instead of 2005 but I can't edit anymore. Moral of the story, don't write comments at 7am.
Same with Docker.
FreeBSD had Jails containers since 1999.
Linux joined the party starting with on 2014.
so bhyve getting live migration finally means sylve might actually be usable for production clusters, been waiting on this.