game_the0ry 5 years ago

As a front end developer, I am really happy to see resources like this.

Developing for the browser is a real challenge. I think working with html / css /js has been a neglected skill for a long time - most software engineers look down on that type of work and its rarely covered in comp sci course work.

Still, its good to see a lot of progress has been made, this book included.

My only critique - why use python instead of node.js?

  • pavpanchekha 5 years ago

    Author here. I wrote up my answer here: http://browser.engineering/blog/why-python.html

    Basically: server-side JavaScript is just not as widely known as Python, and it'd be additionally confusing when our browser starts running JavaScript. And in-browser JavaScript is a bit too restricted (by things like the same-origin policy) to do the whole thing inside a browser.

    • game_the0ry 5 years ago

      I see. I tried searching the table of contents as to "why python" and could not find it, but that link does more than enough to explain the "why."

      I am resisting the urge to disagree, but since you did (literally) write a book about building a browser, I will defer to your expertise and try to learn from you :)

      • grae_QED 5 years ago

        >I am resisting the urge to disagree, but since you did (literally) write a book about building a browser, I will defer to your expertise and try to learn from you

        Respect

    • bogomipz 5 years ago

      This looks fantastic! Really excited to see this. I'm looking forward to reading the updates as well. Cheers.

    • fouc 5 years ago

      I can see that you'd want to pick a language that is good for teaching with, hence why you went with Python. It's probably too late, but I imagine Elixir would also be a great choice for this project.

  • dgb23 5 years ago

    I agree, JS _the language_ would be a more "obvious" choice for this, since it is both generally much faster and more popular in web development. I assume they are using Python _the ecosystem_ here. It probably comes with packages better suited for rendering specifically?

    I don't think either is a super compelling choice anyways for this type of work. I think you want to use a systems language here. However Python is completely fine as a teaching language. Pretty much anyone who knows a similarly structured language can read it. And there is very little noise. So it can serve as a good reference if you want to follow along with a different language.

    • traverseda 5 years ago

      I think writing this in javascript would add some confusion, as people learning about this might have a hard time understanding exactly where the javascript is running.

  • tablespoon 5 years ago

    > Developing for the browser is a real challenge. I think working with html / css /js has been a neglected skill for a long time - most software engineers look down on that type of work and its rarely covered in comp sci course work.

    IMHO, those are job skills and not comp sci topics. They shouldn't be part of a degree program (except the most superficial treatment required to get some ugly UI up that may be required for something else). You have your whole career to pick them up.

  • tyingq 5 years ago

    Python seems like a good match since the tutorial seems to use what comes with a base python install. Python ships with Tk, for example, so the drawing section is cross-platform and not bogged down with a lot of instructions on adding new stuff with a package manager. That would detract from the lesson, IMO.

simonw 5 years ago

I was interested to see that this uses the DukPy wrapper around Duktape for the JavaScript interpreter: https://browser.engineering/scripts.html

This made me start digging into whether this was considered a "safe" way of executing untrusted JavaScript in a sandbox.

It's not completely clear to me if DukPy currently attempts safe evaluation - it's missing options for setting time or memory limits on executed code for example: https://github.com/amol-/dukpy

There's a QuickJS Python wrapper here which offers those limits: https://github.com/PetterS/quickjs

I'm pretty paranoid though any time it comes to security and dependencies written in C, so I'd love to see a Python wrapper around a JavaScript engine that has safe sandbox execution as a key goal plus an extensive track record to back it up!

eatonphil 5 years ago

I can't wait for Browser Engineering to show up as a university course a la compilers, operating systems, networks, etc.

  • amelius 5 years ago

    A browser is basically an operating system inside an operating system.

    The funny thing is we can have the MINIX microkernel discussions all over again :)

    • gnull 5 years ago

      Looking at the book's table of contents, I disagree. Browsers may resemble OSes by the size of the code base or by the amount of optimization involved, or by their importance for the modern world, but not by the types of technologies involved. I doubt the Browsers course will intersect a lot with OSes course.

      EDIT: Reading you comment again I suspect you might have been joking :)

      • groby_b 5 years ago

        Which part of OSs do you expect not to be covered?

        There's IPC. There's memory management. There's process management. There's network management. There's security. There's device management.

        They all happen at a slightly higher layer, but they all exist similar to an OS. (I'm not sure if the higher layer makes it easier or harder to understand - but in terms of what you need to know, an OS class or three is definitely helpful)

        • gnull 5 years ago

          > IPC, memory management, process management, network management.

          I imagine the non-trivial parts of these are done for the JS VM (correct me if I'm wrong), and therefore a VM design course would have more intersection with Browsers with respect to these disciplines than an OS course.

          > security

          This one is everywhere, it has no special connection to OSes.

          > They all happen at a slightly higher layer

          Slightly?! That's an understatement of the week! The difference in abstraction levels is huge here and the specifics of the two levels are very, very different.

          > in terms of what you need to know, an OS class or three is definitely helpful

          Sure. But I think it's as useful as any systems programming course. I can agree that systems programming is a good preliminary for both Browsers and OSes, and learning either of the two will teach you a good deal about systems programming, but I doubt they will repeat each other.

          • groby_b 5 years ago

            These are all part of the browser outside the VM too.

            Multi-process architecture requires you to think deeply about IPC.

            Memory management is all over the place - there isn't a browser without custom allocators, investment into GC, etc.

            Process management -> see multi-process architecture.

            Network management: Browsers need to handle a tremendous amount of network issues. I mean... that's what they do. Outside of the VM, too.

            As for "security is everywhere" - the whole point of an OS (and a browser) is to make it possible for security to be everywhere. To provide the primitives that you can securely build on.

            > Slightly?! That's an understatement of the week!

            Not really, no. I've worked on embedded networking stacks, on full-fledged OSs, and on browsers. I stand by "slightly". Yes, granted, a browser doesn't get quite as bit-fiddly as a on-the-metal OS, but it's a matter of degrees, not quality.

            Can you work on many areas of a browser without ever touching OS-like code? Absolutely. This particular book has a good chance of avoiding most, because it focuses on the rendering part.

            But a browser, as a whole, provides an abstracted platform just like an OS. And it echoes many concepts, if in slightly different forms.

            • gnull 5 years ago

              I think a good example of two solutions used in browsers and OSes that are meaningfully similar (and not inherent to all systems software) would convince me better. Do you have one?

          • rictic 5 years ago

            The security problems of a browser are very similar to the security problems of a traditional mainframe OS.

            You need to execute potentially malicious code in isolated containers (processes / pages). You need to protect the system from these processes, and protect these processes from one another. You need to run some code with elevated privileges / additional capabilities (setuid / permissioned APIs), including some with superuser privileges (root user processes / browser extensions).

        • zeroxfe 5 years ago

          > There's IPC. There's memory management. There's process management. There's network management. There's security. There's device management.

          OSs are built on hardware, browsers are built on OSs. Even though they "sort of" do the same types of things, the abstractions are very different.

          - OS memory management is more about managing hardware page tables, copy-on-write, page sharing, page protections etc. Browser memory management is typically custom allocators optimizing for different types of structures.

          - OS process management is about using CPU primitives to multitask segments of code, context switching, timers, wait queues, interrupt management etc. In the browser, it's more about managing webpage/process relationships, application threading, rendering pipelines, and the JS runtime.

          - OS network management is all about interfaces, packet processing, buffer management, low level protocols (ethernet, IP, etc.) Browser network management is all about protocols, and not really much about the hardware.

          Sure there's a lot of broad-stroke similarity between OSs and browsers, but for a university course, where one typically gets deep into the details, they're entirely different.

          • nyanpasu64 5 years ago

            I think that highly optimized browsers have to deal with memory usage and page sharing as well. And QUIC started out as a userspace implementation shipped with Chrome handling packets and such, so I think browsers and OSes are not "entirely different".

  • chrishtr 5 years ago

    Author here.

    That's exactly what we are hoping for.

    http://browser.engineering/preface.html

    So far, my co-author Pavel has taught from this book multiple times (including this semester). In the spring at least one other university will offer a course. We'll list all known courses offerings on the website.

    Also, if anyone would like to teach from this book, please get in touch!

harry-wood 5 years ago

Building a browser as a desktop application would be quite hard, but I reckon I do it as a web application.

  • posedge 5 years ago

    Maybe we should have a SaaS browser.

    • edub 5 years ago

      s/ already exists: Mighty [https://news.ycombinator.com/item?id=26957215] which is $30-50 a month for remote session to Chrome running on a very powerful Ubuntu system.

      Looks like RemoteHQ launched something called Remote Browser as well, which is a SaaS browser, but more for collaborative purposes with more than one person using the same browser at the same time.

tester756 5 years ago
  • chrishtr 5 years ago

    Author here.

    That article, along with a number of other resources, are listed here:

    https://browser.engineering/bibliography.html

    In my view, a critical part of really learning how something as complicated as a browser works is by trying to build it yourself. That's why our book is oriented around building a browser as you go.

  • bogomipz 5 years ago

    This also looks great. Might you or anyone else know where to find the accompanying video? I get "Sorry This video does not exist." for the embedded video link.

varispeed 5 years ago

Many years ago, probably 20, I went on a task of implementing a web browser. I remember I gave up at rendering tables. I couldn't wrap my head around on how to properly size them. It has become extremely complex quickly to address edge cases and I eventually gave up when I couldn't understand what's going after having a two weeks break. Probably if I had money and was able to commit full time I could eventually get it, but I had to focus on commercial work and putting food on my table.

edit: it's a great article! But nothing on rendering tables :-)

  • dmitriid 5 years ago

    > Many years ago, probably 20, I went on a task of implementing a web browser. I remember I gave up at rendering tables.

    HTML 5 effort has cleaned up a lot of behaviors and specified how browser tags should behave. So it is, possibly, an approachable task now. Still daunting though.

    • pavpanchekha 5 years ago

      Tables are still not that clean! But luckily tables are way way less important than they were in the past, so much so that browser differences in table rendering leaves most pages readable.

amelius 5 years ago

Is this written by actual web browser engineers? If so, what fields did they specialize in?

jturpin 5 years ago

This is awesome. I've always wanted to know how the actual layout portion works (or at least, can work in a simple way). I think these kinds of resources are really valuable and people should be empowered to make bespoke-ish web renderers as the need arises.

  • pavpanchekha 5 years ago

    Author here. Layout is my favorite part of the browser—my dissertation is largely about formalizing browser layout—and so far the book only covers the basics, like the layout tree and how layout is computed with multiple tree traversals, but even understanding those basics I think gives you a huge advantage when thinking about web page layout tasks.

ngai_aku 5 years ago

Awesome to see this here! The course that accompanied this textbook was among my favorites

louisvgchi 5 years ago

There’s a 90’s style browser in Haskell here for demo purposes. Lacks JS.

https://github.com/chrisdone/vado

  • zenithd 5 years ago

    This book's browser supports JavaScript but the book itself skips over it because the browser just uses DuckPy. Which is a reasonable pedagogical choice. Building even a moderately "realistic" JavaScript interpreter would be its own semester-length course.

    (This is a really fantastic resource; I'll probably use it as the basis of course once I get around to semi-retiring and taking a teaching job. Thanks, OP!)

a_c 5 years ago

Love the bibliography section. I have always wanted to reinterpret HTML into other representations. These resources give me good reference

baybal2 5 years ago

It's now nearly impossible to build a web browser from scratch because of runaway explosion of web browser features, and proprietary API extensions.

W3C here is unfortunately a part to the problem.

Standardisation is good, but letting google pour streams halfassedly written RFCs onto other browsermakers is not good.

Non-enforcement of standards is also bad, and it's bad to extend W3C privileges to companies who themselves selectively implement their own proposals, so others' browsers can't match their behaviour.

  • jahewson 5 years ago

    Actually a from-scratch web browser is being built:

    https://www.fastcompany.com/90611677/flow-ekioh-web-browser-...

    Also you should take a look at the WHATWG because it’s far more relevant than the W3C nowadays.

    • dmitriid 5 years ago

      "The total word count of the W3C specification catalogue is 114 million words at the time of writing. If you added the combined word counts of the C11, C++17, UEFI, USB 3.2, and POSIX specifications, all 8,754 published RFCs, and the combined word counts of everything on Wikipedia’s list of longest novels, you would be 12 million words short of the W3C specifications"

      https://drewdevault.com/2020/03/18/Reckless-limitless-scope....

      No idea how Flow does it, but building a browser is nearly impossible.

      • fabrice_d 5 years ago

        It's well known that Drew Devault count is meaningless since it includes dupes, drafts, and unrelated specs. Still, the space to cover for a from scratch browser is huge.

        Flow didn't start "from scratch" recently, it's an evolution of a primarily SVG+CSS renderer for set top boxes. They also re-use Spidermonkey as their Javascript engine.

        • dmitriid 5 years ago

          > It's well known that Drew Devault count is meaningless since it includes dupes, drafts, and unrelated specs.

          It's not meaningless. Because in order to implement a browser, you have to figure out which of them are dupes, deprecated, drafts etc.

          And even that won't help you. Because a huge amount of "deprecated" standards are in the browsers. A huge amount of stuff in the browsers is still at the "community draft" stage, and yes, you have to implement that, too.

          Microsoft simply gave up, forked Chromium... And they still can't keep up: https://web-confluence.appspot.com/#!/confluence

          • fabrice_d 5 years ago

            The Edge graph stops before they switched to chromium though...

            • dmitriid 5 years ago

              Yes, that was my mistake. Doesn't make my words any less true.

        • carapace 5 years ago

          > it includes dupes, drafts, and unrelated specs.

          Even if it's overblown by, say, three times, that's still over thirty million words.

          • fabrice_d 5 years ago

            Note that a large effort has been made to make specs more precise, to be easier to implement in an interoperable way.

            That contributes to "word bloat", but it's not necessarily a bad thing. Picking the right metrics is not always that easy!

            • dmitriid 5 years ago

              > Note that a large effort has been made to make specs more precise, to be easier to implement in an interoperable way

              This is true for HTML5 which defined full browser behaviour, including things like improperly closed and improperly nested tags.

              Many, many other specs? Not so much. Especially the crap that Chrome has been pumping out the past several years.

    • carapace 5 years ago

      Flow browser isn't FOSS.

      > WHATWG [is] far more relevant than the W3C nowadays.

      Which is arguably part of the problem.

kizer 5 years ago

Love first principles stuff. Great job.

ofou 5 years ago

this builds a web browser like the build-your-own-x movement?

  • pavpanchekha 5 years ago

    Author here—the browser the book works through is, uhh, pretty limited, so I don't imagine you'd want to actually use it for web browsing. It's more like writing your own toy compiler or operating system, to learn how they work.

    • ofou 5 years ago

      It's perfect then! I'll read and work the exercises.

      Thanks for answer!

butz 5 years ago

Neat, but what about integrating widevine support?