You can pry my debug prints from my cold, dead hands.
I get the point this person is making, but at times (and depending on language) it's the fastest and most efficient option. "Oh, the value is not what I expected" -> fix -> "value is good after testing, can remove the debug print".
It's not elegant, yes, but it does the job with zero preparation and works for every language.
Now if things are so hopeless the debug print doesn't even work, then yes, get a proper debugger.
The fact that you _know_ where or which variable it is, and know where to put the print in order to see the value indicates that the bug is either trivial, or that you already have an idea of where the bug is!
I argue that printf debugging is more difficult if you dont know _what_ is incorrect, or if you dont know where a value is being changed. Stepping through your code is a great way to find out.
To me, actual stepped debugging is like having a print statement for every single line in the code base.
I argue that printf debugging is more difficult if you dont know _what_ is incorrect
Or even impossible? I use print debugging often, not even only in cases where there's no step debugging possible and/or I'm too lazy to fire up a config/software which allows debugging, but from time to time I encounter bugs which I really can't even begin to imagine how terribly awkward it would be without step debugging. For example: finding a bug in a mark&sweep garbage collector written in C which only occurred in certain LTO builds. To get a picture of the problem I stepped through assembly code while looking at certain memory locations, regsiters, the C stack, and some related variables. How would one tackle that with print debugging? Plus the print statements would affect the very thing being debugged? That should be fun. Not to mention the effort it took to even be able to reach the exact code location where all of this happened. Would almost require to randomly sprinkle print statements all over the place.
If you have _no idea_ where a bug could come from in your code You should spend the time understanding your codebase before you even begin debugging. At the very least you need to be able to reproduce your bug to reliably fix it. How can you have no idea where a bug is happening if you can trigger the error in a code path?
EDIT: I don't want to go so far as to say that your example can never happen, but I can't believe there is any domain where it is the norm
Ever inherited a 500k+ SLOC program with piss poor documentation that you can't rewrite (at least not within the needed timeframe)? Or worked in an office that, on some projects if not overall, experienced high turnover (due to promotions, internal movement, quitting, etc.)? It's very easy to end up with a project in a state where no one actually knows most of the code. A debugger shines here. In fact, the debugger can be a fantastic way to learn the code since the dynamic behavior of a system can often be hard to infer from its static representation (looking at you WPF and XAML, or any project where architecture astronauts went hog wild with certain design patterns).
Changing code that you don't understand because you "found" the bug with a step through debugger is a recipe for disaster. In those situations (high turnover/fast moving tech debt accumulation) perhaps no one cares that the code base is becoming unmaintainable, but it isn't a good practice.
> Changing code that you don't understand because you "found" the bug with a step through debugger is a recipe for disaster.
I don't get your point, because that's not what I suggested someone should do. I think (on rereading my original comment) my point was the exact opposite: Develop an understanding of the code through the effective use of a debugger:
> In fact, the debugger can be a fantastic way to learn the code since the dynamic behavior of a system can often be hard to infer from its static representation (looking at you WPF and XAML, or any project where architecture astronauts went hog wild with certain design patterns).
You seem to have inferred something from my comment that I did not say.
If I did I apologize. I agree that using a debugger can be a good way to understand what a codebase is doing. I think we are on a similar page. Sorry for the miscommunication
Some issues give you roughly zero feedback. If your error doesn't give you a usable stack trace this about the only option although it isn't probably ever the norm. But the key here is that it works with debugging/logging, even when the error is only a low probability issue the only shows up randomly in production. Debuggers are of no use in this case.
Sure. I'm not arguing using a debugger if that is what you prefer. I'm saying that to have no clue where in your code base an issue is cropping up, like _0_ idea, says to me one should spend some time understanding the code before any type of debugging.
Just add more print instructions. And if that doesn't work, add even more print instructions!
Add some instructions to print "**1**", "**2**", "**3**", ... to see how far the program goes, which branches it takes, print a few variables that look interesting. If that fails print the uninteresting variables, remove "**1**" and add some "**2a**", "**2b**", ... in between 2 and 3 ...
After a while, you discover the correct spot and the correct variable, and the error is trivial.
I do this too; but it seems to me this is the case where using a debugging tool would probably save more time.
Looking at the effort involved: here you have to spend the time to add a bunch of print statements throughout bits of code; then compile/run the program (repeating until you've found the right place).
With a debugger: you'd have to take the time to set up the environment and learn how to use it (e.g. set breakpoints in parts where things would be interesting). But then the "look at interesting values" wherever isn't significant additional effort.
This is really the problem. Nobody has spent time and money to make debugging _good_ in the languages you use. Few people pay for good tools any more, and only a tiny fraction of users ever contribute improvements to the open–source tools they use (paid or otherwise). So the average debugger hasn’t gotten better since the 90s.
I do programming in multiple languages too. In those where I can use RR and Pernosco, I use them every time. In those where I can’t, I miss RR and Pernosco every single time.
* Debugging an 8 bit microcontroller in-circuit with an IDE, conditional breakpoints, step in/out/over, full stack visibility, and the ability to manually poke at values/memory/io
* Debugging an AWS lambda function with printf and high latency logs
It doesn't matter how hip, how modern, how fast, or how big the environment is. What matters is whether or not the ecosystem cares enough to make debugging nice.
Print debugging is sometimes acceptable. Scrubbing dishes with your fingertips totally works, but we all have dishwashers. Dish–washing machines are way better than scrubbing manually, with your fingers or with anything else. Load the dishwasher, then go do literally anything else while the dishes are washed. Even if all you do is go watch television, you will be better off just because you didn’t stand there scrubbing dishes by hand for an hour. More human potential has been wasted by drudgery than by any other cause.
Print debugging is like scrubbing dishes by hand, and neglecting your dishwasher.
Pernsoco is like those commercial dishwashers that blast the dishes with super–heated steam to wash them in a few minutes instead of 45.
I don't have a dishwasher and it isn't because of cost or space. I just don't think they are worth it unless you save up dirty dishes for a few days. If you wash everything right away it's easier to not use the dishwasher, especially with a good sized drying rack for easy reuse of common items.
I suppose I feel the same way with print debugging. Mostly that's all I need and the problems I have which would benefit from better tools are rare enough and diverse enough it isn't worth investing time in them.
Dishwashers are better for the environment. There’s even a new Bosch model with a passive heating element (crazy) for the dry cycle. Perhaps if you live solo and don’t do much cooking you don’t need a dishwasher. But even in that cases I’d bet you’re better off running say a half load once a day.
I've heard that before but I doubt it. The embodied energy in a dishwasher installation will certainly take a while to recoup and it would depend a lot on how one measures hand washing.
Anyway I don't really care about some insignificant "environment" difference - the affect on daily convenience far outweighs it.
Just speaking personally, if I cared enough about efficiency to get a dishwasher, I'd rather instead have a bigger sink and setup a wash and rinse bin for dishes. Its the classic way to preserve water at a campsite when doing lots of dishes.
A tub setup would be more environmentally friendly considering the resources put into a dishwasher these days. Certainly drying via a towel is better then an electric heater.
Modern dishwashers recycle water. I’m not sure a wash/rinse setup actually uses less water. I guess it depends on how big the sink is and how much water you fill it with. The Bosch passive heating element is really neat btw and avoids electric heating.
Sure, if you live alone and don’t cook, then washing dishes by hand won’t take very long. Maybe it’ll take less time than loading the dishwasher. But that’s a fairly uncommon scenario, I think. Most people cook dinner for their family, and then have a whole dishwasher full of dishes to wash every single night.
Interestingly, its good to be uncommon. In my household, the one who cooks does not wash up the plates etc. We don't have a dishwasher. Many hands make light work. We call this "social cleaning".
That’s true, though leaving plates with food on them to dry out is a bad idea. Run them through the dishwasher sooner rather than later!
Of course I’m as lazy as the next person; there are dishes in my sink right now that could be put in the dishwasher, if only someone would empty the dishwasher of the dishes I washed the other day…
True, though be aware that not all dishwashers have one. Mine just does a few repeated wash cycles with no detergent, then one with detergent, followed by a rinse and dry cycle. It’s not great, but since it’s not actually broken I can’t really justify tossing it.
Even so, I can only recall one time where it didn’t get a baking dish fully clean. I just left it in the dishwasher and ran another load the next day and it was fine. Then I baked some more enchiladas in it.
The pre-wash detergent "slot" in my dishwasher is a slight depression labelled "Pre-wash" on the flap of the normal-wash detergent dispenser. In other words, just squirt a bit of detergent into the dishwasher (anywhere, it doesn't matter) before running it.
A new dishwasher doesn't mean a good one. They vary in quality.
Are you using a rinse-aid? If not, you're doing it wrong.
I never pre-wash. I'll manually remove solids when I'm done eating (e.g. with a spoon), but most people do that anyway because it'll be a pain to hand wash if you don't.
Are you filling the pre-wash slot with detergent? Because that's actually pretty vital for them to properly clean dried-up stuff. By not filling that slot the machines are crippled, essentially.
> Scrubbing dishes with your fingertips totally works, but we all have dishwashers.
I don't have a dishwasher. I don't know anyone who has a dishwasher. Heck, I don't even think I've ever seen a dishwasher on any appliance store I've been to. Our apartment was built with space for a dishwasher and a clothes dryer, and I've always wondered why, because nobody I know personally has either of these appliances.
There's no hyperbole (except perhaps in the part about appliance stores, it's possible that I have seen a dishwasher in one in the past and I'm just not remembering). I really have never seen anyone using a dishwasher or a clothes dryer, and I really do wonder why was my apartment built with space for both.
I'm really curious where you live, because it sounds highly unlikely for any western country.
I might imagine a Christian or Muslim (fundamentalist) society looking down on appliances that do 'womens work'. And I've spent some months in southafrican townships. Such places have no dishwashers. Or did not have them.
I'm not trying to doxx you, or attempting to get people to post prviate info, but I would love to know your demographic.
Ah, ok. I didn’t realize such places still existed. Around here, dishwashers are virtually ubiquitous. It’s actually more common for people to not have washing machines for clothes than to not have a dishwasher. People without their own washing machine can go to a laundromat, for example, or to a community–owned washing machine in their apartment complex. Even people who own washing machines will go to a laundromat for some things, like washing quilts or blankets, which require a larger machine.
Meanwhile there are no equivalent places where you can take your dirty dishes to wash.
> Meanwhile there are no equivalent places where you can take your dirty dishes to wash.
Considering that "throwaway packaging and cutlery" is a common solution to this: Order food, the packaging, plastic forks and spoons and napkins included. And seeing that a large demography is trying to live off "less waste", I do see an opportunity here.
A community "diswashing service". It would need solid infra and tech to not immediately turn into a rat-infested-mould-party, but I truly think that a place where e.g. you drop your dishes, cutlery, glasses etc, and can pick them up in the evening filled with a meal, is an opportunity. For example.
That would be interesting, but I suspect that transporting your dishes there and returning to pick them up would be more work than just loading them into a dishwasher. Unless the service were quite close by.
Which reminds me, wasn’t room service once a perk you could get in some apartment buildings? You don’t often find dumbwaiters in apartment buildings any more, for example. I would pay for that if it were available.
I'm not trying to start an Off-topic religious debate here. Sorry for coming over condescending. If you read "all Muslims" you read far too much in it: the fundamentialist part is important. Nor did I say, nor imply that this goes for anyone who identifies as Muslim. If you read that: I'm truly sorry.
I know from experience that the statement holds true for several Christian fundamentalists and Jewish fundamentalist (not all!)communities. I have only read (but not from experience) about how women (especially those of other sects) were treated in the Chalifate from ISIS and how their "work" was deemed slave-worthy.
It's still outright wrong. Your experience with Christian and Jewish fundamentalists does not apply to Muslims. Please stop with spreading incorrect information about Muslims.
ISIS is a fringe group whom we don't give any semblance of credence to, and even they would not look down at washing machines "because it's a woman's job". This is plain ridiculous.
The secular West continues to group all religions together and marks any religious person as some sort of sheep.
Also, let's stop with this "identifies as" business. A person is either a Muslim or a non-Muslim. A person who does not believe in God cannot "identify as" a Muslim for example. That being said, the religion is open to anyone who wants to accept it.
As long as you accept it is their right to define that, and not you!
One might use "identifies as" to avoid the classic response of "ah but x was not really a y because z". Someone who believes themselves to be a Muslim simply is. You can call them a bad Muslim, ignorant, misinformed, misguided etc. But they are what they believe themselves to be.
(I am entirely sympathetic to your original frustration. GP was applying very odd assumptions)
> As long as you accept it is their right to define that, and not you!
That's not how Islam works. That's how it was able to survive against the torrent of corruption that affected Christianity and Judaism. The Quran and Hadiths are preserved so that we're easily able to consult them to determine who is and isn't a Muslim. Note that this isn't gate keeping, Islam is wide open for anyone to accept. However, the tenants are strict enough to survive the passage of time.
> You can call them a bad Muslim, ignorant, misinformed, misguided etc. But they are what they believe themselves to be.
This already happens. I'm talking about the extremes that the secular West has adopted, whereby people can inherit their parents religion, even if they don't believe in it.
> The Quran and Hadiths are preserved so that we're easily able to consult them to determine who is and isn't a Muslim.
And yet those poor benighted souls believe they follow those very same texts that you say would reject them. How is that possible? Because their religious leaders interpret the texts differently, emphasise different aspects.
For a Christian equivalent. The Catholic church can excommunicate someone. They will no longer be Catholic but there is no power that can stop a person who believes themselves to be Christian from being termed that. "Normal" Christians might prefer not to be associated with various people, can call sects Extremist or Fundamentalist etc. We simply cannot claim they are Not Christians.
Not everything is up for interpretation in Islam. For example, homosexuality is prohibited, and no amount of "interpretation" is going to change that. Same goes with drinking wine or intoxicants. There is no valid interpretation of Islam that allows such behaviors.
You'd be surprised to find that said "religious leaders" you're referring to don't know the basics of exegesis. This applies to ISIS for example. I invite you to read the many Hadiths about Khawarij, those immediately reject your premise that all interpretations are equally valid. They're not.
No matter how good the debugger is, it still requires learning a new tool and the interface. As long as a debugger has a gdb-like simple CLI, I will often use it because I know it. But I will also still use print statements when I want to scroll through a log of values, rather than press c <enter> For an indefinite number of times.
Debugging is rather good in elixir (Iex.pry is quite nice). I'll still print debug for a faster through-time experience, since most work in elixir is about transforming data across several steps.
It is, also much much easier to identify a race condition with print debugging than a debugger.
Print debugging is invaluable for actually having any chance to understand what happened in a complex bug in production - especially if you're working on user-run apps, not SaaS.
Debuggers help immensely if you can reproduce the bug reliably and in relatively decent time, but if you can't logging/print debugging is your only hope, since no one is going to run their code in production with RR attached.
Came to comment this. Print debugging makes me so fast at work. Unless you give me an easier and better tool I so no reason to change. Works in every language, exactly how I'd expect, and is exceedingly easy to explain to others.
Yep. You can blow peoples minds in ANY LANGUAGE by using print debugging sometimes. It's amazing, but true. At least it's been that way for the ~25 years I've been doing it.
You think you're here? Let's put a few print statements in there to confirm your execution path is actually what you think it is, and those values are what you expect them to be.
> You think you're here? Let's put a few print statements in there to confirm your execution path is actually what you think it is, and those values are what you expect them to be.
You could just put a breakpoint there, look at as many values as you want, and not have to re-run it if you think of another.
True, but I don't want to have to exit out of 5 breakpoints each time I run through the scenario. I just want to run it with the print output logging the procession and key variables as a summary of relevant info.
To me that's the key difference. If I know what I'm looking for, prints are great. If I don't know what I'm looking for, breakpoints are a must.
Print debugging makes me fast too, but there is eventually a breaking point (get it) where print debugging isn't getting me the answers I need and I have to switch to using other options like interactive debugging.
There are a few languages where print debugging isn't as fast, but that always requires a ton of tooling. I found that debugging Java using Eclipse or Intellij was faster than printing, as I could introspect state and change values while the process was running. Similarly, some scripting environments I've used have allowed for robust state and flow analytics.
But in a normal environment ? or using C++ ? It's generally quicker (though much dirtier) to find simple bugs using print statements to track down assumptions ; when you're in the weeds with a complicated problem or need to track down a crash, then yes, bring out the big guns. But for normal development, I don't think it's completely necessary.
Debugging in C++, is a bit of a pain sometimes. GDB tools are good paired with an IDE like VSCode, but due to the abstractions in C++, inspecting variables require verbose code - beyond the convenience of debugging in a higher level language like Python or even Java as you suggested.
Eg: tracking the values of a std::vector isn't obvious, since there is std:begin and std::end iterators in the vector namespace and you have to manually dereference the whole range in the debugger.
Are you under the impression that investing the time to learn to use a purpose-built debugger will be a net time loss?
I used to print-debug a lot, but since I started working with a stack where I use an IntelliJ derivative - that is to say, where the debugger actually works - and I'm never ever going back. It's better by a huge margin.
Have looked at, say, the Python debugger as used by PyCharm and thought "well this seems like a bludgeon, I only need to see how <variable> isn't working as expected" and bung in various print() statements in and around the Thing concerned, sprinkling them wider and wider if the bug isn't localised to a function or even the class it's in - I may have supplied naff information to a class when instantiating it, for example.
Debuggers are simply too much of a complicated bludgeon. I can understand a need for debuggers for machine code/assembly programming, or deep system programming when you're hitting the metal, or even low level languages like C.
I've yet to find a need for a debugger for high level languages like Python. Perhaps some day. Until then, I'll plough the time which would be required to Get To Know the debugger, into writing code.
Whenever I try to print debug, I end up missing the mark on my first print statement(s).
If I just set a breakpoint, I can jump back in the stack, or step forwards, so I can easily find the real place the error happens.
Of course I sometimes still print debug - when using languages with bad debuggers, languages where I don't know the tooling, languages with horrible control flow (javascript callbacks), or constrained environments, printing/logging is sometimes the best you can do. But that doesn't mean it's the best there is.
You could, but then you're programming in yet another language, at the same time, on one single line, without type-checking, maybe without turing-completeness, functions or expressions.
Agree. Print debug with compartmentalized domain tracing using flow control to print or not print is the only way to debug live life safety embedded systems.
1. Ability to prevent wasting resources on a system that does not need to be debugged.
2. Retain most print debug branches that were useful.
3. See how the internals are functioning on a live system with the issue that has yet to having a lab level proof.
Embedded systems, where there are timing things going on, are an interesting special case. You can't just hit a breakpoint without (in the worst case) causing physical damage.
It's even worse if it's multithreaded as well. If you hit a breakpoint, one thread stops and the others keep going, but they keep going in an environment where one thread is effectively dead (the one at the breakpoint). Everything is invalid from then on, in all threads. So breakpoint debugging is certain to be doomed.
Print debugging is not doomed, but it's challenging. The problem is that the time to print out the message can change the timing, and therefore change what happens. It's not certain to do so, but it's something you always have to keep in the back of your mind: "What if this changed the sequence? What if what I'm seeing in the printouts isn't what I would see if the printouts weren't there?"
I think that print debugs are a great first step. The problem are people who can't use a proper debugger, but that's not the same as saying print debugs are always bad.
I tend to use print debugs in two cases. One is when I'm pretty sure I know what the problem is, and it seems trivial. It's often faster to just confirm & fix that way. The other is when I am not even sure where to start, so I put a few canary prints here and there based on hunches until I start to narrow in mentally on what the issue might be.
> You can pry my debug prints from my cold, dead hands.
I clicked on this about to type exactly that phrase.
Here's why: the CORE PROBLEM of debugging has nothing to do with process, it's the need for the human developer to understand the operation of machine code when faced with a specific set of runtime conditions.
You can't debug by just reading code, because the bug only shows up at runtime. Most people agree with this.
Likewise, you can't debug just by WATCHING a program, for the same reason. You need to be reasoning about the operation of the abstracted code at the same time. And almost always you need to be inspecting and reasoning about the internal states driving the behavior.
Debugging tools, as nice as they might be, can only do the "watching" part. They are a fine way to do it, but you still need to be making the decision, based on careful analysis of the code, on which elements to watch.
So... given that you're staring at code in your editor and thinking about how to extract the needed internal state to validate your predictions... why not just print it?
That is: print/console debugging isn't a hard task you "have to do" in the absence of a fancy debugger, it's exactly the task that you must do if you're going to solve the problem. The fancy debugger, at best, is only saving your the time taken to TYPE the printf() expression. And in practice even that tends to parallelize with the analysis in my experience, as you realize things while typing out their expression.
Pernosco lets you record your program once, then query the resulting database over and over. This is like adding printfs to the program after it has run, any time you have a new idea for what it should have printed out. Here’s a demo:
Notice in this demo how he clicked on a line number, and got back a list of all the times in the program where that line was executed. Not just a single breakpoint with the option to continue, but all of them. Then he queries for the value of a variable at all of those points, and gets all the values printed out simultaneously, as if there had been a printf at that line. Then he filters the list to show only calls made on a specific object, as if the printf had an if statement around it. All without needing to rerun the program, or even be on the same computer as where the recording was captured.
This is what the author means when he says that all debugging tools can be _better_ than printf debugging, if only we put some more time into improving our tools.
Again: that simply saves me the time it takes to add logging to trace the value of that variable. It fails to account for the much larger amount of time needed to understand which variable to trace.
By all means, use fancy tools like replay debuggers (of which Pernosco is only one example) if they make you happy. I'm just saying that (1) there is a hard cap on the value that tooling can provide, because the real limitation is between our ears. And so (2) I'm going to spend the limited learning bandwidth between my ears on things like genuine new technology and not better tools that might provide a mildly incremental improvement to tasks I do already.
Hmm. I see what you’re saying, but I’m not sure that there is a hard cap. Pernosco has allowed me to solve harder problems than I would be able to solve without it. Or perhaps there is a hard cap, but that cap is much higher than you would think. High–quality tools might get you closer to that cap than simpler tools would.
For example, I participated in a project recently to extend Flex so that it could generate both Rust and Go code, in addition to C. Pernosco was an invaluable aid here. I was able to get Flex to generate a C program and also a Rust program, then compare the execution of those two programs. I recorded them both, and opened them each in their own window. Keeping them side–by–side, I could probe the innards of both programs simultaneously. Since both programs were supposed to implement the same state machine (and thus recognize the same list of tokens), it was easy to find places where the Rust version went off the rails. I was able to do in hours what would have taken days with any other method, if I had succeeded at all. The behavior of that state machine is not at all obvious, and I might never have unraveled it without Pernosco.
> I was able to do in hours what would have taken days with any other method
See... I think that's just wrong. Here's how I'd do that (knowing nothing about flex but having written similar state machine generators for non-lexing problems):
+ augment the generated code to dump the sequence of state transitions and tokens to stderr or whatever
+ run it in both environments
+ diff the output
+ Go back and add some metadata like __FILE__/__LINE__ or whatever as needed to find the bad data or state
+ Repeat as needed
Now, is that as button-pushing-trivial as using a replay debugger? No, probably not. But it's not "days" of work either, it's like an hour to get a rig like that put together.
Sure, I’ve done the same thing in the past; I used to use print debugging all the time. I still do, if I can’t use rr or Pernosco.
But I think that you’re underestimating the time that it would take you. Sure, if you knew ahead of time what you would need to log, it might take an hour or even less. In practice though you’re going to end up going through that loop dozens of times, adding some logging to both sets of source code (or removing something that turned out to be useless or confusing), rebuilding, rerunning the test case, and diffing the outputs.
Much better to record everything, and I do mean everything, using rr. Then you can add and remove data from your queries until you understand the bug (or rather, one of the handful of bugs that all existed at the same time) and how to fix it. No changes to the source code needed, no need to recompile them or rerun the test case.
However, I am perfectly willing to suppose that you could do in hours what would have taken me days. There are a lot of people in the world, and some of them are bound to be better engineers than I am. In that case, I suspect that using Pernosco, you could have done it in minutes instead of hours. I think it’s safe enough to say that gives me a straight 5× improvement to my abilities, and it would do the same for you.
Incidentally, roc has talked about adding automatic diffing of recordings to Pernosco. This would be diffing between multiple runs of the same program, rather than between two related programs that happen to be trying to accomplish the same task, but I can well imagine how much time it will save. Imagine those intermittent tests that fail one time out of a hundred. Currently you can rerun the test until you manage to capture the failure in a recording, and that helps to debug the problem quite a lot. But then imagine comparing a successful run to the failed run, so that you know the bug is in one of the differences between them. It’ll give anyone superpowers.
Of course there is some "cap" on the value tooling can provide, but whatever it is, for some people at least it's close to the total time they spend debugging.
I don't think this is what OP was talking about: if your mental model of what the code is going to do is correct, then sure, you can just read the code and find bugs. AFAICT, OP is talking about debugging as updating your incorrect mental model of what the code is doing. If your mental model of the code does not result in the same output as the code itself, it is extremely helpful to dig into the behavior of the program rather than just reading the code again. For which inputs does my mental model of the code break? At what point in the code do things start to diverge from my mental model? Etc.
Exactly. It's mostly a semantics thing. Finding errors statically via code review and analysis (there are fancy tools to assist this process too!) is important, but it's not sufficient as we all know too well. Sometimes you just need to puzzle things out with runtime state.
So yeah, in the text above I'd view "code review" and "static analysis" as distinct tasks from "debugging". And I'm only talking about the latter.
I take it even further. Debuggers have to go and print statements should be converted into log statements where appropriate.
The reality is that you're not going to attach a debugger to a production service. If you are doing things correctly and have separate environments for test/integ/prod you are not going to attach a debugger. Even if you can attach a debugger, the odds of you catching something are pretty small (not talking about trivial things).
So you should: 1) instrument your code and have the proper debug/verbosity log levels for things 2) have a trivial way of altering the log levels 3) have a way to fetch the logs and diagnose what is going on solely based on the logs (and hopefully with something as simple as grep).
This is the way. There are times and places when debuggers can be useful, but in most situations I see them as a crutch that adds more friction instead of accelerating solving the problem
> Many people prefer print debugging over interactive debugging tools.
Then explains that eventually developers come around to debugging. He explains all the advantages debuggers have while simultaneously touting his own tools. Then he blames adoption on the lack of engineering effort put into overcoming teething problems and calls for platform vendors to better support debuggers.
I feel like the author is living in a vacuum and has no idea how most developers work. Watching his demo for Pernosco makes my eyes bleed.
The steps he takes for debugging a crashing Python App are:
1) Reproduce bug and capture with rr
2) Transmit capture to Pernosco
3) Wait Minutes for an Email to arrive.
4) Follow link in Email to Web based Debugger
5) Review Core Call Stack and Assembly for your Python App.
If it gets to the point where I need to review core call stacks or assembly to figure out why my code is crashing then I'm probably just going to rewrite it. For most developers writing line of business Apps or scripts to parse data and shove it in a DB, the errors you encounter don't require inspecting CPU registers or walking through assembly. They boil down to a misunderstanding about an object state or contents that's quickly rectified by runtime inspection which is why print statements work so well.
By the time the author got to step 4 of his Pernosco demo, most developers using Print Debugging would have fixed their bug and been on to the next one.
> [Most bugs] boil down to a misunderstanding about an object state or contents that's quickly rectified by runtime inspection which is why print statements work so well.
This, in particular, is why print debugging won't go away.
Perhaps the bug we chose to demo there is not representative of most bugs, but the underlying bug is in C code and Brendan Gregg, the person who debugged it "for real", debugged it using gdb. The erroneous state in this case is in ncurses, packaged as a system library, so using print debugging here would require building ncurses from source and wedging that into LLVM/Python somehow. So no, I don't believe your claim.
I recognize that there are bugs that exist in downstream packages and some developers take it upon themselves to debug and troubleshoot those occurrences, which we should appreciate. I think we should all recognize that as a special case and not the norm. Most developers, myself included, lack the knowledge or skill to do that.
I would speculate that most developers, once they identify a bug in a downstream package, would instead either go online to find consensus around the bug and report it to the maintainer of that package, or just find an alternative solution.
I have not read anything more dumb than this in a long time. The article fundamentally says, "Print debugging is not as good as magical tools will be if anyone ever makes them." What's annoying is that he's wrong.
Print debugging is awesome and good. It allows one to explore flow and data. It allows intuitive, multi-pronged examination of a program. It can easily be combined with all the tools the programming language offers. It is incredibly easy to use. It is the same 'UI' everywhere – and it is available – in all languages.
I rarely see something this bad make it onto the main page of Hacker News.
> The article fundamentally says, "Print debugging is not as good as magical tools will be if anyone ever makes them."
...except he did make some of those magical tools. This blog post is an advertisement for them.
> Print debugging is awesome and good. It allows one to explore flow and data.
It's the most primitive way to do that, which just happens to be "good enough" in many cases. This hinders the improvement of existing tools. This is also true for debuggers, they have been stagnant for decades. It's probably part of the same trend that pushed UNIX to domination despite being so primitive: It is cheap, portable and already ubiquitous.
I think it’s generally pretty sad when people are plodding along with much worse tools than they could feasibly have.
I’m not sure “walking should go away” but it is not great when people have to walk for hours to go a few miles, in a universe where bicycles also exist.
Can we imagine a future where we're using both a hammer and a screwdriver, depending on situation, without hammer-only and screwdriver-only people yelling at each other?
If only there were some kind of forum where people could debate which situations are most appropriate for screwdriver and which are most appropriate for hammer and respect each others opinions while learning new things.
If I hadn't frequently been given the back of a screw driver and tasked with banging in a nail, I wouldn't be complaining. I just want a $2 hammer on the shelf, is that so much to ask?
Hum... Let's not pretend that the discourse has been dominated for decades by people claiming that print-debuging should go away and that anybody using it is incompetent, and that the reaction was nearly all people just saying "nope, that's stupid, print debug if perfectly fine" without any complaint about anything else.
It's perfectly fine for you to want a step-based debugger. In fact, if there's anything not fine is that you should probably want more than that, but don't even get to see better tools around. It's great that somebody took the time to develop better tools and is trying to sell it... But, let's stop attacking print, because it's a perfectly fine tool.
I agree with the sentiment, but the analogy is not a perfect one. Hammers and screwdrivers are alike in that they are both simple tools that do different jobs. Meanwhile we’re talking about tools that do the same job, but of different capabilities.
Log or printf debugging is like a shovel, RR is like a steam shovel, and Pernosco is almost like the entire railroad logistics system set up by John Stevens for digging out the Panama canal.
He completely rebuilt the rail system so that trains were moving past the shovels in the cuts continuously. Each shovel would simply unload dirt on whatever train car was closest. The trains in the cut moved slowly and never stopped, so there was always a car available. At the fill sites where the dirt was unloaded, he built what was effectively a huge plow system that would scrape the dirt off of the flat–bed cars all in one go (The French just had hundreds of men climb into the cars with shovels to dig the dirt out of the cars, effectively handling every shovel–full twice). The result was like a continuous conveyor belt over 50 miles long. He also built towns, stores, machine shops, water treatment and sewage systems for the towns at either end of the canal, and allowed Gorgas to virtually eliminate mosquitoes from the canal zone.
Pernosco is more like the entire logistics system than the steam shovel, though there are still a few rough edges and some features that I wish it had.
True, but nobody will take your shovel away. The folks building the Panama canal had plenty of shovels to go with the logistics system. I still use print debugging from time to time, if the problem seems simple enough or if rr or Pernosco isn’t available.
I'm 100% in the camp that print statements are terrible for debugging, and I never use them as a debugging tool for my personal projects or on my local box. However here is why, despite something like gdb/rr being the best tool for the job, I fallback to print statements for most code that's shared beyond my personal scope:
- Works in any programming language.
- Works in programming languages that aren't easy to debug with gdb out of the box without setup.
- Works if you're trying to figure out why a problem is occurring without a core dump
- Other developers understand how it works -- I find that there's no reasonable expectation that a developer knows how to use gdb.
- Works on systems you only have log access to -- true for production systems that have sensitive customer data.
So instead of debugging in two different ways, I've submitted myself to print statements -- tragedy of the commons I guess.
You really should try recording with RR; you can play the recording back afterwards, stopping anywhere you want and resuming execution or even running the program backwards to go back to something you missed the first time. You can record it on the server and replay it on your laptop. You can record your tests and send recordings of your test failures to your bug tracker for efficient bug fixing.
My recollection, though, is that RR can only do single-core debugging, and it doesn't help with issues that involve multiple cores stomping on each other in multiple threads.
Sometimes that’s true, though you’ve stated it imprecisely.
RR emulates pthreads in userspace, converting your multithreaded programs to run in a single thread. It does context switching between your threads much as the OS would if you had only one CPU core, but there is no true concurrency. This can hide some problems that are the result of race conditions, making them never show up in any of your recordings. If the problem doesn’t show up in any recordings, then RR isn’t going to help you debug them.
However, RR does have a mitigation for this, called chaos mode. In chaos mode it doesn’t schedule your threads fairly. It randomly starves some threads, not allowing them to be scheduled, or interrupts them before their usual time slice is up. This can make it easier to capture race conditions in a recording, though it isn’t perfect. You should give it a try some time.
I feel like this doesn’t really make a case for why (or how) print debugging should go away. I absolutely buy the argument that debugging tools can do the same job, and that they can improve, but that’s not a reason to avoid prints at all costs.
For me, the biggest hurdle to using a debugger is having to stop and re-learn how to use it. I reach for print sometimes only because I can’t remember the command line for the python debugger off the top of my head, or I can’t remember how to inspect a hex value in gdb. Yes it’s only a quick Google search away, but I can try the print without searching Google. Perhaps irrationally, having a long list of good debuggers I’ve never used like rr and Pernosco and others actually semi-consciously causes me to want to use them less because of the time investment required to figure them out (and repeatedly, like it has in the past with other debuggers.) There are hundreds of tools that I’m not using to their full potential for my job.
Anyway, it’s silly to pretend we can’t have both. I land heavily on the side of using a good debugger whenever I can, and yet I still use prints all the time for quick & easy debugging. It’s silly to suggest that print debugging could ever go away; it can’t unless you want to remove print statements.
One thing both this article and the one on print debugging the other day completely fail to address is debugging embedded systems and GPUs, where both types of debugging are much harder. Try debugging your shader on ShaderToy.com - neither method is available.
With some discipline, you can also make those debug statements bonafide log statements, which you will likely need for production anyway. You can't do this with a debugger (although you can attach a debugger in certain circumstances, it is rare in production, it's not a complete solution, and it may violate compliance).
Absolutely, once a project hits a certain size, I can attest to the value of taking some time to establish a logger with a verbosity knob and command line control. Making it as easy to use as a print statement, but with a default quiet setting, means I can more easily plan and multiply the value of my efforts printing things. Plus, a good logging system works in release builds where a debugger might not.
Print debugger is the debugger which you carry with yourself to anywhere, any infrastructure, any codebase, any language, any organization.
It just works, and it works everywhere. That's why people use it.
Surely, if your work involves working in a certain stack permanently with a certain set of tools, definitely a dedicated debugger could improve your debugging. But for people who work with a large array of technologies and languages, especially senior developers who have to switch back and forth in between stacks - especially if they occupy a lead or man role in their org - that's not feasible. But print just works.
Additionally print debug is the program telling you how things are without something else interacting with it. Leave aside not having to go through creating breakpoints and then going through them slowly one by one (god).
You just place a print in a suspected location and run the program hands off. It doesnt trigger. Ok. You go to the other half of the potentially affected code. Put the print in there. Ha! it triggers. Then you repeat the process again, closing down on affected bit fast. Its a binary search pattern, and its fast. With debugger you have to create a mini program (thought-out breakpoints) to go through the actual execution of the code from the start.
The primary reason I use print debugging because the UI is consistent across all languages and environments. You can literally just learn the hello world program of any language or environment (like the Linux kernel) and you are ready to use print debugging.
Compare this to debuggers, you have to learn the special syntax of the debugger, which is usually very different to the host language, and only then can you start doing some actual debugging.
> If the software industry took debugging seriously — instead of just grumbling about the tools and reverting to print debugging (or, at best, building a polished implementation of the features debuggers have had since the 1980s) — and invested accordingly we could make enormous strides, and not many people would feel the need to resort to print debugging.
This doesn't make any sense, people don't "revert" or "resort" to print debugging, they use print debugging because it is so much simpler compared to debuggers. Print debugging is already as simple as it gets, you can't cut any complexity by adding a new tool.
> Compare this to debuggers, you have to learn the special syntax of the debugger...
In VS Code, you get the basic visual debugger UI that has been around for decades, to observe variables, set breakpoints, move about the stack, and step through the code. This works for pretty much all major languages. It's dead simple and easy to learn. It even works over SSH.
Now, if you insist on doing everything through a terminal, nothing will save you from a miserable debugging experience.
Yeah... every time I tried to use GDB from the terminal, I always felt like I was trying to edit text with ed. Sure, people actually did that, but holy fatcats, even vi would be so much better. You'd think that one of the points of the debugger is to let you easily inspect the program's state, but the default interface is very adept at hiding it.
even so, it can easily be worse than printf debugging, depending what you're trying to investigate. Often those who advocate for always using the debugger say 'well you can just do x to get a printf through the debugger', but that's usually not easy or accessible even in GUI debuggers.
With some languages, you can use "logpoints" in VS Code. To be clear, I have no issue with using print statements, but I also maintain a very short build/run cycles.
I never said debuggers aren't nice. I said that people prefer print debugging because that is a consistent UI across all environments. OP talks about many different debuggers and they all have different UIs. VS Code's debugger is nice, but can you seamlessly plug in rr to get time travel features? What about Pernosco?
Wow, I didn't know that VS Code supports rr as well. I will have to try that, thank you.
I think you misunderstood my point, there are plenty of good GUI debuggers out there. But they don't support all environments. Some IDEs only support C/C++ or Java or something else. The point is that their support is nowhere near as good as print debugging. From webdev to kernel development the API is nearly the same. Even in VS Code sometimes you have to fall back to the terminal to deal with weird environments. It is never really as "seamless" as print debugging.
Surprised nobody has posted this: https://lwn.net/2000/0914/a/lt-debugger.php3. The big problem with debuggers in general is that they encourage you to repair bad stuff rather than to create good stuff. When you reach for a debugger or a print statement, you're essentially saying to the machine, "I don't know how this code works, please tell me" and neither vehicle can really do that; all they can do is tell you what happened in a particular run. On the other hand, reading code can tell you how it works. However, frequently we have trouble reading and understanding which can often be resolved by seeing an output and properly generalizing a conclusion based on what you've read. But, you don't need to see a lot of output to answer a specific question. Odds are good that if you feel you need to see a lot of output then what you're doing is debugging a particular run rather than debugging the program itself.
IMO, there is a big exception here when you want to understand the shape of data in a duck typed language. In that case, dumping an entire object can be useful just to see the shape of you data. This is something you could glean from reading but it generally doesn't change from one run to the next and is frequently left undocumented.
Yes? I think we all agree that the point of a debugger is to repair stuff that's broken, right? Do you actively read and improve the surrounding body of code while you use the debugger? If so, what do you get out of the debugger?
That's not an argument against debuggers, it's an argument in favor of more machine verification e.g. through strict type checking. The expectation that we must understand every single line in a codebase to avoid minor mistakes like null pointers is a bit too much.
I'm definitely a fan of type checkers and formal verification. The problem is that even those tools permit not just specific bugs, but whole classes of bugs through verification; there are inherent limits on what any tool can give you. It's a similar thing with test cases, too.
So, I'd say that when you use a debugger in tandem with a bunch of other good tools then the bad habits that the debugger encourages just matter less. There are fewer classes of bugs that will make it through verification and those other tools force you to understand the code at least well enough that you can pass their checks. However, you still need to read to generalize and fix those remaining bugs properly, and the debugger will push you toward a fast single point resolution rather than a slow but general resolution which is what you'd get from careful reading and thinking.
The big argument against this is I think is if you're working with something where a large number of bugs is acceptable. Like, if you're hacking together a demo/prototype and you just need X to work then a debugger might be the way to go: Just ignore the other bugs because who cares? Or if you're working with some legacy software that's generally known to be buggy and you just need to fix some specific critical use case quickly.
It's also just hard. Like you said, being able to read complex code and then systematically coax the bugs out of it with your brain and eyes definitely takes a lot of practice and the more that that can be supplemented by automation and good habits, the better.
Using a "proper" debugger on a supercomputer tends to be super tricky and the correct setup tends to depend on the machine. And even if you have a debugger attached to all 100k MPI processes, where do you look? And if you try to debug a production run you certainly don't have enough RAM left over for a time travel debugger like rr. The "bad" print debugging still works at that scale. Sure, once you have understood the problem well enough that you can reproduce it on a single node within a couple of minutes a "proper" debugger can be very useful. But you need to get to that point first.
And no core dumps are not a "solves everything solution" either. A) because dumping the entire 1500 TB of memory to disk takes a lot of time and B) because bugs can trash the stack, leading to misleading stack traces.
I am surprised anyone still wants to use print debugging. Certainly it's required sometimes, and sometimes it's the quickest easiest option. I use a mix. But the level of insight a good debugger gives you at a glance is not possible to achieve with print debugging, and I hate when I don't have access to a debugger, it's so cumbersome.
Inside significantly complex programs, plopping a breakpoint and then climbing the recorded stack frames gives you a snapshot of the program at all calls all the way up the stack. It lets me find information in a matter of moments instead of minutes. You can run commands right there in the current moment of the program you paused on, and you can even change values to what you think they should be and let it run to make sure your assumption is correct and your fix will actually work.
Instead of running the program ten times in a row to debug, I do all that in one go, and I can let my curiosity run wild as I poke and prod the program as I see fit.
For programs that take a while to run during development, it's not even feasible to print debug. If you're waiting 1-5 minutes to run the program, and it takes you 5 goes, you've blown nearly half an hour twiddling your thumbs.
Quite frankly if you were on my team and not using decent debuggers to troubleshoot, I'd have a chat. You and I are both on the clock, use your tools and get it done!
Caveat: Some debuggers suck, and print debugging is less painful, but still painful.
Do you ever have to deal with concurrency or external services?
Trying to debug a program which is making an RPC call will likely cause the remote service to timeout and fail by the time you resume program, ruining your experience.
And there is also classical logic/algorithms problems: your program did not crash, but after 50 iterations of the main loop, the result set is empty. With prints, you get it all on screen and stare at the intermediate progress for a few minutes. With debugger, you can set a breakpoint... and then you have to hit "continue" 50 times, trying to remember what the values were 20 steps ago.
You do have to work around concurrency sometimes, it will obviously depend entirely on your stack but for me there is usually way to still get a debugger on the thread I want.
For your second point, logic/algorithmic problems are when the debugger shines the most. If you don't know where the problem is, you can put a breakpoint somewhere you know is after the problem then climb back up the recorded stack frames to find the issue. If the issue is in a loop, you can put in a conditional watcher to only break when your condition is met, or just set a breakpoint one line after the loop and inspect the loop result right there.
The most useful part of debugging is the ability to rapidly excercise your "aha" moments. When you print debug, you see the issue and go "Aha!" now you have to go verify with another print. If I see the issue in a debugger, I go "Aha!" and then immediately get to look at the variable or variables I've discovered is part of the problem.
I want to make sure it's clear that I don't think print debugging is useless, it's got it's moments of usefulness. I just don't understand how someone would not want to take advantage of an advanced debugger when it's available. They are so powerful, it would be like watching a mechanic use a pair of pliers when they have access to an impact gun.
You really should try Pernosco. You can record your program, and replay it as often as you like without ever having to contact the RPC service again; it replays the same response from the server in exactly the same way as it was recorded. The server never times out while you’re paused, because RR remembers the exact pattern of syscalls that your program executed, and always replays them deterministically. The server won’t time out in the play back because it didn’t time out in the recording. It really is a superpower.
And instead of putting a breakpoint and then hitting continue 50 times, you query the Pernosco database for all executions of that line, and it shows them to you as a list. You can then have it evaluate expressions at those points in time, and it shows them in the same list. It’s fast too; it can evaluate your expression at hundreds of points in time faster than you can type them in.
I can give you a pretty useful scenario: tracing a sequence (calls, assignments, whatever). Adding a print statement at critical points will allow you to see what happened and in what order (and possibly the time intervals as well). You can copy and paste that in a document, collect other cases, and think a bit about it offline.
If you would do that inside a debugger, you'd have to make a note when you hit a break point. That's not helpful. It gets worse when you can't really halt the program because there's some kind of real-time effect in your problem.
Yes, you can tell debuggers to print messages instead of halting, but that's just like adding a print statement. IMO, it's even better, because when you make a change, your print statement will still be at the correct point, while your breakpoint might have shifted a line or two...
A debugger is something that has to be developed.
Printing is for the most case already there.
It's not always the case if you like the one or the other. With out of the ordinary issues or places where you can't afford to sacrifice performance but still would like to have some insight printing usually wins by a large margin.
Right, but that's a choice someone has made. You or someone has decided "I am going to build or use tools that can't be debugged".
We should be at a point of maturity in software tooling where that never happens, at least for commercial software companies. I don't much mind what anyone uses in the free time or on their solo SaaS, but as service providers we should be aiming to use mature and effective tooling.
I'm not sure what you mean about performance, since debugging is something you should be doing once off in a local development environment, it shouldn't impact performance for users. Even videogame engines can be debugged. If you've got an issue only happening in production, you should probably try and replicate the production environment so you can debug it locally instead of messing about in prod.
> For programs that take a while to run during development, it's not even feasible to print debug. If you're waiting 1-5 minutes to run the program, and it takes you 5 goes, you've blown nearly half an hour twiddling your thumbs.
For me it's the opposite, a build-compile-run cycle takes 10 seconds, but loading symbols in gdb... I generally can go and make me a coffee, even with gdb index
I'm not sure if the post applies debugging in production systems or only when developing. For production systems it was not always possible to use a debugger, and you need logging there anyway, so finding bugs by looking at the logs has worked well for me. Not sure if that counts as print debugging.
Some reasons why using a debugger in prod was hard:
- knowing where to start. With logs, you get a picture of what has happened.
- not stopping threads
- not everybody can do it, but everybody can look at logs.
If you're in an IDE, though, you can use logpoints rather than breakpoints and have the best of both worlds - "print" debugging without the need to recompile to test (and without having to remember to delete the print statement when you're finished with it).
Even if you're in an IDE, the debugger will often require configuration if the app is not running directly on your local setup. This can range from rather trivial (run app in debug mode, connect to debug port) to rather complex (change the docker image for your app to include a debugger service of some kind).
For example, my Go microservices run in containers built like this:
FROM scratch
COPY --from=build-container /go/bin/the-binary /go/bin/the-binary
So, the final container only contains my Go binary and nothing else. If I add some debug logging, I just rebuild my binary, rebuild the container, and re-deploy.
However, if I want to debug this, I need to use a different container that includes Go delve (the Go debugger) and any dependencies it relies on, possibly open some extra ports, deploy that container instead of the standard one etc.
That’s an interesting point. The way RR works is that it intercepts syscalls, records the execution point of the program that generated the syscall, records the arguments to the syscall, and the kernel’s response. During playback, it intercepts those syscalls, but instead of sending them to the kernel it just supplies the recorded response instead.
It also records several hardware performance counters that were added to Intel CPUs over a decade ago to ensure that it can stop the playback of the program at the correct points reliably. It could also do this purely by single–stepping the program, but that would be horribly slow.
For an embedded system, or for debugging the Linux kernel itself, this doesn’t work. The kernel doesn’t make syscalls, and your embedded system might not even have a kernel to make syscalls to. And your hardware probably doesn’t have the right performance counters; AMD cpus only got equivalent capabilities quite recently.
On the other hand, perhaps you could capture and record interrupts instead of syscalls. Maybe the performance counters on the Cortex–M cpu could be made to work for RR’s purposes (I’m just assuming it has some performance counters; I haven’t checked). But that would take a lot of work, and most people just want to be notified when someone else has completed the work.
Cortex processors have a tracing functionality which allows capturing the entire instruction and data stream of the processor, allowing rr-style debugging. The main problem is this is almost always a very high-bandwidth stream which required dedicated pins on the microcontroller and a specialised (and usually expensive) adapter to collect. I've never actually seen it used, I don't know how many embedded developers actually use it (the only place I've heard tell of it being used is inside ARM). It's possible there's some configuration which only records the minimal trace needed to reconstruct the trace later in a way that is accessible to the software running on the chip (allowing extracting the data without needing a debugger attached), but I've not seen it, and the details of this functionality seem to vary substantially across different devices.
The go-to companies for these devices are Segger and Lauterbach. I'm a Segger user for JTAG debugging, although it's rare that the shops I work in will have a J-TRACE device on the shelf.
J-TRACE is the thing you buy when it's 5 minutes to production and your hair is on fire and you can't figure out what the problem is. Which is why Segger prices it at $2,000.
Yea, that’s how program recording is done natively by GDB as well; it inserts a breakpoint after every single instruction, and records the the state of the program after every one. It’s mind–bogglingly slow, and the log files are gigantic. You generally have to limit it to record only a few instructions, generally tens of thousands rather than billions, in order to get any use out of it.
At least if you can do it in hardware then it doesn’t have to be slow, even if the recorded state is still huge.
In my head, there is this (incomplete) concept that I call "functional debugging" (for a lack of a better term). Current debuggers are oriented very imperatively, i.e. here's what's bound to happen and here you see it happening, step by step.
I would like a debugger that would let me set up some query about the evaluation inside the program, for example, find me all input data in this test run that caused the output of a given function/expression to be such and such (possibly another data set produced during the debugging). Find me what are the typical inputs to the function. And so on.
One of the ideas would be instead of having breakpoints set to a place of execution, they would be set to passed values or program state. This is much more sensible in the world of functional programming, and processing of large amounts of data.
I can imagine a system like that to replace print debugging for me, but until then.. I find it easier to print.
Ill settle for ability to rewind. Whats the difference between console.log(somevariable) and breaking on that variable if the outcome is the same - me rewriting and running again?
Notice how the user clicked on a line number, and a window opened that shows every execution of that line. It’s not a breakpoint where you see only a single point in time; instead you see all the points in time. He then types in an expression, printing the value of the named variable at every single one of those points. Lastly, he types in a condition expression, narrowing the list to examine just the calls for a specific object.
While I agree with author that record-and-replay debuggers are nice, I don't think they will ever entirely go away. Those debuggers are ultimate "trace loggers", logging each statement and variable -- and thus they have inevitable overhead.
I can see how replay debuggers work with programs like Firefox, where you start with a few megabytes of input data and have a working set of a few hundred megs -- but there are programs with much higher memory/io usage out there.
Something as simple as "download a few gigs of data from internet and calculate its hash" will require enough trace to record all data downloaded, which would make the overhead huge.
A program running under RR generally only runs about 40% slower than normal, which is often unnoticeable. The recordings are usually smaller than you would expect too; the largest one I have is 74GB, though that was because the program read in a whole lot of data. I didn’t record it reading in the whole 400+GB file… most of them are under 200MB. Actually, about 60% of them are less than a single megabyte. There must be a power–law thing going on there.
Still, disk space is cheap; I may have to delete that recording one day but there’s plenty more disk space available.
As far as I understand, rr forces your program to run single-threaded while recording, so the performance overhead would be arbitrarily high depending on your degree of actual parallelism.
Not exactly. Your program will still have threads, but RR intercepts the pthreads API and runs all your threads in a single OS thread; it handles context switches between your threads as if your computer only had a single CPU core. This does mean that the overhead can be much higher than 40%! The 40% number is for a program like Firefox, that might jump up to 16 active threads to render a webpage, but also had to idle for 50ms while it waited for the TCP connection to start up.
I found that it was also a reasonable estimate for Reposurgeon, which is written in Go and uses concurrency during some repository operations, and when doing garbage collection. Other parts cannot be parallelized at all, so it balances out. I never measured that very precisely though; it would take anywhere from seconds to minutes to run a test without RR (depending on the data I loaded, etc), and only somewhat longer with RR.
> Those debuggers are ultimate "trace loggers", logging each statement and variable -- and thus they have inevitable overhead.
rr only logs sources of non-determinism, such as system calls (which includes I/O). Then all the deterministic state of the program is reconstructed at runtime using the recorded non-determinism.
So having lots of state doesn't necessarily affect trace size but you're right that lots of I/O will.
I work on servers that are 150 to 200ms away in terms of latency. They're impossible to do interactive debugging on them (well not impossible, but really unpleasant).
Debuggers are not latency aware, they suck at it, they assume the debugged process is close to the UI showing the tooling, close in terms of latency and bandwidth.
Granted, most of the debugging is local, but sometimes replicating the environment so you can reproduce the issue is impossible or at least really time consuming, so remote debugging is a must, and when that happens, print debugging tends to win.
Not going to ever happen because print debugging eventually gets formalized into logging and metrics as a best practice.
My bias here is infrastructure, and you simply can't attach a debugger to a running production process to slow down customers.
At core, debuggers are an educational tool for new developers or inexperienced developers. Experienced developers will add metrics, assertions, and logs to the code which will make print debugging mostly redundant.
I don't want to contribute to the printf vs debugger polarization that seems to be going on. But what I would like to mention:
The first thing I do after `git clone` and getting the build working is spend a couple minutes trying to get a debugger working, preferably using the IDE debugger. Usually it's pretty simple and it works out of the box. Sometimes it requires fiddling a bit with debug ports and cli arguments (legacy Tomcat applications) or making sure sourcemaps are available (web frontend dev).
If I can't get it working within 15 minutes, printf it is. But in most higher level projects it's usually a trivial initial time investment that pays off in the long run.
Debugging is one of the key skills all devs must master. Visual studio has one of the best debuggers I've used and it's still not perfect. That said, the title is still clickbait and print debugging will probably always have its place.
This article is an ad that doesn't even mention the real issues with print debugging:
1. Sometimes the information you can print about an object is not complete due to console memory constraints, depth of the information, dereferencing, formatting, etc. So you will have to come back again and print more ranges of that information, and format the data to understand it.
2. Logging takes "a lot" time if you are working on high performace algorithms. Then it will be dangerous to forget removing a print.
3. Prints can leak security information.
4. In web browser consoles you get a live print of objects, not snapshots.
That said, print is my favorite debugging method, but I constantly search and remove them, specially on release.
Can anyone recommend a youtube video of someone using rr effectively? I've used it a few times, and enjoyed it, but I feel limited by my beginner gdb skills. I think I need to watch someone more experienced.
That’s an excellent idea. Debuggers certainly need better documentation, and guided tours are a good way to teach skills.
Some of the Mozilla folks did livestreams of their development work on Firefox, including debugging sessions. I haven’t watched any lately, but you might search them out. If they kept it up then they might be using RR now, instead of just GDB.
Pernosco is based on RR, but this probably won’t teach you anything about using RR, since the UI is so different. But it was a memorable one because Brendan Gregg produced a really good tutorial for GDB, where he debugged a real problem in a real program (instead of a made–up problem in a program written just to have a made–up problem inserted into it). The Pernosco demo debugs the same problem in the same program, so comparing the two can be quite easy. I would like to see someone the same type of tutorial for RR, and for other debuggers that people like.
The print statement is persisted in your code so you can easily see where you've been with `hg/git diff`.
Using a debugger I find myself repeating the same steps over and over again that I would have done/written once via print debugging.
That and once my system is deployed and I need to debug issues in production I'm relying on logs which are effectively print statements. So why would I want my dev debug and production debug flows to diverge? Doing print debugging gives me experience in what sort of information is useful to log as well as what is loggable.
Setting up a debugging environment is painfully difficult and time consuming that even the most pro-efficient devs (testers/qa) on debugging don't even bother wasting time.
This is the only thing that visual studio does better than any IDE on the market... zero-config debug, even some mature IDE's like jetbrains ones still to this day don't have zero-config debug, it always take some external and obscure configuration to get things right.
Good debug logs have a value beyond software debugging by its developers - they are useful for ops too. More than a few times nginx and postfix debug logs allowed me to see how requests are processed and find a mistake which would be hard to spot just by looking at the config. Debug logs can be enabled for requests from configured IP even on a server with high load if traffic from the IP is low; rr I assume has no such option.
Record-and-replay debuggers are amazing. My best debugging experience is still smalltalk from decades ago. You get an error unwind the code, edit live values, then hit run from any point in your code old or new and it’ll go on from there.
You can even edit your call stack manually, save the stack manually and restart it later on any machine.
RR probably is pretty similar for nodeJS but I haven’t had a chance to use it. Is it a game changer?
I agree (somewhat) that print debugging shouldn't be praised over interactive debugging, but when you write a revolutionary new debugger like rr, please please build a good UI workflow first, and make it work at least on Windows and macOS too.
For instance, from a "time travelling debugger" I would expect some sort of graphical time line view, where I can immediately see where on the timeline I am currently, and jump back and forth while inspecting program state and memory in UI panels.
A good UI is the most important part of a debugger, and the area with the most potential for improvement (Visual Studio is generally praised as the best debugger around, but when you think about it, the UI is archaic, yet it's the best we have).
Without a good debugging UI and intuitive workflows, it's no wonder people are falling back to printf-debugging.
Print debugging is perhaps just a sloppy version of debug logging.
I can't quite tell if the author is arguing against DEBUG/TRACE logging which is the gold standard in debugging, particularly in low reproducibility and multi-threaded/actor-based systems or systems with many 3rd party dependencies.
For those who do use print debugging, and are in Java, here's a trick that is occasionally helpful: When you know you're getting to a function, but you don't know how, create an exception in that function, print the stack trace, and then don't throw the exception.
Print debugging is called debug tracing. And it will never go away. It's not used for debugging mostly, but to analyse traces.
Several systems cannot even use debugging. Such as realtime, kernel or embedded. The only tool they have is debug tracing. Looking at logs. I've worked years only looking at logs, tons of logs from tons of connected real-time systems. Writing tons of log analysis tools.
I love debuggers, but debuggers are a rare luxury to have. In many cases we even added embedded shells (read-eval-print loops, lua, picoc, ...) into embedded systems to query its state. Similar to a debugger, but also completely different.
You should also not forget LED debugging. Sometimes you only have two LED's.
There's a similar war between logging and tracing. Tracing promises to give you everything you ever wanted to do with your logs and then botches it by sampling the data so you can't correlate anymore, or you don't sample and suffocate in data just as with logging but without the benefits.
For debuggers it's probably the same: if they make the assumption that you need less than complete freedom in instrumenting the code, you will need print and if they don't assume that, they likely are harder to use than the language itself.
Well, gdb promised to be able to record/replay too, did not work for me for anything nontrivial, noone bothered with the bugreport for years, left a bitter aftertaste. I'm reluctant to sink more time into trying attempts to fix C debugging.
Even when GDB’s record and replay works, it’s terribly slow. I hope that one day GDB has features equivalent to Pernsoco, but putting a breakpoint after every single instruction is not going to scale.
I hope you’ll give RR a try; it’s been fantastic for me. (rr-project.org)
Print debugging is fine in an environment where the edit/compile/run cycle is fast or doesn't even exist. In fact it's probably the easiest way to debug in such environments.
What I've been thinking about is a code generation tool where I can specify the variable(s) I want to watch and it takes the original code as input and produces code with traces everywhere that variable is referenced or modified. If you're going to do print debugging then let's go all the way!
I hate print debugging. But the C/C++ gods in my workplace, who designed the whole system love it. So they did not care to make it easy for people like me to use debuggers. I am not skilled enough to figure out the exact incantations to run that will put in the correct symbols in the correct place such that gdb will pick them up.
Print lines also comes with the natural flow of the program, so you have the added benefit of fixing something in a strong way with print line debugging.
If you make any wrong assumptions about freezing a program at a certain spot with a debugger ... I’m not sure, I guess I’m super biased about the simplicity of a good sharp kitchen knife.
Print debugging also lets me debug precisely how I want, across all languages that allow display of a value at a particular point in code. I'm not dissing folks who like the debuggers, but I think theres always going to be some folks who prefer one method over the other. "Print Debugging Should Go Away" is awful presumptive in suggesting that the author is an exclusive authority on how other people go about their workday. I mean the guy probably doesn't even know me.
Carrying water from a well to your house in a bucket every day should go away too. Buckets are fine if they’re all you’ve got, but there’s a reason people install plumbing. More than one, I should say.
The difference is that people have proven to be extremely quick to spend time and money on installing plumbing, but these days hardly anyone spends much time improving their editor or other tools. Some companies, like Microsoft, are rich enough to employ dozens or hundreds of engineers on VSCode, and then give it away for free in order to build good–will with a whole generation of programmers. I’m given to understand that Google has teams devoted to developer experience, who spend their time sharping the tools used by Googlers so that most Googlers won’t need to. But most of us don’t spend much time improving our tools, and we mostly can’t buy better ones.
It may be presumptuous, but I agree with Roc. If we spend more time on our tools, then print debugging _will_ go away. Not because we’ve forced you to use terrible tools, but because the tools will be so much better than print debugging that you’ll voluntarily start using them.
sure, but doesn't that sort of require me to believe print debugging is the same as carrying water from a well? I'm happy that debuggers work for folks who like them, and I'm not above using one but for me, the plain old fashioned bandwidth and flexibility I can achieve with logged output just seems to have natural advantages in situations that I seem to find my way into.
I'm not saying that "debuggers should go away" because I'm not going to presume that I know what works in someone elses situation, but it seems like this article is actually doing the exact same thing to me.
Everyone on that "side" seems to be more emphatic that "print debugging should go away" (the very title), yet for the most part, I'm not seeing print debugger types demanding that interactive debuggers should go away.
So there's a disparity in those two attitudes, that implies some advantages on one side or the other, wouldn't it be more productive to list the advantages and let the merits of each approach win out in each situation we programmers find ourselves in?
Not to mention my "vendor paranoia", I mean the guy already is making my agent orange act up by almost titling his article "the way mousepilot works should go away," that isn't exactly the sales pitch thats gonna win me over.
This is not at all what he is saying. Pernosco makes print debugging even better, faster, more precise, and with less room for errors. It also enhances every other aspect of the debugger, by making them omniscient as well, but nobody will force you to use those features. No one will make you click on a value to trace backwards in the program to find out where that value came from (https://pernos.co/about/dataflow/), or to debug multiple processes that were running at the same time (https://pernos.co/about/multiprocess/). You don’t have to use these advanced capabilities if you don’t want to. Perhaps you’ll never care who the callees of a function are (https://pernos.co/about/callees/). Maybe your programs run purely in the terminal, so you never need to click on a pixel to find out who drew it (https://pernos.co/about/screenshots/).
He’s saying that he thinks that once you have the option to do more than just print debugging, you will. Once the debuggers for your favorite languages support these features, you’ll gradually use them more and more, until you hardly ever do “just” print debugging any more. But to get there, someone has to put in the time and effort to make the debuggers for those languages good, not merely acceptable. Or Pernosco has to be extended so that it recognizes the interpreters or runtimes for those languages, so that it can extract the debug info necessary to let you debug your program instead of the interpreter. It can already do this with Chrome and node.js, which both use v8 to run Javascript. (https://pernos.co/about/javascript/)
When Python developers see that, they should sit up and wonder what they can do to have that feature for themselves. They can implement it themselves in their own debuggers, or they can extend GDB to provide extra support for the Python interpreter, so that GDB can show the source code of the Python program instead of the source code of the interpreter. Maybe they could help extend Pernosco too. Same with Erlang, and Scheme, and Go, and Java, and so on.
> doesn't that sort of require me to believe print debugging is the same as carrying water from a well?
Sure. And I get that it’s not as obvious as comparing buckets to pipes. Once you see how a pipe works, and a faucet, it’s pretty hard to deny that they have advantages over buckets in nearly all situations. It’s also hard to argue that buckets have any advantages over pipes at all.
But with debugging it’s harder, because the PHP debuggers are awful, and the Python debuggers are barely adequate, and the Bash debugger barely exists, GDB’s command set is about as tidy as kudzu smothering a stand of trees, and so on. It’s easy to argue that these tools are inadequate, and easy to dismiss all debuggers as a result.
But the sad state of the debuggers for many languages is due to primarily to neglect, not lack of potential. Even the fact that new languages need to write their own debuggers is sad commentary on the human condition.
So yes, print debugging needs to go away, just like poverty and death need to go away. We can solve the problems, we just have to get up and do it.
>Carrying water from a well to your house in a bucket every day should go away too. Buckets are fine if they’re all you’ve got, but there’s a reason people install plumbing. More than one, I should say.
Ok, but I am not going to set up plumbing everywhere I go when a water bottle does the trick.
>If we spend more time on our tools, then print debugging will go away.
No matter how much time we spend on plumbing will make the water bottle go away.
Most of my unit tests have print statements in them, permanently, because I usually know what things I expect will break. There's a ton of utility in being able to understand a problem by reading a transcript.
That's print debugging, and it's not going anywhere as long as unit tests are around.
While I much prefer debuggers, I've never had a print statement "crash" and stop working.
But I've had debugger crashes or weirdness where they stop working in every language and debugger option I've ever used (gdb, lldb, MS debugger, JVM, Rust, V8, etc).
I'm not against using a debugger, however one benefit print debugging is that you can print out intermediate states in a loop and look over all of them at once. Using a debugger would require stepping until you find the wrong iteration.
The saddest command I've ever typed was 'gdb gdb qtcreator', where inside qtcreator I was trying to debug using gdb. qtcreator was crashing when I tried to do that, and gdb was crashing when I tried to debug qtcreator crashing, and eventually I located a bug in gdb (which it turned out had been patched in a later version). It was a massive waste of my time and way off into yak shaving territory that I should have abandoned long before.
Embedded debugging is a pile of woe as well, in my experience. On the current system I'm working on, with 4 different chips using 2 different architectures, 3-4 different JTAG adaptors and different vendors gdb integrations, none of the debugging interfaces work reliably. If you can get the software to start running under the debugger, usually the first breakpoint will get hit correctly. If you're lucky it'll even continue running after you restart it from that breakpoint. The next breakpoint is even less likely to work, let along actually being able to trace out the path of some code. In theory on one of the architectures the interface understands freertos threads, but that just reduces the odds of a successfull breakpoint even lower (and you can forget about support on the other ones).
> It was a massive waste of my time and way off into yak shaving territory that I should have abandoned long before.
I don’t know if I agree with you there. I suppose you did waste your time, since the bug turned out to already be fixed, but every crash has to be investigated by somebody. Otherwise, the wouldn’t get fixed. If they don’t get fixed, they simply waste everyone’s time, over and over again.
The sad statement above was that the commenter saw a crash in a tool, and gave up instead of investigating. Thank you for not giving up when you were in the same situation!
> Embedded debugging is a pile of woe as well
I’ve never done anything of note on embedded platforms, but it certainly sounds like it. You would think that you could pay the vendors for better tools, but it sounds like they all just do the bare minimum, or they just do the bare minimum for anyone who doesn’t pay through the nose.
I didn't. I've a hunch that it's a very deep rabbit hole, which means understanding and fixing those crashes is going to be a significant time investment.
I can't bill a customer for that time, and my boss won't be able to pay my salary if I spend weeks studying and fixing broken old debuggers instead of doing the work a customer pays for.
Either way, the debugging experience without something like rr sucks hard when it comes to heavily multi-threaded, asynchronous, event-driven systems. So even if gdb didn't crash, its usefulness is very limited compared to simple tracing.
"AArch32 support is WONTFIX, as far as we are concerned. Even if implementing rr on 32 bit ARM were possible I don't think anybody cares about it at this point."
I might be spoiled, since I started working on Windows using Visual Studio back in 1996. The debugger always seemed a fundamental part of developing (an learning to develop) applications.
10 years later, I was still being spoiled while building Windows CE OS images. There you had KITL to debug (over serial, ethernet or USB) the OS in real time from the IDE.
When none of that worked, I purchased myself a Lauterbach probe (6K USD). The debugger IDE is the ugliest thing ever, but it's the most powerful tool around for embedded.
Some years later I entered into kernel (Linux) building for embedded. When asked the guru what's there to debug the kernel, the answer I got was "printf". Man, I laughed so hard until I saw his "I'm being serious" face. Then I get used to it, but I still hate it.
Again, some years later, I have to build Android images. Again, skimming through thousands of log messages thanks to dmesg and logcat... I still hate it. It seems archaic.
The better moments of my current work is going bare-metal, debugging with an IDE, with my 15 euro JTAG plus OpenOCD and Eclipse. I couldn't be happier.
In fact, for bare-metal embedded, there is no other way around. There is no printf. You can have it with semihosting or a serial port, but it's painful... it messes up with my timings. I need to debug using an IDE. And GDB from the command line... come on... it's 2021. It can be avoided.
Even when I started with Rust for embedded I wanted a full debugging IDE (which I didn't find yet). I want to see how things get arranged in memory (how does a slice or enum look in memory? and, is it possible no one checked this before?) side-to-side with generated assembler.
TL;DR, my opinion is that printf might be fine for some desktop development, but as a "I don't have any else to try" resource.
For embedded microcontrollers, I don't think there is anything to really replace print debugging. Especially when the program interacts with external electronics components.
If it lets you avoid recompilation, it's very likely it could save you a significant amount of time. I really learned GDB for an embedded Linux system where it took 5-10 minutes to compile and 5-10 more minutes to load the updated build and then you have to restart the whole device. I don't really feel like spending 20 minutes (worst case) or even 10 minutes (best case) to get to, hopefully, the right printf. Make a build with debug symbols, load it, run it with a debugger attached [0], maybe restart a few times to recreate the error path (which might have to be done with printfs as well). And once that process was established and written down (extra build target, half page of instructions on getting things properly configured for our environment) it became much, much faster than print debugging.
Coming from embedded Linux programming, I gotta say sometimes (for easier problems) the overhead in altering the init and spinning up a gdbserver is too much; just pushing a few prints to debug something is easier. However - just having learned about rr, which seems like an amazing tool - I think that can breakpoint might change...
Does "print debugging" include debug logging? If yes, then I think it's absurd to claim that debuggers can replace debug logging entirely.
I for one use both print debugging and actual debuggers (I've used the C# VS debugger, windbg, Java debugging in Eclipse, NetBeans, and IntelliJ, CLI gdb and pdb, Emacs GUD for gdb, pdb and Go delve, Emacs edb and the CCL debugger from SLIME). Sometimes it's awesome to point a debugger at a core dump and fix a year-old memory leak, or just see windbg point out a deadlock involving 5 locks in 2 different processes. Other times it's awesome to interpret debug logs and figure out a how a flow in a customer setup got interleaved and produced a race condition two weeks ago, which is causing a crash today.
You can pry my debug prints from my cold, dead hands.
I get the point this person is making, but at times (and depending on language) it's the fastest and most efficient option. "Oh, the value is not what I expected" -> fix -> "value is good after testing, can remove the debug print". It's not elegant, yes, but it does the job with zero preparation and works for every language. Now if things are so hopeless the debug print doesn't even work, then yes, get a proper debugger.
The fact that you _know_ where or which variable it is, and know where to put the print in order to see the value indicates that the bug is either trivial, or that you already have an idea of where the bug is!
I argue that printf debugging is more difficult if you dont know _what_ is incorrect, or if you dont know where a value is being changed. Stepping through your code is a great way to find out.
To me, actual stepped debugging is like having a print statement for every single line in the code base.
You aren't wrong, but you do need a debugger to be available, and it's not the case with every language, unfortunately.
I argue that printf debugging is more difficult if you dont know _what_ is incorrect
Or even impossible? I use print debugging often, not even only in cases where there's no step debugging possible and/or I'm too lazy to fire up a config/software which allows debugging, but from time to time I encounter bugs which I really can't even begin to imagine how terribly awkward it would be without step debugging. For example: finding a bug in a mark&sweep garbage collector written in C which only occurred in certain LTO builds. To get a picture of the problem I stepped through assembly code while looking at certain memory locations, regsiters, the C stack, and some related variables. How would one tackle that with print debugging? Plus the print statements would affect the very thing being debugged? That should be fun. Not to mention the effort it took to even be able to reach the exact code location where all of this happened. Would almost require to randomly sprinkle print statements all over the place.
If you have _no idea_ where a bug could come from in your code You should spend the time understanding your codebase before you even begin debugging. At the very least you need to be able to reproduce your bug to reliably fix it. How can you have no idea where a bug is happening if you can trigger the error in a code path?
EDIT: I don't want to go so far as to say that your example can never happen, but I can't believe there is any domain where it is the norm
Ever inherited a 500k+ SLOC program with piss poor documentation that you can't rewrite (at least not within the needed timeframe)? Or worked in an office that, on some projects if not overall, experienced high turnover (due to promotions, internal movement, quitting, etc.)? It's very easy to end up with a project in a state where no one actually knows most of the code. A debugger shines here. In fact, the debugger can be a fantastic way to learn the code since the dynamic behavior of a system can often be hard to infer from its static representation (looking at you WPF and XAML, or any project where architecture astronauts went hog wild with certain design patterns).
Changing code that you don't understand because you "found" the bug with a step through debugger is a recipe for disaster. In those situations (high turnover/fast moving tech debt accumulation) perhaps no one cares that the code base is becoming unmaintainable, but it isn't a good practice.
> Changing code that you don't understand because you "found" the bug with a step through debugger is a recipe for disaster.
I don't get your point, because that's not what I suggested someone should do. I think (on rereading my original comment) my point was the exact opposite: Develop an understanding of the code through the effective use of a debugger:
> In fact, the debugger can be a fantastic way to learn the code since the dynamic behavior of a system can often be hard to infer from its static representation (looking at you WPF and XAML, or any project where architecture astronauts went hog wild with certain design patterns).
You seem to have inferred something from my comment that I did not say.
If I did I apologize. I agree that using a debugger can be a good way to understand what a codebase is doing. I think we are on a similar page. Sorry for the miscommunication
Some issues give you roughly zero feedback. If your error doesn't give you a usable stack trace this about the only option although it isn't probably ever the norm. But the key here is that it works with debugging/logging, even when the error is only a low probability issue the only shows up randomly in production. Debuggers are of no use in this case.
Sure. I'm not arguing using a debugger if that is what you prefer. I'm saying that to have no clue where in your code base an issue is cropping up, like _0_ idea, says to me one should spend some time understanding the code before any type of debugging.
Just add more print instructions. And if that doesn't work, add even more print instructions!
Add some instructions to print "**1**", "**2**", "**3**", ... to see how far the program goes, which branches it takes, print a few variables that look interesting. If that fails print the uninteresting variables, remove "**1**" and add some "**2a**", "**2b**", ... in between 2 and 3 ...
After a while, you discover the correct spot and the correct variable, and the error is trivial.
I do this too; but it seems to me this is the case where using a debugging tool would probably save more time.
Looking at the effort involved: here you have to spend the time to add a bunch of print statements throughout bits of code; then compile/run the program (repeating until you've found the right place).
With a debugger: you'd have to take the time to set up the environment and learn how to use it (e.g. set breakpoints in parts where things would be interesting). But then the "look at interesting values" wherever isn't significant additional effort.
Or, you know, just step through with a debugger once. Maybe twice
(I use both)
> it works for every language
This is really the problem. Nobody has spent time and money to make debugging _good_ in the languages you use. Few people pay for good tools any more, and only a tiny fraction of users ever contribute improvements to the open–source tools they use (paid or otherwise). So the average debugger hasn’t gotten better since the 90s.
I do programming in multiple languages too. In those where I can use RR and Pernosco, I use them every time. In those where I can’t, I miss RR and Pernosco every single time.
Yep. The other day I was:
* Debugging an 8 bit microcontroller in-circuit with an IDE, conditional breakpoints, step in/out/over, full stack visibility, and the ability to manually poke at values/memory/io
* Debugging an AWS lambda function with printf and high latency logs
It doesn't matter how hip, how modern, how fast, or how big the environment is. What matters is whether or not the ecosystem cares enough to make debugging nice.
> AWS lambda
Live by the sword, die by the sword.
Now if the sword strikes would only leave a GDB stub...
Why isn't print debugging good? Seems good to me.
As pg said, it's like scrubbing dishes with your fingertips: rarely discussed, highly effective.
Print debugging is sometimes acceptable. Scrubbing dishes with your fingertips totally works, but we all have dishwashers. Dish–washing machines are way better than scrubbing manually, with your fingers or with anything else. Load the dishwasher, then go do literally anything else while the dishes are washed. Even if all you do is go watch television, you will be better off just because you didn’t stand there scrubbing dishes by hand for an hour. More human potential has been wasted by drudgery than by any other cause.
Print debugging is like scrubbing dishes by hand, and neglecting your dishwasher.
Pernsoco is like those commercial dishwashers that blast the dishes with super–heated steam to wash them in a few minutes instead of 45.
I don't have a dishwasher and it isn't because of cost or space. I just don't think they are worth it unless you save up dirty dishes for a few days. If you wash everything right away it's easier to not use the dishwasher, especially with a good sized drying rack for easy reuse of common items.
I suppose I feel the same way with print debugging. Mostly that's all I need and the problems I have which would benefit from better tools are rare enough and diverse enough it isn't worth investing time in them.
Dishwashers are better for the environment. There’s even a new Bosch model with a passive heating element (crazy) for the dry cycle. Perhaps if you live solo and don’t do much cooking you don’t need a dishwasher. But even in that cases I’d bet you’re better off running say a half load once a day.
I've heard that before but I doubt it. The embodied energy in a dishwasher installation will certainly take a while to recoup and it would depend a lot on how one measures hand washing.
Anyway I don't really care about some insignificant "environment" difference - the affect on daily convenience far outweighs it.
Just speaking personally, if I cared enough about efficiency to get a dishwasher, I'd rather instead have a bigger sink and setup a wash and rinse bin for dishes. Its the classic way to preserve water at a campsite when doing lots of dishes.
A tub setup would be more environmentally friendly considering the resources put into a dishwasher these days. Certainly drying via a towel is better then an electric heater.
Modern dishwashers recycle water. I’m not sure a wash/rinse setup actually uses less water. I guess it depends on how big the sink is and how much water you fill it with. The Bosch passive heating element is really neat btw and avoids electric heating.
Sure, if you live alone and don’t cook, then washing dishes by hand won’t take very long. Maybe it’ll take less time than loading the dishwasher. But that’s a fairly uncommon scenario, I think. Most people cook dinner for their family, and then have a whole dishwasher full of dishes to wash every single night.
Interestingly, its good to be uncommon. In my household, the one who cooks does not wash up the plates etc. We don't have a dishwasher. Many hands make light work. We call this "social cleaning".
> Dish–washing machines are way better than scrubbing manually
Oh hell no they aren't. They're terrible at handling anything that's dried on plates.
I've never found a dishwasher that could clean truly dirty plates. NEVER. Even if you run them ten times in a row.
Maybe you haven't tried recently. They’re pretty good now.
That’s true, though leaving plates with food on them to dry out is a bad idea. Run them through the dishwasher sooner rather than later!
Of course I’m as lazy as the next person; there are dishes in my sink right now that could be put in the dishwasher, if only someone would empty the dishwasher of the dishes I washed the other day…
It's mostly fine if you fill the pre-wash slot with detergent.
True, though be aware that not all dishwashers have one. Mine just does a few repeated wash cycles with no detergent, then one with detergent, followed by a rinse and dry cycle. It’s not great, but since it’s not actually broken I can’t really justify tossing it.
Even so, I can only recall one time where it didn’t get a baking dish fully clean. I just left it in the dishwasher and ran another load the next day and it was fine. Then I baked some more enchiladas in it.
The pre-wash detergent "slot" in my dishwasher is a slight depression labelled "Pre-wash" on the flap of the normal-wash detergent dispenser. In other words, just squirt a bit of detergent into the dishwasher (anywhere, it doesn't matter) before running it.
There's a dishwasher in my flat. It's pretty new, large and good quality. It's been collecting dust.
You need to pre-wash everything before loading it anyway, and in the end I still need to scrub and re-wash some of the dishes in the sink.
I'd rather just do it myself from the beginning. Takes less time and effort, and comes out cleaner.
Are you filling the pre-wash slot with detergent?
A new dishwasher doesn't mean a good one. They vary in quality.
Are you using a rinse-aid? If not, you're doing it wrong.
I never pre-wash. I'll manually remove solids when I'm done eating (e.g. with a spoon), but most people do that anyway because it'll be a pain to hand wash if you don't.
Are you filling the pre-wash slot with detergent? Because that's actually pretty vital for them to properly clean dried-up stuff. By not filling that slot the machines are crippled, essentially.
> Scrubbing dishes with your fingertips totally works, but we all have dishwashers.
I don't have a dishwasher. I don't know anyone who has a dishwasher. Heck, I don't even think I've ever seen a dishwasher on any appliance store I've been to. Our apartment was built with space for a dishwasher and a clothes dryer, and I've always wondered why, because nobody I know personally has either of these appliances.
Hmm. I detect some hyperbole in your statement.
There's no hyperbole (except perhaps in the part about appliance stores, it's possible that I have seen a dishwasher in one in the past and I'm just not remembering). I really have never seen anyone using a dishwasher or a clothes dryer, and I really do wonder why was my apartment built with space for both.
I'm really curious where you live, because it sounds highly unlikely for any western country.
I might imagine a Christian or Muslim (fundamentalist) society looking down on appliances that do 'womens work'. And I've spent some months in southafrican townships. Such places have no dishwashers. Or did not have them.
I'm not trying to doxx you, or attempting to get people to post prviate info, but I would love to know your demographic.
> I'm really curious where you live
Southeastern Brazil, in the Rio de Janeiro metropolitan area.
Ah, ok. I didn’t realize such places still existed. Around here, dishwashers are virtually ubiquitous. It’s actually more common for people to not have washing machines for clothes than to not have a dishwasher. People without their own washing machine can go to a laundromat, for example, or to a community–owned washing machine in their apartment complex. Even people who own washing machines will go to a laundromat for some things, like washing quilts or blankets, which require a larger machine.
Meanwhile there are no equivalent places where you can take your dirty dishes to wash.
> Meanwhile there are no equivalent places where you can take your dirty dishes to wash.
Considering that "throwaway packaging and cutlery" is a common solution to this: Order food, the packaging, plastic forks and spoons and napkins included. And seeing that a large demography is trying to live off "less waste", I do see an opportunity here.
A community "diswashing service". It would need solid infra and tech to not immediately turn into a rat-infested-mould-party, but I truly think that a place where e.g. you drop your dishes, cutlery, glasses etc, and can pick them up in the evening filled with a meal, is an opportunity. For example.
That would be interesting, but I suspect that transporting your dishes there and returning to pick them up would be more work than just loading them into a dishwasher. Unless the service were quite close by.
Which reminds me, wasn’t room service once a perk you could get in some apartment buildings? You don’t often find dumbwaiters in apartment buildings any more, for example. I would pay for that if it were available.
I live in france and almost none of my friends own a dishwasher but we all have a washing machine
Really? Wow. Maybe washing machine would have been a better analogy :)
> Muslim (fundamentalist) society looking down on appliances that do 'womens work'.
That's extremely condescending, not to mention plain wrong.
I'm not trying to start an Off-topic religious debate here. Sorry for coming over condescending. If you read "all Muslims" you read far too much in it: the fundamentialist part is important. Nor did I say, nor imply that this goes for anyone who identifies as Muslim. If you read that: I'm truly sorry.
I know from experience that the statement holds true for several Christian fundamentalists and Jewish fundamentalist (not all!)communities. I have only read (but not from experience) about how women (especially those of other sects) were treated in the Chalifate from ISIS and how their "work" was deemed slave-worthy.
> the fundamentialist part is important
It's still outright wrong. Your experience with Christian and Jewish fundamentalists does not apply to Muslims. Please stop with spreading incorrect information about Muslims.
ISIS is a fringe group whom we don't give any semblance of credence to, and even they would not look down at washing machines "because it's a woman's job". This is plain ridiculous.
The secular West continues to group all religions together and marks any religious person as some sort of sheep.
Also, let's stop with this "identifies as" business. A person is either a Muslim or a non-Muslim. A person who does not believe in God cannot "identify as" a Muslim for example. That being said, the religion is open to anyone who wants to accept it.
> A person is either a Muslim or a non-Muslim.
As long as you accept it is their right to define that, and not you!
One might use "identifies as" to avoid the classic response of "ah but x was not really a y because z". Someone who believes themselves to be a Muslim simply is. You can call them a bad Muslim, ignorant, misinformed, misguided etc. But they are what they believe themselves to be.
(I am entirely sympathetic to your original frustration. GP was applying very odd assumptions)
> As long as you accept it is their right to define that, and not you!
That's not how Islam works. That's how it was able to survive against the torrent of corruption that affected Christianity and Judaism. The Quran and Hadiths are preserved so that we're easily able to consult them to determine who is and isn't a Muslim. Note that this isn't gate keeping, Islam is wide open for anyone to accept. However, the tenants are strict enough to survive the passage of time.
> You can call them a bad Muslim, ignorant, misinformed, misguided etc. But they are what they believe themselves to be.
This already happens. I'm talking about the extremes that the secular West has adopted, whereby people can inherit their parents religion, even if they don't believe in it.
> The Quran and Hadiths are preserved so that we're easily able to consult them to determine who is and isn't a Muslim.
And yet those poor benighted souls believe they follow those very same texts that you say would reject them. How is that possible? Because their religious leaders interpret the texts differently, emphasise different aspects.
For a Christian equivalent. The Catholic church can excommunicate someone. They will no longer be Catholic but there is no power that can stop a person who believes themselves to be Christian from being termed that. "Normal" Christians might prefer not to be associated with various people, can call sects Extremist or Fundamentalist etc. We simply cannot claim they are Not Christians.
Not everything is up for interpretation in Islam. For example, homosexuality is prohibited, and no amount of "interpretation" is going to change that. Same goes with drinking wine or intoxicants. There is no valid interpretation of Islam that allows such behaviors.
You'd be surprised to find that said "religious leaders" you're referring to don't know the basics of exegesis. This applies to ISIS for example. I invite you to read the many Hadiths about Khawarij, those immediately reject your premise that all interpretations are equally valid. They're not.
No matter how good the debugger is, it still requires learning a new tool and the interface. As long as a debugger has a gdb-like simple CLI, I will often use it because I know it. But I will also still use print statements when I want to scroll through a log of values, rather than press c <enter> For an indefinite number of times.
Debugging is rather good in elixir (Iex.pry is quite nice). I'll still print debug for a faster through-time experience, since most work in elixir is about transforming data across several steps.
It is, also much much easier to identify a race condition with print debugging than a debugger.
I've frequently wanted RR to debug an issue. Not one of those times could I actually use it (usually because of hardware).
Print debugging is invaluable for actually having any chance to understand what happened in a complex bug in production - especially if you're working on user-run apps, not SaaS.
Debuggers help immensely if you can reproduce the bug reliably and in relatively decent time, but if you can't logging/print debugging is your only hope, since no one is going to run their code in production with RR attached.
Came to comment this. Print debugging makes me so fast at work. Unless you give me an easier and better tool I so no reason to change. Works in every language, exactly how I'd expect, and is exceedingly easy to explain to others.
Yep. You can blow peoples minds in ANY LANGUAGE by using print debugging sometimes. It's amazing, but true. At least it's been that way for the ~25 years I've been doing it.
You think you're here? Let's put a few print statements in there to confirm your execution path is actually what you think it is, and those values are what you expect them to be.
I use prints a lot too, but you must admit:
> You think you're here? Let's put a few print statements in there to confirm your execution path is actually what you think it is, and those values are what you expect them to be.
You could just put a breakpoint there, look at as many values as you want, and not have to re-run it if you think of another.
True, but I don't want to have to exit out of 5 breakpoints each time I run through the scenario. I just want to run it with the print output logging the procession and key variables as a summary of relevant info.
To me that's the key difference. If I know what I'm looking for, prints are great. If I don't know what I'm looking for, breakpoints are a must.
Print debugging makes me fast too, but there is eventually a breaking point (get it) where print debugging isn't getting me the answers I need and I have to switch to using other options like interactive debugging.
Still though, I reach for print debugging first.
There are a few languages where print debugging isn't as fast, but that always requires a ton of tooling. I found that debugging Java using Eclipse or Intellij was faster than printing, as I could introspect state and change values while the process was running. Similarly, some scripting environments I've used have allowed for robust state and flow analytics.
But in a normal environment ? or using C++ ? It's generally quicker (though much dirtier) to find simple bugs using print statements to track down assumptions ; when you're in the weeds with a complicated problem or need to track down a crash, then yes, bring out the big guns. But for normal development, I don't think it's completely necessary.
Debugging in C++, is a bit of a pain sometimes. GDB tools are good paired with an IDE like VSCode, but due to the abstractions in C++, inspecting variables require verbose code - beyond the convenience of debugging in a higher level language like Python or even Java as you suggested.
Eg: tracking the values of a std::vector isn't obvious, since there is std:begin and std::end iterators in the vector namespace and you have to manually dereference the whole range in the debugger.
I don't know any ide which doesn't display an std::vector as an array in their gui
It could be I may be missing a plugin. Which IDE do you use?
Are you under the impression that investing the time to learn to use a purpose-built debugger will be a net time loss?
I used to print-debug a lot, but since I started working with a stack where I use an IntelliJ derivative - that is to say, where the debugger actually works - and I'm never ever going back. It's better by a huge margin.
100% this.
Have looked at, say, the Python debugger as used by PyCharm and thought "well this seems like a bludgeon, I only need to see how <variable> isn't working as expected" and bung in various print() statements in and around the Thing concerned, sprinkling them wider and wider if the bug isn't localised to a function or even the class it's in - I may have supplied naff information to a class when instantiating it, for example.
Debuggers are simply too much of a complicated bludgeon. I can understand a need for debuggers for machine code/assembly programming, or deep system programming when you're hitting the metal, or even low level languages like C.
I've yet to find a need for a debugger for high level languages like Python. Perhaps some day. Until then, I'll plough the time which would be required to Get To Know the debugger, into writing code.
I really don't understand this.
Whenever I try to print debug, I end up missing the mark on my first print statement(s).
If I just set a breakpoint, I can jump back in the stack, or step forwards, so I can easily find the real place the error happens.
Of course I sometimes still print debug - when using languages with bad debuggers, languages where I don't know the tooling, languages with horrible control flow (javascript callbacks), or constrained environments, printing/logging is sometimes the best you can do. But that doesn't mean it's the best there is.
It's kind of funny, but Python is the only language I regular use the debugger--presumably because it's built-in.
I put "if (condition): import pdb; pdb.set_trace()" all the time and almost never use "print".
So, clearly I don't love "print", I just hate debuggers that are annoying to set up.
I suspect that if other languages had such a convenient "drop me into the debugger" statement, more people would use debuggers.
The “debugger” keyword in Javascript is similar. I’ve used it frequently!
A good debugger lets you add print breakpoints, so you can add more print statements at runtime as needed.
You could, but then you're programming in yet another language, at the same time, on one single line, without type-checking, maybe without turing-completeness, functions or expressions.
Agree. Print debug with compartmentalized domain tracing using flow control to print or not print is the only way to debug live life safety embedded systems.
1. Ability to prevent wasting resources on a system that does not need to be debugged. 2. Retain most print debug branches that were useful. 3. See how the internals are functioning on a live system with the issue that has yet to having a lab level proof.
Embedded systems, where there are timing things going on, are an interesting special case. You can't just hit a breakpoint without (in the worst case) causing physical damage.
It's even worse if it's multithreaded as well. If you hit a breakpoint, one thread stops and the others keep going, but they keep going in an environment where one thread is effectively dead (the one at the breakpoint). Everything is invalid from then on, in all threads. So breakpoint debugging is certain to be doomed.
Print debugging is not doomed, but it's challenging. The problem is that the time to print out the message can change the timing, and therefore change what happens. It's not certain to do so, but it's something you always have to keep in the back of your mind: "What if this changed the sequence? What if what I'm seeing in the printouts isn't what I would see if the printouts weren't there?"
I think that print debugs are a great first step. The problem are people who can't use a proper debugger, but that's not the same as saying print debugs are always bad.
I tend to use print debugs in two cases. One is when I'm pretty sure I know what the problem is, and it seems trivial. It's often faster to just confirm & fix that way. The other is when I am not even sure where to start, so I put a few canary prints here and there based on hunches until I start to narrow in mentally on what the issue might be.
Beyond that, it's a proper debugger.
It seems to me like the debugger analogy is that "you should use power tools to make all home repairs".
No, it's quite possible to have a screwdriver in your pencil holder and turn it manually at the moment you need it.
"Low tech", yet quick and precise and no batteries to charge.
> You can pry my debug prints from my cold, dead hands.
I clicked on this about to type exactly that phrase.
Here's why: the CORE PROBLEM of debugging has nothing to do with process, it's the need for the human developer to understand the operation of machine code when faced with a specific set of runtime conditions.
You can't debug by just reading code, because the bug only shows up at runtime. Most people agree with this.
Likewise, you can't debug just by WATCHING a program, for the same reason. You need to be reasoning about the operation of the abstracted code at the same time. And almost always you need to be inspecting and reasoning about the internal states driving the behavior.
Debugging tools, as nice as they might be, can only do the "watching" part. They are a fine way to do it, but you still need to be making the decision, based on careful analysis of the code, on which elements to watch.
So... given that you're staring at code in your editor and thinking about how to extract the needed internal state to validate your predictions... why not just print it?
That is: print/console debugging isn't a hard task you "have to do" in the absence of a fancy debugger, it's exactly the task that you must do if you're going to solve the problem. The fancy debugger, at best, is only saving your the time taken to TYPE the printf() expression. And in practice even that tends to parallelize with the analysis in my experience, as you realize things while typing out their expression.
Pernosco lets you record your program once, then query the resulting database over and over. This is like adding printfs to the program after it has run, any time you have a new idea for what it should have printed out. Here’s a demo:
https://pernos.co/about/expressions/
Notice in this demo how he clicked on a line number, and got back a list of all the times in the program where that line was executed. Not just a single breakpoint with the option to continue, but all of them. Then he queries for the value of a variable at all of those points, and gets all the values printed out simultaneously, as if there had been a printf at that line. Then he filters the list to show only calls made on a specific object, as if the printf had an if statement around it. All without needing to rerun the program, or even be on the same computer as where the recording was captured.
This is what the author means when he says that all debugging tools can be _better_ than printf debugging, if only we put some more time into improving our tools.
Again: that simply saves me the time it takes to add logging to trace the value of that variable. It fails to account for the much larger amount of time needed to understand which variable to trace.
By all means, use fancy tools like replay debuggers (of which Pernosco is only one example) if they make you happy. I'm just saying that (1) there is a hard cap on the value that tooling can provide, because the real limitation is between our ears. And so (2) I'm going to spend the limited learning bandwidth between my ears on things like genuine new technology and not better tools that might provide a mildly incremental improvement to tasks I do already.
Hmm. I see what you’re saying, but I’m not sure that there is a hard cap. Pernosco has allowed me to solve harder problems than I would be able to solve without it. Or perhaps there is a hard cap, but that cap is much higher than you would think. High–quality tools might get you closer to that cap than simpler tools would.
For example, I participated in a project recently to extend Flex so that it could generate both Rust and Go code, in addition to C. Pernosco was an invaluable aid here. I was able to get Flex to generate a C program and also a Rust program, then compare the execution of those two programs. I recorded them both, and opened them each in their own window. Keeping them side–by–side, I could probe the innards of both programs simultaneously. Since both programs were supposed to implement the same state machine (and thus recognize the same list of tokens), it was easy to find places where the Rust version went off the rails. I was able to do in hours what would have taken days with any other method, if I had succeeded at all. The behavior of that state machine is not at all obvious, and I might never have unraveled it without Pernosco.
> I was able to do in hours what would have taken days with any other method
See... I think that's just wrong. Here's how I'd do that (knowing nothing about flex but having written similar state machine generators for non-lexing problems):
Now, is that as button-pushing-trivial as using a replay debugger? No, probably not. But it's not "days" of work either, it's like an hour to get a rig like that put together.
Sure, I’ve done the same thing in the past; I used to use print debugging all the time. I still do, if I can’t use rr or Pernosco.
But I think that you’re underestimating the time that it would take you. Sure, if you knew ahead of time what you would need to log, it might take an hour or even less. In practice though you’re going to end up going through that loop dozens of times, adding some logging to both sets of source code (or removing something that turned out to be useless or confusing), rebuilding, rerunning the test case, and diffing the outputs.
Much better to record everything, and I do mean everything, using rr. Then you can add and remove data from your queries until you understand the bug (or rather, one of the handful of bugs that all existed at the same time) and how to fix it. No changes to the source code needed, no need to recompile them or rerun the test case.
However, I am perfectly willing to suppose that you could do in hours what would have taken me days. There are a lot of people in the world, and some of them are bound to be better engineers than I am. In that case, I suspect that using Pernosco, you could have done it in minutes instead of hours. I think it’s safe enough to say that gives me a straight 5× improvement to my abilities, and it would do the same for you.
Incidentally, roc has talked about adding automatic diffing of recordings to Pernosco. This would be diffing between multiple runs of the same program, rather than between two related programs that happen to be trying to accomplish the same task, but I can well imagine how much time it will save. Imagine those intermittent tests that fail one time out of a hundred. Currently you can rerun the test until you manage to capture the failure in a recording, and that helps to debug the problem quite a lot. But then imagine comparing a successful run to the failed run, so that you know the bug is in one of the differences between them. It’ll give anyone superpowers.
Some of our customers report regular 5x speedups in debugging time. https://twitter.com/rocallahan/status/1371607271826878464
Of course there is some "cap" on the value tooling can provide, but whatever it is, for some people at least it's close to the total time they spend debugging.
that tweet was not helpful at all since it isn't available for viewing lol
> You can't debug by just reading code, because the bug only shows up at runtime. Most people agree with this.
Bullshit. I regularly find bugs in code review, just by reading code. I think this is true of most experienced programmers.
I don't think this is what OP was talking about: if your mental model of what the code is going to do is correct, then sure, you can just read the code and find bugs. AFAICT, OP is talking about debugging as updating your incorrect mental model of what the code is doing. If your mental model of the code does not result in the same output as the code itself, it is extremely helpful to dig into the behavior of the program rather than just reading the code again. For which inputs does my mental model of the code break? At what point in the code do things start to diverge from my mental model? Etc.
Exactly. It's mostly a semantics thing. Finding errors statically via code review and analysis (there are fancy tools to assist this process too!) is important, but it's not sufficient as we all know too well. Sometimes you just need to puzzle things out with runtime state.
So yeah, in the text above I'd view "code review" and "static analysis" as distinct tasks from "debugging". And I'm only talking about the latter.
Heh. Aaron Patterson (aka tenderlove), core Ruby and Rails contributor, agrees with you. Although maybe not as forcefully :)
https://tenderlovemaking.com/2016/02/05/i-am-a-puts-debugger...
I take it even further. Debuggers have to go and print statements should be converted into log statements where appropriate.
The reality is that you're not going to attach a debugger to a production service. If you are doing things correctly and have separate environments for test/integ/prod you are not going to attach a debugger. Even if you can attach a debugger, the odds of you catching something are pretty small (not talking about trivial things).
So you should: 1) instrument your code and have the proper debug/verbosity log levels for things 2) have a trivial way of altering the log levels 3) have a way to fetch the logs and diagnose what is going on solely based on the logs (and hopefully with something as simple as grep).
This is the way. There are times and places when debuggers can be useful, but in most situations I see them as a crutch that adds more friction instead of accelerating solving the problem
The author starts out with a a simple statement:
> Many people prefer print debugging over interactive debugging tools.
Then explains that eventually developers come around to debugging. He explains all the advantages debuggers have while simultaneously touting his own tools. Then he blames adoption on the lack of engineering effort put into overcoming teething problems and calls for platform vendors to better support debuggers.
I feel like the author is living in a vacuum and has no idea how most developers work. Watching his demo for Pernosco makes my eyes bleed.
The steps he takes for debugging a crashing Python App are:
If it gets to the point where I need to review core call stacks or assembly to figure out why my code is crashing then I'm probably just going to rewrite it. For most developers writing line of business Apps or scripts to parse data and shove it in a DB, the errors you encounter don't require inspecting CPU registers or walking through assembly. They boil down to a misunderstanding about an object state or contents that's quickly rectified by runtime inspection which is why print statements work so well.
By the time the author got to step 4 of his Pernosco demo, most developers using Print Debugging would have fixed their bug and been on to the next one.
> [Most bugs] boil down to a misunderstanding about an object state or contents that's quickly rectified by runtime inspection which is why print statements work so well.
This, in particular, is why print debugging won't go away.
Perhaps the bug we chose to demo there is not representative of most bugs, but the underlying bug is in C code and Brendan Gregg, the person who debugged it "for real", debugged it using gdb. The erroneous state in this case is in ncurses, packaged as a system library, so using print debugging here would require building ncurses from source and wedging that into LLVM/Python somehow. So no, I don't believe your claim.
I recognize that there are bugs that exist in downstream packages and some developers take it upon themselves to debug and troubleshoot those occurrences, which we should appreciate. I think we should all recognize that as a special case and not the norm. Most developers, myself included, lack the knowledge or skill to do that.
I would speculate that most developers, once they identify a bug in a downstream package, would instead either go online to find consensus around the bug and report it to the maintainer of that package, or just find an alternative solution.
ok now I have to watch this demo. thats crazy! i get a sense that i would hate pernosco as a vendor.
I have not read anything more dumb than this in a long time. The article fundamentally says, "Print debugging is not as good as magical tools will be if anyone ever makes them." What's annoying is that he's wrong.
Print debugging is awesome and good. It allows one to explore flow and data. It allows intuitive, multi-pronged examination of a program. It can easily be combined with all the tools the programming language offers. It is incredibly easy to use. It is the same 'UI' everywhere – and it is available – in all languages.
I rarely see something this bad make it onto the main page of Hacker News.
> The article fundamentally says, "Print debugging is not as good as magical tools will be if anyone ever makes them."
...except he did make some of those magical tools. This blog post is an advertisement for them.
> Print debugging is awesome and good. It allows one to explore flow and data.
It's the most primitive way to do that, which just happens to be "good enough" in many cases. This hinders the improvement of existing tools. This is also true for debuggers, they have been stagnant for decades. It's probably part of the same trend that pushed UNIX to domination despite being so primitive: It is cheap, portable and already ubiquitous.
So bad, we need occasional reminders to re-examine our debugging methodology.
I think it’s generally pretty sad when people are plodding along with much worse tools than they could feasibly have.
I’m not sure “walking should go away” but it is not great when people have to walk for hours to go a few miles, in a universe where bicycles also exist.
Also print debugging adds zero dependencies.
Can we imagine a future where we're using both a hammer and a screwdriver, depending on situation, without hammer-only and screwdriver-only people yelling at each other?
If only there were some kind of forum where people could debate which situations are most appropriate for screwdriver and which are most appropriate for hammer and respect each others opinions while learning new things.
> and respect each others opinions while learning new things.
So basically, this hypothetical forum is not on the internet.
If I hadn't frequently been given the back of a screw driver and tasked with banging in a nail, I wouldn't be complaining. I just want a $2 hammer on the shelf, is that so much to ask?
Hum... Let's not pretend that the discourse has been dominated for decades by people claiming that print-debuging should go away and that anybody using it is incompetent, and that the reaction was nearly all people just saying "nope, that's stupid, print debug if perfectly fine" without any complaint about anything else.
It's perfectly fine for you to want a step-based debugger. In fact, if there's anything not fine is that you should probably want more than that, but don't even get to see better tools around. It's great that somebody took the time to develop better tools and is trying to sell it... But, let's stop attacking print, because it's a perfectly fine tool.
I agree with the sentiment, but the analogy is not a perfect one. Hammers and screwdrivers are alike in that they are both simple tools that do different jobs. Meanwhile we’re talking about tools that do the same job, but of different capabilities.
Log or printf debugging is like a shovel, RR is like a steam shovel, and Pernosco is almost like the entire railroad logistics system set up by John Stevens for digging out the Panama canal.
He completely rebuilt the rail system so that trains were moving past the shovels in the cuts continuously. Each shovel would simply unload dirt on whatever train car was closest. The trains in the cut moved slowly and never stopped, so there was always a car available. At the fill sites where the dirt was unloaded, he built what was effectively a huge plow system that would scrape the dirt off of the flat–bed cars all in one go (The French just had hundreds of men climb into the cars with shovels to dig the dirt out of the cars, effectively handling every shovel–full twice). The result was like a continuous conveyor belt over 50 miles long. He also built towns, stores, machine shops, water treatment and sewage systems for the towns at either end of the canal, and allowed Gorgas to virtually eliminate mosquitoes from the canal zone.
Pernosco is more like the entire logistics system than the steam shovel, though there are still a few rough edges and some features that I wish it had.
> Pernosco is more like the entire logistics system than the steam shovel
Sometimes you don't need the entire railroad logistics system if you're only removing a single shrub.
True, but nobody will take your shovel away. The folks building the Panama canal had plenty of shovels to go with the logistics system. I still use print debugging from time to time, if the problem seems simple enough or if rr or Pernosco isn’t available.
That's crazy talk.
I'm 100% in the camp that print statements are terrible for debugging, and I never use them as a debugging tool for my personal projects or on my local box. However here is why, despite something like gdb/rr being the best tool for the job, I fallback to print statements for most code that's shared beyond my personal scope:
- Works in any programming language.
- Works in programming languages that aren't easy to debug with gdb out of the box without setup.
- Works if you're trying to figure out why a problem is occurring without a core dump
- Other developers understand how it works -- I find that there's no reasonable expectation that a developer knows how to use gdb.
- Works on systems you only have log access to -- true for production systems that have sensitive customer data.
So instead of debugging in two different ways, I've submitted myself to print statements -- tragedy of the commons I guess.
Add one big thing: log debugging is convenient when debugging something with concurrency. When stopping on a breakpoint isn't an option.
You really should try recording with RR; you can play the recording back afterwards, stopping anywhere you want and resuming execution or even running the program backwards to go back to something you missed the first time. You can record it on the server and replay it on your laptop. You can record your tests and send recordings of your test failures to your bug tracker for efficient bug fixing.
My recollection, though, is that RR can only do single-core debugging, and it doesn't help with issues that involve multiple cores stomping on each other in multiple threads.
Sometimes that’s true, though you’ve stated it imprecisely.
RR emulates pthreads in userspace, converting your multithreaded programs to run in a single thread. It does context switching between your threads much as the OS would if you had only one CPU core, but there is no true concurrency. This can hide some problems that are the result of race conditions, making them never show up in any of your recordings. If the problem doesn’t show up in any recordings, then RR isn’t going to help you debug them.
However, RR does have a mitigation for this, called chaos mode. In chaos mode it doesn’t schedule your threads fairly. It randomly starves some threads, not allowing them to be scheduled, or interrupts them before their usual time slice is up. This can make it easier to capture race conditions in a recording, though it isn’t perfect. You should give it a try some time.
I feel like this doesn’t really make a case for why (or how) print debugging should go away. I absolutely buy the argument that debugging tools can do the same job, and that they can improve, but that’s not a reason to avoid prints at all costs.
For me, the biggest hurdle to using a debugger is having to stop and re-learn how to use it. I reach for print sometimes only because I can’t remember the command line for the python debugger off the top of my head, or I can’t remember how to inspect a hex value in gdb. Yes it’s only a quick Google search away, but I can try the print without searching Google. Perhaps irrationally, having a long list of good debuggers I’ve never used like rr and Pernosco and others actually semi-consciously causes me to want to use them less because of the time investment required to figure them out (and repeatedly, like it has in the past with other debuggers.) There are hundreds of tools that I’m not using to their full potential for my job.
Anyway, it’s silly to pretend we can’t have both. I land heavily on the side of using a good debugger whenever I can, and yet I still use prints all the time for quick & easy debugging. It’s silly to suggest that print debugging could ever go away; it can’t unless you want to remove print statements.
One thing both this article and the one on print debugging the other day completely fail to address is debugging embedded systems and GPUs, where both types of debugging are much harder. Try debugging your shader on ShaderToy.com - neither method is available.
With some discipline, you can also make those debug statements bonafide log statements, which you will likely need for production anyway. You can't do this with a debugger (although you can attach a debugger in certain circumstances, it is rare in production, it's not a complete solution, and it may violate compliance).
Absolutely, once a project hits a certain size, I can attest to the value of taking some time to establish a logger with a verbosity knob and command line control. Making it as easy to use as a print statement, but with a default quiet setting, means I can more easily plan and multiply the value of my efforts printing things. Plus, a good logging system works in release builds where a debugger might not.
Print debugger is the debugger which you carry with yourself to anywhere, any infrastructure, any codebase, any language, any organization.
It just works, and it works everywhere. That's why people use it.
Surely, if your work involves working in a certain stack permanently with a certain set of tools, definitely a dedicated debugger could improve your debugging. But for people who work with a large array of technologies and languages, especially senior developers who have to switch back and forth in between stacks - especially if they occupy a lead or man role in their org - that's not feasible. But print just works.
Additionally print debug is the program telling you how things are without something else interacting with it. Leave aside not having to go through creating breakpoints and then going through them slowly one by one (god).
You just place a print in a suspected location and run the program hands off. It doesnt trigger. Ok. You go to the other half of the potentially affected code. Put the print in there. Ha! it triggers. Then you repeat the process again, closing down on affected bit fast. Its a binary search pattern, and its fast. With debugger you have to create a mini program (thought-out breakpoints) to go through the actual execution of the code from the start.
The primary reason I use print debugging because the UI is consistent across all languages and environments. You can literally just learn the hello world program of any language or environment (like the Linux kernel) and you are ready to use print debugging.
Compare this to debuggers, you have to learn the special syntax of the debugger, which is usually very different to the host language, and only then can you start doing some actual debugging.
> If the software industry took debugging seriously — instead of just grumbling about the tools and reverting to print debugging (or, at best, building a polished implementation of the features debuggers have had since the 1980s) — and invested accordingly we could make enormous strides, and not many people would feel the need to resort to print debugging.
This doesn't make any sense, people don't "revert" or "resort" to print debugging, they use print debugging because it is so much simpler compared to debuggers. Print debugging is already as simple as it gets, you can't cut any complexity by adding a new tool.
> Compare this to debuggers, you have to learn the special syntax of the debugger...
In VS Code, you get the basic visual debugger UI that has been around for decades, to observe variables, set breakpoints, move about the stack, and step through the code. This works for pretty much all major languages. It's dead simple and easy to learn. It even works over SSH.
Now, if you insist on doing everything through a terminal, nothing will save you from a miserable debugging experience.
Yeah... every time I tried to use GDB from the terminal, I always felt like I was trying to edit text with ed. Sure, people actually did that, but holy fatcats, even vi would be so much better. You'd think that one of the points of the debugger is to let you easily inspect the program's state, but the default interface is very adept at hiding it.
The secret to using GDB is to type “layout split” or “layout asm” as your first command.
even so, it can easily be worse than printf debugging, depending what you're trying to investigate. Often those who advocate for always using the debugger say 'well you can just do x to get a printf through the debugger', but that's usually not easy or accessible even in GUI debuggers.
With some languages, you can use "logpoints" in VS Code. To be clear, I have no issue with using print statements, but I also maintain a very short build/run cycles.
I never said debuggers aren't nice. I said that people prefer print debugging because that is a consistent UI across all environments. OP talks about many different debuggers and they all have different UIs. VS Code's debugger is nice, but can you seamlessly plug in rr to get time travel features? What about Pernosco?
> I never said debuggers aren't nice. I said that people prefer print debugging because that is a consistent UI across all environments.
You said debuggers require learning "some special syntax", which is not true. This led me to believe that you weren't aware of things I wrote.
> VS Code's debugger is nice, but can you seamlessly plug in rr to get time travel features?
I haven't tried that, but apparently it should work:
https://github.com/rr-debugger/rr/wiki/Using-rr-in-an-IDE
> What about Pernosco?
The way I understand what Pernosco is (a recording), I would expect that to work as well.
Wow, I didn't know that VS Code supports rr as well. I will have to try that, thank you.
I think you misunderstood my point, there are plenty of good GUI debuggers out there. But they don't support all environments. Some IDEs only support C/C++ or Java or something else. The point is that their support is nowhere near as good as print debugging. From webdev to kernel development the API is nearly the same. Even in VS Code sometimes you have to fall back to the terminal to deal with weird environments. It is never really as "seamless" as print debugging.
Surprised nobody has posted this: https://lwn.net/2000/0914/a/lt-debugger.php3. The big problem with debuggers in general is that they encourage you to repair bad stuff rather than to create good stuff. When you reach for a debugger or a print statement, you're essentially saying to the machine, "I don't know how this code works, please tell me" and neither vehicle can really do that; all they can do is tell you what happened in a particular run. On the other hand, reading code can tell you how it works. However, frequently we have trouble reading and understanding which can often be resolved by seeing an output and properly generalizing a conclusion based on what you've read. But, you don't need to see a lot of output to answer a specific question. Odds are good that if you feel you need to see a lot of output then what you're doing is debugging a particular run rather than debugging the program itself.
IMO, there is a big exception here when you want to understand the shape of data in a duck typed language. In that case, dumping an entire object can be useful just to see the shape of you data. This is something you could glean from reading but it generally doesn't change from one run to the next and is frequently left undocumented.
> The big problem with debuggers in general is that they encourage you to repair bad stuff rather than to create good stuff.
Is that verifiable true?
Yes? I think we all agree that the point of a debugger is to repair stuff that's broken, right? Do you actively read and improve the surrounding body of code while you use the debugger? If so, what do you get out of the debugger?
That's not an argument against debuggers, it's an argument in favor of more machine verification e.g. through strict type checking. The expectation that we must understand every single line in a codebase to avoid minor mistakes like null pointers is a bit too much.
I'm definitely a fan of type checkers and formal verification. The problem is that even those tools permit not just specific bugs, but whole classes of bugs through verification; there are inherent limits on what any tool can give you. It's a similar thing with test cases, too.
So, I'd say that when you use a debugger in tandem with a bunch of other good tools then the bad habits that the debugger encourages just matter less. There are fewer classes of bugs that will make it through verification and those other tools force you to understand the code at least well enough that you can pass their checks. However, you still need to read to generalize and fix those remaining bugs properly, and the debugger will push you toward a fast single point resolution rather than a slow but general resolution which is what you'd get from careful reading and thinking.
The big argument against this is I think is if you're working with something where a large number of bugs is acceptable. Like, if you're hacking together a demo/prototype and you just need X to work then a debugger might be the way to go: Just ignore the other bugs because who cares? Or if you're working with some legacy software that's generally known to be buggy and you just need to fix some specific critical use case quickly.
It's also just hard. Like you said, being able to read complex code and then systematically coax the bugs out of it with your brain and eyes definitely takes a lot of practice and the more that that can be supplemented by automation and good habits, the better.
Using a "proper" debugger on a supercomputer tends to be super tricky and the correct setup tends to depend on the machine. And even if you have a debugger attached to all 100k MPI processes, where do you look? And if you try to debug a production run you certainly don't have enough RAM left over for a time travel debugger like rr. The "bad" print debugging still works at that scale. Sure, once you have understood the problem well enough that you can reproduce it on a single node within a couple of minutes a "proper" debugger can be very useful. But you need to get to that point first.
And no core dumps are not a "solves everything solution" either. A) because dumping the entire 1500 TB of memory to disk takes a lot of time and B) because bugs can trash the stack, leading to misleading stack traces.
I am surprised anyone still wants to use print debugging. Certainly it's required sometimes, and sometimes it's the quickest easiest option. I use a mix. But the level of insight a good debugger gives you at a glance is not possible to achieve with print debugging, and I hate when I don't have access to a debugger, it's so cumbersome.
Inside significantly complex programs, plopping a breakpoint and then climbing the recorded stack frames gives you a snapshot of the program at all calls all the way up the stack. It lets me find information in a matter of moments instead of minutes. You can run commands right there in the current moment of the program you paused on, and you can even change values to what you think they should be and let it run to make sure your assumption is correct and your fix will actually work.
Instead of running the program ten times in a row to debug, I do all that in one go, and I can let my curiosity run wild as I poke and prod the program as I see fit.
For programs that take a while to run during development, it's not even feasible to print debug. If you're waiting 1-5 minutes to run the program, and it takes you 5 goes, you've blown nearly half an hour twiddling your thumbs.
Quite frankly if you were on my team and not using decent debuggers to troubleshoot, I'd have a chat. You and I are both on the clock, use your tools and get it done!
Caveat: Some debuggers suck, and print debugging is less painful, but still painful.
Do you ever have to deal with concurrency or external services? Trying to debug a program which is making an RPC call will likely cause the remote service to timeout and fail by the time you resume program, ruining your experience.
And there is also classical logic/algorithms problems: your program did not crash, but after 50 iterations of the main loop, the result set is empty. With prints, you get it all on screen and stare at the intermediate progress for a few minutes. With debugger, you can set a breakpoint... and then you have to hit "continue" 50 times, trying to remember what the values were 20 steps ago.
You do have to work around concurrency sometimes, it will obviously depend entirely on your stack but for me there is usually way to still get a debugger on the thread I want.
For your second point, logic/algorithmic problems are when the debugger shines the most. If you don't know where the problem is, you can put a breakpoint somewhere you know is after the problem then climb back up the recorded stack frames to find the issue. If the issue is in a loop, you can put in a conditional watcher to only break when your condition is met, or just set a breakpoint one line after the loop and inspect the loop result right there.
The most useful part of debugging is the ability to rapidly excercise your "aha" moments. When you print debug, you see the issue and go "Aha!" now you have to go verify with another print. If I see the issue in a debugger, I go "Aha!" and then immediately get to look at the variable or variables I've discovered is part of the problem.
I want to make sure it's clear that I don't think print debugging is useless, it's got it's moments of usefulness. I just don't understand how someone would not want to take advantage of an advanced debugger when it's available. They are so powerful, it would be like watching a mechanic use a pair of pliers when they have access to an impact gun.
You really should try Pernosco. You can record your program, and replay it as often as you like without ever having to contact the RPC service again; it replays the same response from the server in exactly the same way as it was recorded. The server never times out while you’re paused, because RR remembers the exact pattern of syscalls that your program executed, and always replays them deterministically. The server won’t time out in the play back because it didn’t time out in the recording. It really is a superpower.
And instead of putting a breakpoint and then hitting continue 50 times, you query the Pernosco database for all executions of that line, and it shows them to you as a list. You can then have it evaluate expressions at those points in time, and it shows them in the same list. It’s fast too; it can evaluate your expression at hundreds of points in time faster than you can type them in.
You hit the nail on the head.
> when I don't have access to a debugger
Why can't we get rid of print debugging?
Because we don't always have access to a debugger.
I can give you a pretty useful scenario: tracing a sequence (calls, assignments, whatever). Adding a print statement at critical points will allow you to see what happened and in what order (and possibly the time intervals as well). You can copy and paste that in a document, collect other cases, and think a bit about it offline.
If you would do that inside a debugger, you'd have to make a note when you hit a break point. That's not helpful. It gets worse when you can't really halt the program because there's some kind of real-time effect in your problem.
Yes, you can tell debuggers to print messages instead of halting, but that's just like adding a print statement. IMO, it's even better, because when you make a change, your print statement will still be at the correct point, while your breakpoint might have shifted a line or two...
I think you might underestimate an advanced debugger. In the one I most commonly use I can collate data from different frames with watchers.
I haven't personally used it, but I think the debugger in the suggested article even keeps a queryable database of data for every recorded execution.
I hear you and others though, I wasn't personally suggesting print debugging should go away, I still use it here and there.
A debugger is something that has to be developed. Printing is for the most case already there.
It's not always the case if you like the one or the other. With out of the ordinary issues or places where you can't afford to sacrifice performance but still would like to have some insight printing usually wins by a large margin.
Right, but that's a choice someone has made. You or someone has decided "I am going to build or use tools that can't be debugged".
We should be at a point of maturity in software tooling where that never happens, at least for commercial software companies. I don't much mind what anyone uses in the free time or on their solo SaaS, but as service providers we should be aiming to use mature and effective tooling.
I'm not sure what you mean about performance, since debugging is something you should be doing once off in a local development environment, it shouldn't impact performance for users. Even videogame engines can be debugged. If you've got an issue only happening in production, you should probably try and replicate the production environment so you can debug it locally instead of messing about in prod.
> For programs that take a while to run during development, it's not even feasible to print debug. If you're waiting 1-5 minutes to run the program, and it takes you 5 goes, you've blown nearly half an hour twiddling your thumbs.
For me it's the opposite, a build-compile-run cycle takes 10 seconds, but loading symbols in gdb... I generally can go and make me a coffee, even with gdb index
I'm not sure if the post applies debugging in production systems or only when developing. For production systems it was not always possible to use a debugger, and you need logging there anyway, so finding bugs by looking at the logs has worked well for me. Not sure if that counts as print debugging.
Some reasons why using a debugger in prod was hard:
- knowing where to start. With logs, you get a picture of what has happened.
- not stopping threads
- not everybody can do it, but everybody can look at logs.
Wrote more about it here: https://henrikwarne.com/2014/01/01/finding-bugs-debugger-ver... -
Print debugging requires zero configuration. So, for "easy" debugging sessions, I will use print debugging (it does work using plain Vim!).
For more serious debugging, sure, a proper debugger is the way to go.
If you're in an IDE, though, you can use logpoints rather than breakpoints and have the best of both worlds - "print" debugging without the need to recompile to test (and without having to remember to delete the print statement when you're finished with it).
Even if you're in an IDE, the debugger will often require configuration if the app is not running directly on your local setup. This can range from rather trivial (run app in debug mode, connect to debug port) to rather complex (change the docker image for your app to include a debugger service of some kind).
Ok, but isn't that true of print debugging too? Wherever the application is running, you have to recompile and redeploy it.
Sure, but I'm talking about more than that.
For example, my Go microservices run in containers built like this:
So, the final container only contains my Go binary and nothing else. If I add some debug logging, I just rebuild my binary, rebuild the container, and re-deploy.
However, if I want to debug this, I need to use a different container that includes Go delve (the Go debugger) and any dependencies it relies on, possibly open some extra ports, deploy that container instead of the standard one etc.
Pernosco currently targets C, C++, Ada, Rust and V8 JS (e.g. node.js) applications that run on x86 Linux and work with rr.
Cortex-M developer here. Let me know when you support bare metal.
That’s an interesting point. The way RR works is that it intercepts syscalls, records the execution point of the program that generated the syscall, records the arguments to the syscall, and the kernel’s response. During playback, it intercepts those syscalls, but instead of sending them to the kernel it just supplies the recorded response instead.
It also records several hardware performance counters that were added to Intel CPUs over a decade ago to ensure that it can stop the playback of the program at the correct points reliably. It could also do this purely by single–stepping the program, but that would be horribly slow.
For an embedded system, or for debugging the Linux kernel itself, this doesn’t work. The kernel doesn’t make syscalls, and your embedded system might not even have a kernel to make syscalls to. And your hardware probably doesn’t have the right performance counters; AMD cpus only got equivalent capabilities quite recently.
On the other hand, perhaps you could capture and record interrupts instead of syscalls. Maybe the performance counters on the Cortex–M cpu could be made to work for RR’s purposes (I’m just assuming it has some performance counters; I haven’t checked). But that would take a lot of work, and most people just want to be notified when someone else has completed the work.
Cortex processors have a tracing functionality which allows capturing the entire instruction and data stream of the processor, allowing rr-style debugging. The main problem is this is almost always a very high-bandwidth stream which required dedicated pins on the microcontroller and a specialised (and usually expensive) adapter to collect. I've never actually seen it used, I don't know how many embedded developers actually use it (the only place I've heard tell of it being used is inside ARM). It's possible there's some configuration which only records the minimal trace needed to reconstruct the trace later in a way that is accessible to the software running on the chip (allowing extracting the data without needing a debugger attached), but I've not seen it, and the details of this functionality seem to vary substantially across different devices.
The go-to companies for these devices are Segger and Lauterbach. I'm a Segger user for JTAG debugging, although it's rare that the shops I work in will have a J-TRACE device on the shelf.
J-TRACE is the thing you buy when it's 5 minutes to production and your hair is on fire and you can't figure out what the problem is. Which is why Segger prices it at $2,000.
Yea, that’s how program recording is done natively by GDB as well; it inserts a breakpoint after every single instruction, and records the the state of the program after every one. It’s mind–bogglingly slow, and the log files are gigantic. You generally have to limit it to record only a few instructions, generally tens of thousands rather than billions, in order to get any use out of it.
At least if you can do it in hardware then it doesn’t have to be slow, even if the recorded state is still huge.
In my head, there is this (incomplete) concept that I call "functional debugging" (for a lack of a better term). Current debuggers are oriented very imperatively, i.e. here's what's bound to happen and here you see it happening, step by step.
I would like a debugger that would let me set up some query about the evaluation inside the program, for example, find me all input data in this test run that caused the output of a given function/expression to be such and such (possibly another data set produced during the debugging). Find me what are the typical inputs to the function. And so on.
One of the ideas would be instead of having breakpoints set to a place of execution, they would be set to passed values or program state. This is much more sensible in the world of functional programming, and processing of large amounts of data.
I can imagine a system like that to replace print debugging for me, but until then.. I find it easier to print.
Ill settle for ability to rewind. Whats the difference between console.log(somevariable) and breaking on that variable if the outcome is the same - me rewriting and running again?
Pernosco is that debugger. It builds a huge database of every recorded state of the program, and lets you query it. Here’s an example:
https://pernos.co/about/expressions/
Notice how the user clicked on a line number, and a window opened that shows every execution of that line. It’s not a breakpoint where you see only a single point in time; instead you see all the points in time. He then types in an expression, printing the value of the named variable at every single one of those points. Lastly, he types in a condition expression, narrowing the list to examine just the calls for a specific object.
While I agree with author that record-and-replay debuggers are nice, I don't think they will ever entirely go away. Those debuggers are ultimate "trace loggers", logging each statement and variable -- and thus they have inevitable overhead.
I can see how replay debuggers work with programs like Firefox, where you start with a few megabytes of input data and have a working set of a few hundred megs -- but there are programs with much higher memory/io usage out there.
Something as simple as "download a few gigs of data from internet and calculate its hash" will require enough trace to record all data downloaded, which would make the overhead huge.
A program running under RR generally only runs about 40% slower than normal, which is often unnoticeable. The recordings are usually smaller than you would expect too; the largest one I have is 74GB, though that was because the program read in a whole lot of data. I didn’t record it reading in the whole 400+GB file… most of them are under 200MB. Actually, about 60% of them are less than a single megabyte. There must be a power–law thing going on there.
Still, disk space is cheap; I may have to delete that recording one day but there’s plenty more disk space available.
As far as I understand, rr forces your program to run single-threaded while recording, so the performance overhead would be arbitrarily high depending on your degree of actual parallelism.
Not exactly. Your program will still have threads, but RR intercepts the pthreads API and runs all your threads in a single OS thread; it handles context switches between your threads as if your computer only had a single CPU core. This does mean that the overhead can be much higher than 40%! The 40% number is for a program like Firefox, that might jump up to 16 active threads to render a webpage, but also had to idle for 50ms while it waited for the TCP connection to start up.
I found that it was also a reasonable estimate for Reposurgeon, which is written in Go and uses concurrency during some repository operations, and when doing garbage collection. Other parts cannot be parallelized at all, so it balances out. I never measured that very precisely though; it would take anywhere from seconds to minutes to run a test without RR (depending on the data I loaded, etc), and only somewhat longer with RR.
> Those debuggers are ultimate "trace loggers", logging each statement and variable -- and thus they have inevitable overhead.
rr only logs sources of non-determinism, such as system calls (which includes I/O). Then all the deterministic state of the program is reconstructed at runtime using the recorded non-determinism.
So having lots of state doesn't necessarily affect trace size but you're right that lots of I/O will.
It will never go away.
I work on servers that are 150 to 200ms away in terms of latency. They're impossible to do interactive debugging on them (well not impossible, but really unpleasant).
Debuggers are not latency aware, they suck at it, they assume the debugged process is close to the UI showing the tooling, close in terms of latency and bandwidth.
Granted, most of the debugging is local, but sometimes replicating the environment so you can reproduce the issue is impossible or at least really time consuming, so remote debugging is a must, and when that happens, print debugging tends to win.
Not going to ever happen because print debugging eventually gets formalized into logging and metrics as a best practice.
My bias here is infrastructure, and you simply can't attach a debugger to a running production process to slow down customers.
At core, debuggers are an educational tool for new developers or inexperienced developers. Experienced developers will add metrics, assertions, and logs to the code which will make print debugging mostly redundant.
In the Linux Kernel I'm still using printk and I don't see that changing any time soon.
The best feature about print debugging is that it works in the vast majority of environments and languages and is trivial to get.
> Many people prefer print debugging over interactive debugging tools.
I don't "prefer" print debugging over interactive debugging tools.
I need something that works reliably, repeatably and with as much difference from the production build as possible.
gdb fails for me in all of these ways.
I would prefer a working debugger, yes.
I don't want to contribute to the printf vs debugger polarization that seems to be going on. But what I would like to mention:
The first thing I do after `git clone` and getting the build working is spend a couple minutes trying to get a debugger working, preferably using the IDE debugger. Usually it's pretty simple and it works out of the box. Sometimes it requires fiddling a bit with debug ports and cli arguments (legacy Tomcat applications) or making sure sourcemaps are available (web frontend dev).
If I can't get it working within 15 minutes, printf it is. But in most higher level projects it's usually a trivial initial time investment that pays off in the long run.
Debugging is one of the key skills all devs must master. Visual studio has one of the best debuggers I've used and it's still not perfect. That said, the title is still clickbait and print debugging will probably always have its place.
This article is an ad that doesn't even mention the real issues with print debugging:
1. Sometimes the information you can print about an object is not complete due to console memory constraints, depth of the information, dereferencing, formatting, etc. So you will have to come back again and print more ranges of that information, and format the data to understand it.
2. Logging takes "a lot" time if you are working on high performace algorithms. Then it will be dangerous to forget removing a print.
3. Prints can leak security information.
4. In web browser consoles you get a live print of objects, not snapshots.
That said, print is my favorite debugging method, but I constantly search and remove them, specially on release.
Can anyone recommend a youtube video of someone using rr effectively? I've used it a few times, and enjoyed it, but I feel limited by my beginner gdb skills. I think I need to watch someone more experienced.
That’s an excellent idea. Debuggers certainly need better documentation, and guided tours are a good way to teach skills.
Some of the Mozilla folks did livestreams of their development work on Firefox, including debugging sessions. I haven’t watched any lately, but you might search them out. If they kept it up then they might be using RR now, instead of just GDB.
Here’s a demo video for Pernosco: https://robert.ocallahan.org/2019/10/pernosco-demo-video.htm...
Pernosco is based on RR, but this probably won’t teach you anything about using RR, since the UI is so different. But it was a memorable one because Brendan Gregg produced a really good tutorial for GDB, where he debugged a real problem in a real program (instead of a made–up problem in a program written just to have a made–up problem inserted into it). The Pernosco demo debugs the same problem in the same program, so comparing the two can be quite easy. I would like to see someone the same type of tutorial for RR, and for other debuggers that people like.
The print statement is persisted in your code so you can easily see where you've been with `hg/git diff`.
Using a debugger I find myself repeating the same steps over and over again that I would have done/written once via print debugging.
That and once my system is deployed and I need to debug issues in production I'm relying on logs which are effectively print statements. So why would I want my dev debug and production debug flows to diverge? Doing print debugging gives me experience in what sort of information is useful to log as well as what is loggable.
Setting up a debugging environment is painfully difficult and time consuming that even the most pro-efficient devs (testers/qa) on debugging don't even bother wasting time.
This is the only thing that visual studio does better than any IDE on the market... zero-config debug, even some mature IDE's like jetbrains ones still to this day don't have zero-config debug, it always take some external and obscure configuration to get things right.
Good debug logs have a value beyond software debugging by its developers - they are useful for ops too. More than a few times nginx and postfix debug logs allowed me to see how requests are processed and find a mistake which would be hard to spot just by looking at the config. Debug logs can be enabled for requests from configured IP even on a server with high load if traffic from the IP is low; rr I assume has no such option.
Record-and-replay debuggers are amazing. My best debugging experience is still smalltalk from decades ago. You get an error unwind the code, edit live values, then hit run from any point in your code old or new and it’ll go on from there.
You can even edit your call stack manually, save the stack manually and restart it later on any machine.
RR probably is pretty similar for nodeJS but I haven’t had a chance to use it. Is it a game changer?
I just a console.log statement not long ago! It just works. I agree that sometmes it's an obtuse method of finding a bug but it works.
I agree (somewhat) that print debugging shouldn't be praised over interactive debugging, but when you write a revolutionary new debugger like rr, please please build a good UI workflow first, and make it work at least on Windows and macOS too.
For instance, from a "time travelling debugger" I would expect some sort of graphical time line view, where I can immediately see where on the timeline I am currently, and jump back and forth while inspecting program state and memory in UI panels.
A good UI is the most important part of a debugger, and the area with the most potential for improvement (Visual Studio is generally praised as the best debugger around, but when you think about it, the UI is archaic, yet it's the best we have).
Without a good debugging UI and intuitive workflows, it's no wonder people are falling back to printf-debugging.
Print debugging is perhaps just a sloppy version of debug logging.
I can't quite tell if the author is arguing against DEBUG/TRACE logging which is the gold standard in debugging, particularly in low reproducibility and multi-threaded/actor-based systems or systems with many 3rd party dependencies.
For those who do use print debugging, and are in Java, here's a trick that is occasionally helpful: When you know you're getting to a function, but you don't know how, create an exception in that function, print the stack trace, and then don't throw the exception.
Put your nose back down. Print debugging has its place. Just like a lot of other tools for debugging.
Print debugging is called debug tracing. And it will never go away. It's not used for debugging mostly, but to analyse traces.
Several systems cannot even use debugging. Such as realtime, kernel or embedded. The only tool they have is debug tracing. Looking at logs. I've worked years only looking at logs, tons of logs from tons of connected real-time systems. Writing tons of log analysis tools.
I love debuggers, but debuggers are a rare luxury to have. In many cases we even added embedded shells (read-eval-print loops, lua, picoc, ...) into embedded systems to query its state. Similar to a debugger, but also completely different.
You should also not forget LED debugging. Sometimes you only have two LED's.
There's a similar war between logging and tracing. Tracing promises to give you everything you ever wanted to do with your logs and then botches it by sampling the data so you can't correlate anymore, or you don't sample and suffocate in data just as with logging but without the benefits.
For debuggers it's probably the same: if they make the assumption that you need less than complete freedom in instrumenting the code, you will need print and if they don't assume that, they likely are harder to use than the language itself.
Omniscient tools typically require recompilation or reconfiguring of the world which introduces their own bugs and is prone to failure.
Better to use a language which has these included, is there any?
I disagree. RR can record any binary; it doesn’t require you to recompile the binary with any sort of instrumentation.
Well, gdb promised to be able to record/replay too, did not work for me for anything nontrivial, noone bothered with the bugreport for years, left a bitter aftertaste. I'm reluctant to sink more time into trying attempts to fix C debugging.
(https://bugs.gentoo.org/578194)
Even when GDB’s record and replay works, it’s terribly slow. I hope that one day GDB has features equivalent to Pernsoco, but putting a breakpoint after every single instruction is not going to scale.
I hope you’ll give RR a try; it’s been fantastic for me. (rr-project.org)
gdb's implementation of record-and-replay is terrible and to some extent poisoned the well :-(.
Print debugging is fine in an environment where the edit/compile/run cycle is fast or doesn't even exist. In fact it's probably the easiest way to debug in such environments.
What I've been thinking about is a code generation tool where I can specify the variable(s) I want to watch and it takes the original code as input and produces code with traces everywhere that variable is referenced or modified. If you're going to do print debugging then let's go all the way!
I hate print debugging. But the C/C++ gods in my workplace, who designed the whole system love it. So they did not care to make it easy for people like me to use debuggers. I am not skilled enough to figure out the exact incantations to run that will put in the correct symbols in the correct place such that gdb will pick them up.
So I use print debugging exclusively.
Print lines also comes with the natural flow of the program, so you have the added benefit of fixing something in a strong way with print line debugging.
If you make any wrong assumptions about freezing a program at a certain spot with a debugger ... I’m not sure, I guess I’m super biased about the simplicity of a good sharp kitchen knife.
Print debugging also lets me debug precisely how I want, across all languages that allow display of a value at a particular point in code. I'm not dissing folks who like the debuggers, but I think theres always going to be some folks who prefer one method over the other. "Print Debugging Should Go Away" is awful presumptive in suggesting that the author is an exclusive authority on how other people go about their workday. I mean the guy probably doesn't even know me.
Carrying water from a well to your house in a bucket every day should go away too. Buckets are fine if they’re all you’ve got, but there’s a reason people install plumbing. More than one, I should say.
The difference is that people have proven to be extremely quick to spend time and money on installing plumbing, but these days hardly anyone spends much time improving their editor or other tools. Some companies, like Microsoft, are rich enough to employ dozens or hundreds of engineers on VSCode, and then give it away for free in order to build good–will with a whole generation of programmers. I’m given to understand that Google has teams devoted to developer experience, who spend their time sharping the tools used by Googlers so that most Googlers won’t need to. But most of us don’t spend much time improving our tools, and we mostly can’t buy better ones.
It may be presumptuous, but I agree with Roc. If we spend more time on our tools, then print debugging _will_ go away. Not because we’ve forced you to use terrible tools, but because the tools will be so much better than print debugging that you’ll voluntarily start using them.
sure, but doesn't that sort of require me to believe print debugging is the same as carrying water from a well? I'm happy that debuggers work for folks who like them, and I'm not above using one but for me, the plain old fashioned bandwidth and flexibility I can achieve with logged output just seems to have natural advantages in situations that I seem to find my way into.
I'm not saying that "debuggers should go away" because I'm not going to presume that I know what works in someone elses situation, but it seems like this article is actually doing the exact same thing to me.
Everyone on that "side" seems to be more emphatic that "print debugging should go away" (the very title), yet for the most part, I'm not seeing print debugger types demanding that interactive debuggers should go away.
So there's a disparity in those two attitudes, that implies some advantages on one side or the other, wouldn't it be more productive to list the advantages and let the merits of each approach win out in each situation we programmers find ourselves in?
Not to mention my "vendor paranoia", I mean the guy already is making my agent orange act up by almost titling his article "the way mousepilot works should go away," that isn't exactly the sales pitch thats gonna win me over.
> "the way mousepilot works should go away,"
This is not at all what he is saying. Pernosco makes print debugging even better, faster, more precise, and with less room for errors. It also enhances every other aspect of the debugger, by making them omniscient as well, but nobody will force you to use those features. No one will make you click on a value to trace backwards in the program to find out where that value came from (https://pernos.co/about/dataflow/), or to debug multiple processes that were running at the same time (https://pernos.co/about/multiprocess/). You don’t have to use these advanced capabilities if you don’t want to. Perhaps you’ll never care who the callees of a function are (https://pernos.co/about/callees/). Maybe your programs run purely in the terminal, so you never need to click on a pixel to find out who drew it (https://pernos.co/about/screenshots/).
He’s saying that he thinks that once you have the option to do more than just print debugging, you will. Once the debuggers for your favorite languages support these features, you’ll gradually use them more and more, until you hardly ever do “just” print debugging any more. But to get there, someone has to put in the time and effort to make the debuggers for those languages good, not merely acceptable. Or Pernosco has to be extended so that it recognizes the interpreters or runtimes for those languages, so that it can extract the debug info necessary to let you debug your program instead of the interpreter. It can already do this with Chrome and node.js, which both use v8 to run Javascript. (https://pernos.co/about/javascript/)
When Python developers see that, they should sit up and wonder what they can do to have that feature for themselves. They can implement it themselves in their own debuggers, or they can extend GDB to provide extra support for the Python interpreter, so that GDB can show the source code of the Python program instead of the source code of the interpreter. Maybe they could help extend Pernosco too. Same with Erlang, and Scheme, and Go, and Java, and so on.
> doesn't that sort of require me to believe print debugging is the same as carrying water from a well?
Sure. And I get that it’s not as obvious as comparing buckets to pipes. Once you see how a pipe works, and a faucet, it’s pretty hard to deny that they have advantages over buckets in nearly all situations. It’s also hard to argue that buckets have any advantages over pipes at all.
But with debugging it’s harder, because the PHP debuggers are awful, and the Python debuggers are barely adequate, and the Bash debugger barely exists, GDB’s command set is about as tidy as kudzu smothering a stand of trees, and so on. It’s easy to argue that these tools are inadequate, and easy to dismiss all debuggers as a result.
But the sad state of the debuggers for many languages is due to primarily to neglect, not lack of potential. Even the fact that new languages need to write their own debuggers is sad commentary on the human condition.
So yes, print debugging needs to go away, just like poverty and death need to go away. We can solve the problems, we just have to get up and do it.
heh the one thing that needs to go away is me from this conversation. have a good one.
>Carrying water from a well to your house in a bucket every day should go away too. Buckets are fine if they’re all you’ve got, but there’s a reason people install plumbing. More than one, I should say.
Ok, but I am not going to set up plumbing everywhere I go when a water bottle does the trick.
>If we spend more time on our tools, then print debugging will go away.
No matter how much time we spend on plumbing will make the water bottle go away.
And it won't right along with fundamental CS paradigms like functional programming. Useful ideas tend to be sticky because well...they are useful.
This one has been around long before computers existed and the chances of it going away are low to none.
Note: Heck, even Java will probably stick around for quite a while ¯\_(ツ)_/¯
Most of my unit tests have print statements in them, permanently, because I usually know what things I expect will break. There's a ton of utility in being able to understand a problem by reading a transcript.
That's print debugging, and it's not going anywhere as long as unit tests are around.
While I much prefer debuggers, I've never had a print statement "crash" and stop working.
But I've had debugger crashes or weirdness where they stop working in every language and debugger option I've ever used (gdb, lldb, MS debugger, JVM, Rust, V8, etc).
I'm not against using a debugger, however one benefit print debugging is that you can print out intermediate states in a loop and look over all of them at once. Using a debugger would require stepping until you find the wrong iteration.
I personally think the future of debugging is tracing experience like this - https://www.youtube.com/watch?v=4vtKRE9an_I
So does rr work on arm yet? Nope. Back to good old printf.
On a realated note, the number of times I've had gdb crash on arm is so ridiculous that I don't really bother with it unless I absolutely have to.
It makes me sad to hear people say things like this. Did you debug GDB to see why it was crashing?
The saddest command I've ever typed was 'gdb gdb qtcreator', where inside qtcreator I was trying to debug using gdb. qtcreator was crashing when I tried to do that, and gdb was crashing when I tried to debug qtcreator crashing, and eventually I located a bug in gdb (which it turned out had been patched in a later version). It was a massive waste of my time and way off into yak shaving territory that I should have abandoned long before.
Embedded debugging is a pile of woe as well, in my experience. On the current system I'm working on, with 4 different chips using 2 different architectures, 3-4 different JTAG adaptors and different vendors gdb integrations, none of the debugging interfaces work reliably. If you can get the software to start running under the debugger, usually the first breakpoint will get hit correctly. If you're lucky it'll even continue running after you restart it from that breakpoint. The next breakpoint is even less likely to work, let along actually being able to trace out the path of some code. In theory on one of the architectures the interface understands freertos threads, but that just reduces the odds of a successfull breakpoint even lower (and you can forget about support on the other ones).
> It was a massive waste of my time and way off into yak shaving territory that I should have abandoned long before.
I don’t know if I agree with you there. I suppose you did waste your time, since the bug turned out to already be fixed, but every crash has to be investigated by somebody. Otherwise, the wouldn’t get fixed. If they don’t get fixed, they simply waste everyone’s time, over and over again.
The sad statement above was that the commenter saw a crash in a tool, and gave up instead of investigating. Thank you for not giving up when you were in the same situation!
> Embedded debugging is a pile of woe as well
I’ve never done anything of note on embedded platforms, but it certainly sounds like it. You would think that you could pay the vendors for better tools, but it sounds like they all just do the bare minimum, or they just do the bare minimum for anyone who doesn’t pay through the nose.
I didn't. I've a hunch that it's a very deep rabbit hole, which means understanding and fixing those crashes is going to be a significant time investment.
I can't bill a customer for that time, and my boss won't be able to pay my salary if I spend weeks studying and fixing broken old debuggers instead of doing the work a customer pays for.
Either way, the debugging experience without something like rr sucks hard when it comes to heavily multi-threaded, asynchronous, event-driven systems. So even if gdb didn't crash, its usefulness is very limited compared to simple tracing.
There is experimental support for ARM. https://github.com/rr-debugger/rr/issues/1373
Also, Undo works on ARM.
"AArch32 support is WONTFIX, as far as we are concerned. Even if implementing rr on 32 bit ARM were possible I don't think anybody cares about it at this point."
Logs are useful, since they tend to be easy to analyze and aggregate.
However, if you are logging the entire state of the program over time then you're probably better off using an omniscient replay debugger.
I might be spoiled, since I started working on Windows using Visual Studio back in 1996. The debugger always seemed a fundamental part of developing (an learning to develop) applications.
10 years later, I was still being spoiled while building Windows CE OS images. There you had KITL to debug (over serial, ethernet or USB) the OS in real time from the IDE.
When none of that worked, I purchased myself a Lauterbach probe (6K USD). The debugger IDE is the ugliest thing ever, but it's the most powerful tool around for embedded.
Some years later I entered into kernel (Linux) building for embedded. When asked the guru what's there to debug the kernel, the answer I got was "printf". Man, I laughed so hard until I saw his "I'm being serious" face. Then I get used to it, but I still hate it.
Again, some years later, I have to build Android images. Again, skimming through thousands of log messages thanks to dmesg and logcat... I still hate it. It seems archaic.
The better moments of my current work is going bare-metal, debugging with an IDE, with my 15 euro JTAG plus OpenOCD and Eclipse. I couldn't be happier.
In fact, for bare-metal embedded, there is no other way around. There is no printf. You can have it with semihosting or a serial port, but it's painful... it messes up with my timings. I need to debug using an IDE. And GDB from the command line... come on... it's 2021. It can be avoided.
Even when I started with Rust for embedded I wanted a full debugging IDE (which I didn't find yet). I want to see how things get arranged in memory (how does a slice or enum look in memory? and, is it possible no one checked this before?) side-to-side with generated assembler.
TL;DR, my opinion is that printf might be fine for some desktop development, but as a "I don't have any else to try" resource.
For embedded microcontrollers, I don't think there is anything to really replace print debugging. Especially when the program interacts with external electronics components.
I spent two hours debugging React in IE and still think print debugging should be the last resort, not your goto.
Relevant XKCD: "Is it worth the time?" https://xkcd.com/1205/
It's still unclear to me that learning an interactive debugger would save me any time at all.
Figuring out where I need to judiciously place print statements also helps me to figure out the code in front of me.
If it lets you avoid recompilation, it's very likely it could save you a significant amount of time. I really learned GDB for an embedded Linux system where it took 5-10 minutes to compile and 5-10 more minutes to load the updated build and then you have to restart the whole device. I don't really feel like spending 20 minutes (worst case) or even 10 minutes (best case) to get to, hopefully, the right printf. Make a build with debug symbols, load it, run it with a debugger attached [0], maybe restart a few times to recreate the error path (which might have to be done with printfs as well). And once that process was established and written down (extra build target, half page of instructions on getting things properly configured for our environment) it became much, much faster than print debugging.
[0] https://en.wikipedia.org/wiki/Gdbserver
I bet the author uses print debug statements some times.
Ignorance is bliss!
sed "s/should/will not/"
console.log("no");
echo "Never!";
This Type Of Blog Spam Should Go Away
The author of this article should go away. When I saw how to set breakpoints in Node, that was the end of it.
Man discussions in this thread really seem to be between those who know their debugger, and those who don't and doesn't want to learn.
Is there anyone out there who is actually familiar with a debugger for the tool they're using and still prefer print debugging?
No?
That settles that then...
Coming from embedded Linux programming, I gotta say sometimes (for easier problems) the overhead in altering the init and spinning up a gdbserver is too much; just pushing a few prints to debug something is easier. However - just having learned about rr, which seems like an amazing tool - I think that can breakpoint might change...
Does "print debugging" include debug logging? If yes, then I think it's absurd to claim that debuggers can replace debug logging entirely.
I for one use both print debugging and actual debuggers (I've used the C# VS debugger, windbg, Java debugging in Eclipse, NetBeans, and IntelliJ, CLI gdb and pdb, Emacs GUD for gdb, pdb and Go delve, Emacs edb and the CCL debugger from SLIME). Sometimes it's awesome to point a debugger at a core dump and fix a year-old memory leak, or just see windbg point out a deadlock involving 5 locks in 2 different processes. Other times it's awesome to interpret debug logs and figure out a how a flow in a customer setup got interleaved and produced a race condition two weeks ago, which is causing a crash today.
No no, that's not what I meant at all. I meant print debugging can't replace the debugger.
Of course a debugger can't replace the need for logs entirely!
Oh, then we are agreed!
The article is in fact claiming that a debugger can replace print debugging entirely, that's why I read your comment in this same vein.