> As a teacher, I can tell you that students get really angry if you put a question on an exam that requires a concept not explicitly covered in class. Of course, if you work as an engineer and you’re stuck on a problem and you tell your boss it cannot be solved with the ideas you learned in college… you’re going to look like a fool.
Very flawed comparison. At work I get to go off and do research, experiments, can collaborate with peers and people who might have more expertise in a given sub problem, and generally have much more time. An exam trying to test you on material you haven’t studied is supposed to test for what? Your ability to synthesize knowledge out of thin air.
The rest of the article is well written and correct, but this particular aside felt weird.
I think it depends on the question. If it's not a question of the form explicitly presented before, but answerable with a minute of thinking using the knowledge the student has already mastered, then it makes sense.
A time limited exam is probably the wrong place for that, though, due to the stress interfering with that kind of thinking. It would be better for a homework assignment.
If ChatGPT didn't exist.
Okay, maybe in class, on paper is the right place for that.
The while premise of "learn some stuff them take an exam on exactly that stuff" is pretty flawed, and that's the point. So much of the academic structure is about what's convenient for evaluation, rather than what's best for learning. Why not get rid of the exam and replace it with something else entirely? Who says we have to have exams at all?
> So much of the academic structure is about what's convenient for evaluation, rather than what's best for learning
Academia is going to have to focus on the latter anyway because we have AI, which is exactly the same: it’s only useful when you can evaluate the output for accuracy independently. But how do you do this without throwing out objectivity entirely? Maybe we need to move to apprenticeships?
From my experience, the boss is usually a complete moron, so who cares. It also creates this unhealthy assumption that the engineer is subservient to the boss.
From my experience the boss does not know things you know, that does not make them a complete moron because they probably know things you don't.
Here's an example, consulting at a large Danish company, every Friday morning all departments in this big building would share breakfast and the bosses would say some things.
So this one morning they explained that in the coming months people should register time in a particular way because of accounting and how it related to a particular government grant and money that needed to be used up by a particular time in order to get to the next step of blah blah blah.
I realized as my eyes glazed over, damn this is just the same reaction people who don't understand browser rendering engines get when I start telling them about different events.
I also noticed other clueless people gamely trying to question these finance nerds on how things worked, and the patient finance nerds explaining some detailed bit and the clueless person clearly out of their depth with that "uuuuhhhh, hope they don't ask me if I understand" look on their face.
Now, if it hadn't been for them explaining this stuff I would have gone around thinking the boss is a complete moron. I once saw him mistake a nail gun for a drill! He doesn't understand how search engines work and why stemming and decompounding might be important, I know because I tried to explain to the idiot one time!! But since he actually talked about his work for a bit I realized he just happens to know stuff I don't.
I'm betting most of the morons you know are maybe not quite so stupid, although probably not as forthcoming as why things need to be done in a certain way to those who work under them.
I feel like it's an argument for the benefits of abstract reasoning. I don't think they are saying it'll be like that in the real world, I think they just want to test how you do under adverse conditions.
Stress testing the student's academic prowess, if you will.
>> Your ability to synthesize knowledge out of thin air
As someone who graduated high school, I'd hope my more accomplished peers would know the difference between hypothesis, theory and proof. It is entirely possible, and useful, to test someone's ability to form a cogent hypothesis. If you were faced with a question beyond the scope of the ideas you were taught, and could not rely on any assistance, the only useful thing to know about you is how well you would handle it yourself.
If you would synthesize knowledge out of thin air, that would be a failing grade.
> If you were faced with a question beyond the scope of the ideas you were taught
Who knows the scope of the ideas I was taught? I mean, I could use derivatives and sometimes to integrate in 8th grade, should I get a failing grade if I used them in a test?
I see the point of your post, it is important to distinguish between what you know, and what you don't know, but maybe suspect. School tasks are built in a way, that the very form of the task can be used to make good guesses that may help you to solve the tasks. It doesn't work in reality, but it works in school. But the solution is to add questions without answers, not questions with answers beyond someone's knowledge. Like "what was Mozart's opinion on pineapple pizza?" or "what size was Julius Caesar?". Well, tests needs less obviously unanswerable questions than these, but you got the drift.
Edit: Thought it will be tricky with math. For example: "Alice has 3 apples, and Bob has 2. How many oranges they have?". School math problems have a convention, which means that if oranges where not mentioned, then they have 0 oranges. Compare with "Alice has 3 apples and Bob has 2 oranges". If the problem doesn't mention Alice's oranges, then Alice has 0 oranges. It is implied and these implicit rules are the first difficulty kids face when starting with math.
I found the whole article to be a bit heavy on anti-academia. And I went to industry after undergrad.
It's a false dichotomy between the "thinkism" bogeyman (actually reading books and papers and putting work into theoretical design is just bad now? Have they tried building anything in the physical world? Checked in with nuclear physics, ever?) and hands-on experience. Both are important. It should be about balance, not trashing an incredibly valuable set of tools because others exist...
I am team academia more than hands-on experience. And I have 5 years of experience. To me, it felt like most SWE things could eventually be solved by what I learned at school.
Not everything I did I learned at school, such as navigating codebases with more than a million lines of code. But most things? Yea.
With that said, I am curious how people say that they learned much more through experience, what did you specifically learn?
I find that many people can learn a lot by doing but then at some point hit a wall and really struggle to recognize that another kind of learning needs to take place to understand a deeper concept.
Could you tell us what you learned at school that is useful for your SWE career? Im sure there is a lot of us that learned 0 at school and learned everything ourselves as kids on the internet
The problem with "learned everything ourselves" is that you might have niche interests and you miss things. Things I learned that probably I wouldn't have by done by myself: computer architecture (memory, buses, cpu-s, instruction set), and related VHDL/Verilog; how complex is synchronization (implementing from scratch synchronization libraries); different programming paradigms (functional languages); compilers & operating systems (kernel modules, etc.); various types of maths (dsp); algorithm complexity analysis.
Some I ended up using more during my career than others, but knowing more definitely reduced my tendency to think "ah, that should be easy".
Benefits of learning online yourself is that you're self motivated and you get access to the best resources (better than the teacher you are forced to listen to)
Edit: feel free to summarize this with an LLM it will be a ginormous comment.
That's a fair question and I'll do my best to answer this. It'll come in an edit. I think it's fair to say: not all courses were created equal in this regard but I'll do it course by course. I studied a bachelor information science but I tweaked my program so close to computer science that I almost daresay it's computer science (if I had 3 courses different, it was). I studied a bachelor in psychology. A two year master in computer science and a one year master called game studies (officially a specialization of information studies, but in practice it wasn't and it really was game studies as a whole field that we studied).
I'll try to do it in order per study program too.
AI-kaleidoscope: general overview of AI algo's. It's a shame we didn't know how to program or that we knew the usefulness of BFS or DFS but we learned it here. Not a useful course due to scaffolding issues (teach programming first).
Business mathematics: partial derivatives, etc.
Problem solving: useless course (teach programming first).
Privacy and security: we didn't learn much about security. We learned a thing or two about privacy. Should've been a TED talk, not a course.
Graph theory: graph visualization (and when not to do it, helped me out as a data analyst later), mathematical proofs, social networks (helped me to actually network a bit), graph algo's (made leetcode easier), not being scared of math notation. This was a really math heavy course as it was taught by someone that studied math in undergrad and grad and then switched to CS as his PhD (and by the time he taught it he was a full professor). He did not skip on the math which was wild since I was under the impression that I did a "business informatics" bachelor so I didn't need to have advanced math as a high school prerequisite. But I definitely needed that here so this course was hard.
Web technology: this was a bit too early but it was a good overview of web programming at the time. I remember being explained what the DOM was and I remember thinking "wtf is the point?" That was because they explained it way too theoretically and simply should've opened firebug or something to really show it.
Language of Logic and Methods of Reasoning: propositional logic and predicate logic. Practically speaking: after this course if statements are not a problem. This was true for me at least.
Pervasive computing: fun course but could've been a TED talk about how tech is used in interesting ways.
Introduction to programming: basic programming stuff in Java. I learned that Java is a terrible language to start programming in. I recommend JavaScript for app/web-oriented people and Python for "just pick a language" people. Nevertheless, while it was a terrible start, it did teach what it needed to teach which was a basic understanding, and skill, in programming.
Empirical methods: the better name is statistics 1. We learned about statistics and we had to program in R. Since our actual programming ability was quite weak it was a double course. It was the second language ever that I had to take seriously and it taught me a lot of programming stuff and statistics stuff at the same time. It helped that I had friends studying psychology at the time as that degree has a lot of stats in it, so I knew the lingo. Almost everyone else was hopelessly lost.
Programming project: no lectures, just one big programming assignment. After this it was expected that the student could write readable code. I failed this course the first time by a hair. And when I came back to it the second year I realized that I failed because my code was unreadable. So I refactored the whole thing and learned how to write much more readable code. We created the game engine for an Othello game. The graphics library was provided by the TA's. So we also learned to program with a library that was way beyond our heads at the time. And that also implicitly teaches you to trust certain abstractions.
Interactive multimedia project: we learned XIMPEL a hypermedia framework, kind of useless. But through XIMPEL we also learned about storyboarding, creating scenes and general video editing. This teacher was super hands off and allowed students to be creative. So I also learned to create simple PHP websites and learned my way around bash a bit. Then I decided to create an upload script where I used PHP to call the ftp command on bash. I thought it was impossible but it wasn't and had to rethink about what web applications could really do. None of these things were formal course requirements but this teacher encouraged this type of explorations so I do credit it to him that "I learned it in school". It's part of these efforts that he also gave me an amazingly high grade as my XIMPEL story graph was a bit meh but my creativity and extra explorations where a 10 out of 10 effort. So he gave me a 9 out of 10 in total. Also deepened my HTML/CSS knowledge.
-----
I'll write the rest later in another comment this is about the first year and I studied 9 of them. As you can see, I was just learning the basics here. But there are a few patterns:
* Some teachers allowed us to work on real stuff if the student chose to (e.g. Interactive Multimedia) and that got way more serious later on as I created an iPhone app for a client when the teacher taught Multimedia Authoring in the master.
* Some courses were quite useless or could've been condensed to a TED talk.
* Some courses taught something useful or semi-useful but it's not up to industry standard. But as you'll see, this will lead up to a level where - while not quite industry standard - makes the gap between industry standard and whatever I did small enough to just make the jump easily by simply applying what I thought was common sense.
Interesting ! What is the name of the schools you attended, if that's not too private ?
In France at "prepa" (2 year intensive courses to prepare for exams for big eng uni) I learned the theory behind computer science (for example how to modelize a regex machine with graph / automata / matrices). That was useful theory to me, but that's just a drop in the ocean of uselessness
A lot of technical skills sure. Algorithmic analysis sure (although in practice it’s a lot less formal and more intuition + rote knowledge.
But taste about the best ways to structure a codebase? How to balance speed vs quality and when to do that? What does good vs bad observability look like? How to handle services going down in production and run post mortems? How to handle ambiguity both at a leadership and technical level to chart a path forward for a team? Even reading about these things from blog posts and things is nothing like actually experiencing these things first hand and certainly not taught. It requires accurate and constant self reflection and retrospective to guide an exploration process and if that self reflection tool is askew you can end up learning weird lessons and habits that are difficult to correct (and we all have them somewhere).
I’m team learning and was team academia early on, but the real value in academia is in teaching you how to think and to give you some minimal useful knowledge to about that capability. But it often struggles to even that because it’s still following an assembly line model of teaching whereas learning is relational.
Source: nearly 20 years as a SWE from a well regarded university.
Every company structures a codebase differently. So it’s best to be open. I have worked at many places where speed vs quality didn’t come up. Also it depends on the constraints. A HFT company is much more worried about getting cache hits whereas a big tech company seems more concerned about a big n. Good vs bad observability: well you better know your statistics well, otherwise you lull yourself into a false sense of security. Not my strongest area tbh. I haven’t worked at a company where post mortems are needed.
Ambiguity: there is a lot of ambiguity at uni as well. So you should have had the practice, especially working in groups of students.
> It requires accurate and constant self reflection
Skills I sharpened at university by studying psychology. Do you know how big the mind fuck was to grok the reproducibility crisis while being a 3rd year student in that program (5th year student overall)? I had to re-examine my whole degree.
I also had courses where self-reflection was a weekly part of the course. And I took that seriously.
> whereas learning is relational
I consider it a mode.
Try mathacademy.com it will teach you the “shut up and calculate” version of math really well, and there is nothing relational about (since it’s an app).
I also had courses that were about computer and network security in C and assembly. In it, I self-taught vim, assembly, C and git. The self-teaching was partially the point.
One thing I have learned from university is that what many people see as good practices is more akin to a culture rather than an actual good practice, because the actual empirical work in figuring out what works hasn’t been done. And I find what you say to strongly fit a particular culture of software engineering. But there are many forms. For example, in the Smalltalk/Pharo community it is recommended to extend the base classes to your liking. Whereas in other languages they look at you with horror. Back in the day (2012) Airdrop was what people used and didn’t want to use git, and these people have programmed for decades (I disagree with that one personally, just showing the culture).
Theoretical design is pretty worthless until you try it out. Until then it’s just an idea or hunch you have. You hope that your experience and intelligence will tip the scales in your favor but that’s all it is. And often your design still has flaws anyway because there’s bound to be unknowns. So then your hoping some kernels of ideas still hang on after encounter with the world and that the ideas are directionally resistant that you can keep the design going forward.
Design is useful as a tool to sketch out a plan but it can definitely become thinkism if you always hand off your design to someone else to action.
Also, the author is talking specifically about research type problems where there’s a lot of unknowns and the problem space is not well trodden and understood.
Also, I don’t know when the last time you built anything is, but talking with mechanical engineers and construction engineers, it’s all the same. You have your designs and then you open up a building to do repairs or start construction and some surprise presents you that you weren’t expecting. That’s what the research phase is for where they learn all they can about the problem space before they start designing (in this discussion ignoring all the years of trial and error that built a base of experience already) and even still in practice problems arise.
You have your designs and then you open up a building to do repairs or start construction and some surprise presents you that you weren’t expecting.
And yet, the field of architecture is still highly valuable. Without those designs, there would be a lot more problems. Try building something complex without a design. You're going to waste a lot of time and materials. My point there was that it's not like the digital world where mistakes and trial and error and rethinking things can sometimes cost almost nothing.
Theoretical design is pretty worthless until you try it out. Until then it’s just an idea or hunch you have....ignoring all the years of trial and error that built a base of experience already
Lots of (smart) people have tried lots of things, there is a ton to learn from that, you're calling it worthless and taking it as an assumption at the same time.
I've had a research-heavy career in computer science. A central problem with the academic research is it commonly ignores real-world constraints. Or less commonly, it imagines constraints that don't exist. As someone who went deep in a few domains, academic literature is usually disappointing[0] once you've read and understand the entirety of it.
The valuable thing about hands-on experience is that the real world doesn't let you ignore constraints. You get that feedback quickly if you are paying attention. This in turn allows you to build a more accurate mental model of the true nature of the problem you are solving and where the hidden limitations and leverage points are. A lot of academic literature tacitly works from a set of assumptions that don't map to any real-world environment.
Once you have that hands-on mental model, the flaws and limitations of much of what is in the academic literature becomes obvious from first principles. Most of the insights might be academically interesting in a theoretical sense but they often aren't reducible to useful practice in real systems. Non-academic careers require implementations that actually work well.
[0] The computer science literature from the 1970s and earlier is much better in this regard than what came later. Many early papers were written by people that clearly had concrete experience in the trenches with the problems they were writing about. Those papers are both more readable and more applicable. This awareness of constraints is lacking from a lot of modern computer science papers.
I agree that it's a flawed comparison but it does touch on a very real issue in the workplace. The difference between employees who I can send away on a problem and they'll encounter issues and try to solve them, vs those who come back to the senior at any problem, presents a serious ceiling on the level those employees can work at.
Even then I would call it bad teaching, as the usual mode is:
New stuff gets explained in class by the teacher - the details are for self study.
This is how it was communicated to me in school, so I would call it part of the "contract".
But giving a homework on a fundamental new subject without explicitely telling that this is what it is about, will rightfully confuse, as the normal mode is different.
And when it happened to me, it always happened because the teacher messed up and got confused themself what they had and have not teached before.
Teaching to self study is a valuable skill in itself that should not be taught by accident, but with a clear pedagogic concept.
As a leader, I can just tell people literally that: if you can't go away and work independently on a problem, it puts a ceiling on how well you're going to do. Then I ask people what they've tried before asking me.
Just being upfront with people can break low performance patterns of behavior.
I think it’s largely there to set up the point that comes after, which is that it would be absurd in a professional setting to pronounce a problem unsolvable because the entirety of your university education doesn’t provide enough information to solve the problem.
I took an exam once and there was a question on it that wasn't covered. I had vaguely remembered something from somewhere else and was able to piece together the formula and actually got the correct answer. When the test marks came back, the prof apologized about the question and didn't count it in the grades. Myself and two other people in the class who got the answer were held after class and accused of cheating. That is the school system. They do not reward creativity or ingenuity.
Sounds like the synthesis problem could be the difference between a 8-9/10 and a 10/10 score. It would work in an accuracy grading system instead of a hedge against stupidity system.
I grew up poor. I thought my ticket to not being poor was an education. By the time I was applying for colleges (I had good grades) I realized that the education system was a joke.
I got through college, a very well-regarded engineering program, putting in the bare minimum effort to graduate.
I make a lot of money now, 20 years later. I attribute none of my success to public education or college, I didn’t learn anything besides our education system (in the US) is not what they tell kids it is growing up.
We beat the drum of education because it seems like a nice simple solution to fix a lot of perceived problems. “Oh you mean if we just throw money at this problem it’ll fix it?!”
Baltimore city gets some of the highest per-student federal funding in the country. My daughter is in a public middle school there, living with her mom. I’m in the burbs with my wife and kids.
She told me just yesterday she wants to go to high school in my district because “well it just seems more normal than city schools”
She also thought California was a country up until yesterday. Straight A student.
At some point, probably too late, we’ll realize we did it a wrong. Floating kids through high school, dropping an anvil of debt around their neck for a college degree they won’t use. Met a fellow the other day, he has his masters in social work. Making very, very good money, as a regional manager of a car sales group, I forget the specifics. Said he never used his degree and he is very grateful he has a clear path to pay the debt off.
I don’t know what the fix is, but I’ve seen enough to know what we do now, ain’t it.
> We beat the drum of education because it seems like a nice simple solution to fix a lot of perceived problems. “Oh you mean if we just throw money at this problem it’ll fix it?!”
Its strange to say, but its actually thanks to black racism is why we're here.
The result was that the courts determined that "aptitude/IQ tests" unfairly discriminated against black people. However, education requirements are not seen as such.
So, we get "decent paying job" requiring 'a degree'. What degree? Doesnt matter. The degree is the legal form of 'we dont hire blacks'. It also has the side effect of 'we dont hire the poors' as well.
Obviously there are positions that require advanced schooling. Im not talking about those.
It is an article of faith on message boards that IQ tests are somehow unlawful in employment in the US because of Griggs. That's patently not the case. Plenty of large corporations with deep pockets do general cognitive testing (and even IQ testing) of incoming applicants. The companies that administer these tests have logo crawls, like every other product company, and the names are companies your mom will recognize.
Griggs says that you can't employ IQ tests to discriminate for jobs where the IQ threshold you established is immaterial to the job. For modern knowledge work, that's a trivial bar to clear. It is essentially the case that Griggs is a nonfactor in white collar employment.
My motivation for calling this out isn't that I'm OK with discriminating against Black people; quite the opposite, in fact. IQ tests aren't used more widely because they aren't effective at qualifying employees. IQ is an idee fixee among A Certain Part Of The Internet, and part of their pitch is that IQ tests are a secret cheat code that would enable hiring purely on merit, if only the woke wouldn't prevent it. No, false.
> Myself and two other people in the class who got the answer were held after class and accused of cheating.
Wow. I was never able to escalate situation to the direct accusation of cheating. I just got my marks lowered a bit because I used methods that were "above my abilities", as my math teacher believed. Though really I thought school math was boring and tried hard to make it more interesting, less routine, less relying on memorizing things, and more clever shortcuts from general principles.
But she never accused me directly. I know her explanation for lowered marks just because my mother visited her once with a specific test results and asked why marks are so low. I wanted a direct accusation, I was trying to provoke her to doubt my methods, but either I tried too hard and she knew deep inside of her the real situation, or I needed to do something truly exceptional and failed.
Yeah, thinking about it now, I should probably try some existing and very known methods but far above the school level, something she felt herself familiar with, so she could hope to prove that I've used methods without understanding them. And then it would come to my ability to prove, that I understand my methods. Something like raising e^(ix) into n-th power, to derive a formula sin(nx) = f(sin(x), cos(x)). Though, it is might be too weird for her.
> They do not reward creativity or ingenuity.
You can stimulate yourself if you look at it as a rebellion against the system. Teenagers are good at rebelling. Though, rebellion can make matters worse. I rebelled against history trying to get the lowest marks possible. It didn't end well, and I not just didn't learn history, I've got a life-long trauma, I start to regress to my teenager mind level every time I face history.
That's infuriating to me, and I wasn't even the victim!
I'm so lucky such a thing never happened to me. The closest thing was a math teacher in middle school asking me about a problem solution of mine in a test, that he had marked "mysterious" (I had to come up with my own solution path during the test because I hadn't memorized the canonical one we were taught). He asked me to explain my reasoning, and when he was satisfied that it was sound he gave me full marks for the question.
Nope, and nope. I've had tons of such questions during university in various courses, so what? Are they supposed to to babysit you and handle you only exact stuff they went over and over, to not upset some fragile minds? Such stuff is part of educational life. They expected to synthesize the knowledge out of other, related knowledge that was very directly covered, and use a bit of if-this-then-that. All in trivial manner if you actually grokked the subject well enough.
I failed some, did well on others, felt a bit of unfairness emotions completely understandable for that young mind which just wanted to do the test and move on. But when looking back, they were good questions, testing my actual skills in topic I was studying.
Absolutely. I did this just last week. I couldn't figure out why I wasn't able to get out application to use a fileserver in a new test environment. I brought it to the team and got help from someone else.
> As a teacher, I can tell you that students get really angry if you put a question on an exam that requires a concept not explicitly covered in class. Of course, if you work as an engineer and you’re stuck on a problem and you tell your boss it cannot be solved with the ideas you learned in college… you’re going to look like a fool.
I really hope this kind of thinking dies away because the longer you wait the more money you waste, not just your own salary but the opportunity cost of time. And this person is a teacher?
Maybe the impression cones from a different understanding what "concept not explicitly covered in class" means.
Of course you would not test a completely unrelated concept but where I grew up "Transfer of Learning" is supposed to be part of every test, with a proportion that increases as students progress through school.
To test pupils ability of transfer of taught concepts means to let them apply information, strategies, and skills they have learned to a new situation or context.
My problem with my country of origin’s school system (Hungary) was that this “transfer of learning” was used solely during exams. It would have been really great, that after doing only simple, straightforward exercises during classes or home work, not the exam would have been the first time when students had encountered real world, way more complex exercises. (Btw the other really annoying exam type was when they tested your raw speed of recollection, in other words, whether you did that particular type of exercise a thousand times, and not that you understood or could you do it, like we would have practiced to ER or something where muscle memory can be life or death)
supposed to test for what? Your ability to synthesize knowledge out of thin air.
Yes, actually. Synthesis is the penultimate level of Bloom's taxonomy [1]. Perhaps not appropriate for a first quiz on new material, but absolutely in scope for a final exam.
I had many exams in university that challenged me to synthesize a proof to a hypothesis I'd never seen before. I had to bring together my knowledge of the material I'd learned in class with the bit of new material presented in the problem statement, and then devise a proof with one or more steps I'd never done before.
Many of my fellow students in first year struggled mightily with questions like this. There was much wailing and gnashing of teeth after the exams. In later years we'd all gotten used to it, and it was to be expected. That students in high school or younger years have been sheltered from synthesis-type problems on exams is a great disservice to them.
Edit: I should also note that outside of upper-year pure mathematics courses, synthesis-type problems rarely accounted for more than a third of the grade on an exam.
Synthesis problems and time pressure don't feel fair together.
I'd be furious to get some lofty theoretical synthesis problem on an engineering exam where the other problems are grindy analysis of "looks like concept you know but expand into a form you haven't seen and then will take 1/n time on your n question exam". I suppose these are already 1 extension in, so 1 more logical extension isn't that far off
They're absolutely fair if you're given plenty of practice at them. If the only synthesis problem you see all year is on the final exam then you have a point. However, if you're doing novel (to you) proofs on assignments every week throughout the course, you should be used to the novelty by the time you reach the exam.
Expecting a final exam consisting only of calculation and memorization type questions is wholly inappropriate in a mathematics course full of proofs.
I think you are both right and it really depends on the leap that has to be done during the time constraints of the exam and how much practise was included in the curriculum.
Off-topic, I know, but I always disliked the time pressure of exams. All the bad scores I got were down to not feeling quite right at the time of the exam: maybe I didn't sleep well, the food I ate did not quite sit with me, had a fever, etc. None of these things actually impair too badly but if the time is short, there is an impact.
I suppose it's similar to sports where you have to, at least somewhat, adapt on the go. Many teams botch a game here and there. But... your final exam? Botch it and you're screwed. It just does not feel right.
it depends on what you mean by fair, i think. is the purpose of the test to gauge how well a person performs on tests? the difficult problems i encounter in the real world involve collecting a bunch of context on it, sitting with it for a bit, then putting it down and going for a walk. when i come up with something to try it's usually when i am outside and looking at trees.
maybe this is good for weeding out students who would not be a good fit for high stakes speed math.
It's really not high speed math. You'll have 2.5 hours to solve maybe 5 problems on the exam. A math grad student who TAs the course can typically complete the exam in 45 minutes or less, without any preparation whatsoever. An undergraduate student who can't solve the exam in 2.5 hours (without external aid) is not going to solve it in 10 or even 20 hours. They simply do not know what they're doing at that point.
It's really no different from a software dev interviewee who can't solve basic programming questions without being handheld and coached through the entire process.
I think if TA no prep completion time to total time ratio is above 2.5 it's not the time pressure based exams I'm referring to.
I didn't get those exams in math courses. Engineering courses I did get those where some of the point of the test is the speed of your pattern matching ability (did you use design skills to reason about it or did you brute force a solution).
Math you're already extrapolating theory and you need ample time for that.
If you can't do synthesis problems on a exam, you are not prepared for it.
I quite fondly remember a situation with a two question three hour open everything exam where I stared at the 1st problem and relevant equation for 20 minutes because I had no idea what I was looking at and was wondering if I should just leave and drop out.
Then I realized that a bunch of terms could cancel out and the problem became 3 lines trivial. The professor later admitted he wanted to give a "easy" question that you couldn't just copy out of your notes and had to think and to spend the rest of the exam on the actually difficult one.
This seems like you proved my point by staring at it for 20 minutes before starting. I don't think it's fair to grade with extreme time pressure and also expect deep thought.
Being able to do deep thought quickly isn't a valuable skill, and you don't pick it up on the way ti mastery with lots of practice like you do the speed for rote problems
> Synthesis problems and time pressure don't feel fair together.
If you really want to grade the students its fair to put very hard problems on the exam to separate out mastery from merely very good understanding or recital.
What skill are you grading? Can you expect some students to be able to churn out the problems? Are they doing design work? Is it more inventive or more creative? Is there more than one solution? How rigorous is the solution?
My point is that you can design an exam to test for speed and ability to do the day to day well. You prepare for this exam with lots of practice. There are a few problem types but you might not have seen every one. There's a basic level of extrapolation you're expecting.
You can design an exam for abstracting theory, and you prep for this exam by doing the hardest problems you can find, reading more theory, etc. You might not have seen anything like this but the test is to use the tools you're given in a class well. The test is the extrapolation.
I had theory courses where I expected the latter and practical courses where I expected the former.
There's more than one type of mastery; I bet your math professors would get destroyed by gifted 14 year olds in algebra competitions that involved arithmetic and timing. Do those 14 year olds not have more mastery?
Hard problems introduce noise to the grades. A student who gets a hard problem right could be good or lucky. In order to grade the students properly, you need multiple problems of similar difficulty. If you have both ordinary and difficult problems in the exam, it's probably long enough that it should take the entire day.
Undergraduate exams tend to be short. Which means that a perfect grade should be interpreted as "meets the expectations".
Which means that a perfect grade should be interpreted as "meets the expectations".
None of the mathematics exams I wrote during undergrad at Waterloo (as a math major) would fit that description. Nearly every single one of them had midterm grades with unimodal distributions centred below 70%, tending toward 60%. Typically, only 1-5 people in the class (of 100-200) would score a perfect grade. In upper year pure mathematics courses it was common to not have any perfect grades (in a class of about 20-30).
Mathematics is notorious for exams like that. But if you look at the reasons why people fail to get a perfect grade at undergraduate level, it's almost always due to honest mistakes or because they didn't learn what they were supposed to learn.
In my experience, studying mathematics is a bit weird. If you are ready to learn a topic, it's probably the field where you can get top grades with the least effort. But if you can't learn something with reasonable effort, hard work is unlikely to help. Doing something else and trying again after a few months might help.
Not of thin air, but of what you've studied so far. Of course some (many) teachers are imbeciles and just use the same exam they've used in the previous year, even if they haven't covered all the topics yet. But in general, it's a very good idea to have a small question that tests for real understanding, i.e. applying your knowledge to a problem not previously presented, versus mindless reproduction of memorized facts.
> Your ability to synthesize knowledge out of thin air.
The skill the class is teaching is the ad hoc synthesis of knowledge from other building blocks. That kind of reasoning is most of the purpose of the class; the concepts they teach you are in some sense just a foil for teaching this. The derivation from the concepts you were explicitly taught is pretty trivial and you've had many opportunities to practice.
This is common for any kind of systems-y domain. These types of questions were standard on my (physical) engineering exams.
As a kid I noticed that repairing things is the perfect way to combine experiential learning and "thinkism" - you have to develop a mental model of how something should work, what's broken, and how to fix it. Then you combine that with the physical sensations of how tight the nut is, or how hard you need to turn that wrench - which in turn feeds into the mental model and determination of next steps.
Screws and bolts have to be stretched to work. torque wrenches are used to enforce "just enough" tension without too much and that's why we use them. Over tensioning can break the bolt by snapping or strip the threads.
What's frustrating to me is that I learned to repair things, but there is SO MUCH practical knowledge to fix your iphone font size, but it is much harder for you to run into the real practical knowledge of life, like how to turn a nut/bolt/screw.
Wish there was a university of this stuff. how to use a wrench and when power tools. how to ask someone on a date. how to cook easily. how to choose cool clothes to wear.
The author is talking about two orthogonal problems.
1. "Thinkism": As described, over-engineering before writing code for a complex system and seeing where it takes you. Maybe decision by committee, or just overthinking. But its like one form of replacing on-the-ground adaptable, creative thinking, with a dumber process.
2. Which should be completely separate, it's saying that students are mad if they're forced to think for themselves. This is a complaint about underthinking and the tendency of inexperienced coders not to come up with a grand plan before writing a line of code.
So which one is the problem? I'd say the problem is not knowing when to over or under-think something.
Did not know of the "thinkism" expression. When I was studying in France eng. school, I called that "the mythe du cerveau" (literaly "the brain myth", though does not roll on your tongue as well).
It is guaranteed failure mode of large orgs. Curious to hear about more references on how to fight this at an organization level, besides the one given in the OT.
> Stop expecting an AI to cure all diseases or solve all problems just because it can read all the scholarship and “think” for a very long time. No matter how much an AI “knows,” it is always too little.
AI improves our problem solving in another way though! E.g. I can now quickly spin up frameworks to test/eval my hypotheses. This speeds up our ability to trial-and-error towards progress.
Some programming teachers think students should learn all the abstract concepts of programming and then write perfect programs. This is wrong. They should start with hello world, and then hello myname, and they should experiment and write programs that don't work, and after being frustrated for a little while with something just beyond their ability to solve, they should be shown the trick that solves it. Repeat until expertise.
Working experiments constrain our thinking from spiralling out. This is the true hindsight bais, a good one at that. When we have a working prototype, we can try to break it in many and find out lot about the underlying theory. Theoretical Physicists do this all the time with toy models.
I liked the article and the term thinkism (which I hadn’t heard before). I think education should be radically changed to be about doism instead. I think it’s likely we have more engaged kids learning more valuable life skills.
Reminds me of https://xkcd.com/927/ - and to avoid confusions: everybody has good intentions and think they know better.
We definitely should try to improve and experiment with any system, including education, but I really doubt it is that easy to improve education and it will depend on objective, culture and political environment more than doing A or B.
> As a teacher, I can tell you that students get really angry if you put a question on an exam that requires a concept not explicitly covered in class.
Well then I think you omitted a rather important topic in your teaching: that the purpose of teaching is to provide a toolkit with which the student can extend their abilities.
Thank you for this term. In my view, the belief that AGI singularly will rapidly destroy us because it will think 10,000 times faster than us is a form of thinkism.
Like everything there’s always a balance. Sometimes building something and seeing how it works might have a higher cost to “correct” once built. Other times, it’s much faster to build.
> As a teacher, I can tell you that students get really angry if you put a question on an exam that requires a concept not explicitly covered in class. Of course, if you work as an engineer and you’re stuck on a problem and you tell your boss it cannot be solved with the ideas you learned in college… you’re going to look like a fool.
Very flawed comparison. At work I get to go off and do research, experiments, can collaborate with peers and people who might have more expertise in a given sub problem, and generally have much more time. An exam trying to test you on material you haven’t studied is supposed to test for what? Your ability to synthesize knowledge out of thin air.
The rest of the article is well written and correct, but this particular aside felt weird.
I think it depends on the question. If it's not a question of the form explicitly presented before, but answerable with a minute of thinking using the knowledge the student has already mastered, then it makes sense.
A time limited exam is probably the wrong place for that, though, due to the stress interfering with that kind of thinking. It would be better for a homework assignment.
If ChatGPT didn't exist.
Okay, maybe in class, on paper is the right place for that.
The while premise of "learn some stuff them take an exam on exactly that stuff" is pretty flawed, and that's the point. So much of the academic structure is about what's convenient for evaluation, rather than what's best for learning. Why not get rid of the exam and replace it with something else entirely? Who says we have to have exams at all?
> So much of the academic structure is about what's convenient for evaluation, rather than what's best for learning
Academia is going to have to focus on the latter anyway because we have AI, which is exactly the same: it’s only useful when you can evaluate the output for accuracy independently. But how do you do this without throwing out objectivity entirely? Maybe we need to move to apprenticeships?
From my experience, the boss is usually a complete moron, so who cares. It also creates this unhealthy assumption that the engineer is subservient to the boss.
From my experience the boss does not know things you know, that does not make them a complete moron because they probably know things you don't.
Here's an example, consulting at a large Danish company, every Friday morning all departments in this big building would share breakfast and the bosses would say some things.
So this one morning they explained that in the coming months people should register time in a particular way because of accounting and how it related to a particular government grant and money that needed to be used up by a particular time in order to get to the next step of blah blah blah.
I realized as my eyes glazed over, damn this is just the same reaction people who don't understand browser rendering engines get when I start telling them about different events.
I also noticed other clueless people gamely trying to question these finance nerds on how things worked, and the patient finance nerds explaining some detailed bit and the clueless person clearly out of their depth with that "uuuuhhhh, hope they don't ask me if I understand" look on their face.
Now, if it hadn't been for them explaining this stuff I would have gone around thinking the boss is a complete moron. I once saw him mistake a nail gun for a drill! He doesn't understand how search engines work and why stemming and decompounding might be important, I know because I tried to explain to the idiot one time!! But since he actually talked about his work for a bit I realized he just happens to know stuff I don't.
I'm betting most of the morons you know are maybe not quite so stupid, although probably not as forthcoming as why things need to be done in a certain way to those who work under them.
yeah.
someone, somewhere, at some point, will think i’m a clueless idiot.
we’re all clueless idiots at the end of the day.
that's a great story and a remarkable level of self-awareness. thank you for sharing
I feel like it's an argument for the benefits of abstract reasoning. I don't think they are saying it'll be like that in the real world, I think they just want to test how you do under adverse conditions.
Stress testing the student's academic prowess, if you will.
>> Your ability to synthesize knowledge out of thin air
As someone who graduated high school, I'd hope my more accomplished peers would know the difference between hypothesis, theory and proof. It is entirely possible, and useful, to test someone's ability to form a cogent hypothesis. If you were faced with a question beyond the scope of the ideas you were taught, and could not rely on any assistance, the only useful thing to know about you is how well you would handle it yourself.
If you would synthesize knowledge out of thin air, that would be a failing grade.
> If you were faced with a question beyond the scope of the ideas you were taught
Who knows the scope of the ideas I was taught? I mean, I could use derivatives and sometimes to integrate in 8th grade, should I get a failing grade if I used them in a test?
I see the point of your post, it is important to distinguish between what you know, and what you don't know, but maybe suspect. School tasks are built in a way, that the very form of the task can be used to make good guesses that may help you to solve the tasks. It doesn't work in reality, but it works in school. But the solution is to add questions without answers, not questions with answers beyond someone's knowledge. Like "what was Mozart's opinion on pineapple pizza?" or "what size was Julius Caesar?". Well, tests needs less obviously unanswerable questions than these, but you got the drift.
Edit: Thought it will be tricky with math. For example: "Alice has 3 apples, and Bob has 2. How many oranges they have?". School math problems have a convention, which means that if oranges where not mentioned, then they have 0 oranges. Compare with "Alice has 3 apples and Bob has 2 oranges". If the problem doesn't mention Alice's oranges, then Alice has 0 oranges. It is implied and these implicit rules are the first difficulty kids face when starting with math.
I found the whole article to be a bit heavy on anti-academia. And I went to industry after undergrad.
It's a false dichotomy between the "thinkism" bogeyman (actually reading books and papers and putting work into theoretical design is just bad now? Have they tried building anything in the physical world? Checked in with nuclear physics, ever?) and hands-on experience. Both are important. It should be about balance, not trashing an incredibly valuable set of tools because others exist...
I am team academia more than hands-on experience. And I have 5 years of experience. To me, it felt like most SWE things could eventually be solved by what I learned at school.
Not everything I did I learned at school, such as navigating codebases with more than a million lines of code. But most things? Yea.
With that said, I am curious how people say that they learned much more through experience, what did you specifically learn?
I find that many people can learn a lot by doing but then at some point hit a wall and really struggle to recognize that another kind of learning needs to take place to understand a deeper concept.
Could you tell us what you learned at school that is useful for your SWE career? Im sure there is a lot of us that learned 0 at school and learned everything ourselves as kids on the internet
The problem with "learned everything ourselves" is that you might have niche interests and you miss things. Things I learned that probably I wouldn't have by done by myself: computer architecture (memory, buses, cpu-s, instruction set), and related VHDL/Verilog; how complex is synchronization (implementing from scratch synchronization libraries); different programming paradigms (functional languages); compilers & operating systems (kernel modules, etc.); various types of maths (dsp); algorithm complexity analysis.
Some I ended up using more during my career than others, but knowing more definitely reduced my tendency to think "ah, that should be easy".
Benefits of learning online yourself is that you're self motivated and you get access to the best resources (better than the teacher you are forced to listen to)
Edit: feel free to summarize this with an LLM it will be a ginormous comment.
That's a fair question and I'll do my best to answer this. It'll come in an edit. I think it's fair to say: not all courses were created equal in this regard but I'll do it course by course. I studied a bachelor information science but I tweaked my program so close to computer science that I almost daresay it's computer science (if I had 3 courses different, it was). I studied a bachelor in psychology. A two year master in computer science and a one year master called game studies (officially a specialization of information studies, but in practice it wasn't and it really was game studies as a whole field that we studied).
I'll try to do it in order per study program too.
AI-kaleidoscope: general overview of AI algo's. It's a shame we didn't know how to program or that we knew the usefulness of BFS or DFS but we learned it here. Not a useful course due to scaffolding issues (teach programming first).
Business mathematics: partial derivatives, etc.
Problem solving: useless course (teach programming first).
Privacy and security: we didn't learn much about security. We learned a thing or two about privacy. Should've been a TED talk, not a course.
Graph theory: graph visualization (and when not to do it, helped me out as a data analyst later), mathematical proofs, social networks (helped me to actually network a bit), graph algo's (made leetcode easier), not being scared of math notation. This was a really math heavy course as it was taught by someone that studied math in undergrad and grad and then switched to CS as his PhD (and by the time he taught it he was a full professor). He did not skip on the math which was wild since I was under the impression that I did a "business informatics" bachelor so I didn't need to have advanced math as a high school prerequisite. But I definitely needed that here so this course was hard.
Web technology: this was a bit too early but it was a good overview of web programming at the time. I remember being explained what the DOM was and I remember thinking "wtf is the point?" That was because they explained it way too theoretically and simply should've opened firebug or something to really show it.
Language of Logic and Methods of Reasoning: propositional logic and predicate logic. Practically speaking: after this course if statements are not a problem. This was true for me at least.
Pervasive computing: fun course but could've been a TED talk about how tech is used in interesting ways.
Introduction to programming: basic programming stuff in Java. I learned that Java is a terrible language to start programming in. I recommend JavaScript for app/web-oriented people and Python for "just pick a language" people. Nevertheless, while it was a terrible start, it did teach what it needed to teach which was a basic understanding, and skill, in programming.
Empirical methods: the better name is statistics 1. We learned about statistics and we had to program in R. Since our actual programming ability was quite weak it was a double course. It was the second language ever that I had to take seriously and it taught me a lot of programming stuff and statistics stuff at the same time. It helped that I had friends studying psychology at the time as that degree has a lot of stats in it, so I knew the lingo. Almost everyone else was hopelessly lost.
Programming project: no lectures, just one big programming assignment. After this it was expected that the student could write readable code. I failed this course the first time by a hair. And when I came back to it the second year I realized that I failed because my code was unreadable. So I refactored the whole thing and learned how to write much more readable code. We created the game engine for an Othello game. The graphics library was provided by the TA's. So we also learned to program with a library that was way beyond our heads at the time. And that also implicitly teaches you to trust certain abstractions.
Interactive multimedia project: we learned XIMPEL a hypermedia framework, kind of useless. But through XIMPEL we also learned about storyboarding, creating scenes and general video editing. This teacher was super hands off and allowed students to be creative. So I also learned to create simple PHP websites and learned my way around bash a bit. Then I decided to create an upload script where I used PHP to call the ftp command on bash. I thought it was impossible but it wasn't and had to rethink about what web applications could really do. None of these things were formal course requirements but this teacher encouraged this type of explorations so I do credit it to him that "I learned it in school". It's part of these efforts that he also gave me an amazingly high grade as my XIMPEL story graph was a bit meh but my creativity and extra explorations where a 10 out of 10 effort. So he gave me a 9 out of 10 in total. Also deepened my HTML/CSS knowledge.
-----
I'll write the rest later in another comment this is about the first year and I studied 9 of them. As you can see, I was just learning the basics here. But there are a few patterns:
* Some teachers allowed us to work on real stuff if the student chose to (e.g. Interactive Multimedia) and that got way more serious later on as I created an iPhone app for a client when the teacher taught Multimedia Authoring in the master.
* Some courses were quite useless or could've been condensed to a TED talk.
* Some courses taught something useful or semi-useful but it's not up to industry standard. But as you'll see, this will lead up to a level where - while not quite industry standard - makes the gap between industry standard and whatever I did small enough to just make the jump easily by simply applying what I thought was common sense.
Interesting ! What is the name of the schools you attended, if that's not too private ?
In France at "prepa" (2 year intensive courses to prepare for exams for big eng uni) I learned the theory behind computer science (for example how to modelize a regex machine with graph / automata / matrices). That was useful theory to me, but that's just a drop in the ocean of uselessness
Vrije Universiteit Amserdam
A lot of technical skills sure. Algorithmic analysis sure (although in practice it’s a lot less formal and more intuition + rote knowledge.
But taste about the best ways to structure a codebase? How to balance speed vs quality and when to do that? What does good vs bad observability look like? How to handle services going down in production and run post mortems? How to handle ambiguity both at a leadership and technical level to chart a path forward for a team? Even reading about these things from blog posts and things is nothing like actually experiencing these things first hand and certainly not taught. It requires accurate and constant self reflection and retrospective to guide an exploration process and if that self reflection tool is askew you can end up learning weird lessons and habits that are difficult to correct (and we all have them somewhere).
I’m team learning and was team academia early on, but the real value in academia is in teaching you how to think and to give you some minimal useful knowledge to about that capability. But it often struggles to even that because it’s still following an assembly line model of teaching whereas learning is relational.
Source: nearly 20 years as a SWE from a well regarded university.
Every company structures a codebase differently. So it’s best to be open. I have worked at many places where speed vs quality didn’t come up. Also it depends on the constraints. A HFT company is much more worried about getting cache hits whereas a big tech company seems more concerned about a big n. Good vs bad observability: well you better know your statistics well, otherwise you lull yourself into a false sense of security. Not my strongest area tbh. I haven’t worked at a company where post mortems are needed.
Ambiguity: there is a lot of ambiguity at uni as well. So you should have had the practice, especially working in groups of students.
> It requires accurate and constant self reflection
Skills I sharpened at university by studying psychology. Do you know how big the mind fuck was to grok the reproducibility crisis while being a 3rd year student in that program (5th year student overall)? I had to re-examine my whole degree.
I also had courses where self-reflection was a weekly part of the course. And I took that seriously.
> whereas learning is relational
I consider it a mode.
Try mathacademy.com it will teach you the “shut up and calculate” version of math really well, and there is nothing relational about (since it’s an app).
I also had courses that were about computer and network security in C and assembly. In it, I self-taught vim, assembly, C and git. The self-teaching was partially the point.
One thing I have learned from university is that what many people see as good practices is more akin to a culture rather than an actual good practice, because the actual empirical work in figuring out what works hasn’t been done. And I find what you say to strongly fit a particular culture of software engineering. But there are many forms. For example, in the Smalltalk/Pharo community it is recommended to extend the base classes to your liking. Whereas in other languages they look at you with horror. Back in the day (2012) Airdrop was what people used and didn’t want to use git, and these people have programmed for decades (I disagree with that one personally, just showing the culture).
Theoretical design is pretty worthless until you try it out. Until then it’s just an idea or hunch you have. You hope that your experience and intelligence will tip the scales in your favor but that’s all it is. And often your design still has flaws anyway because there’s bound to be unknowns. So then your hoping some kernels of ideas still hang on after encounter with the world and that the ideas are directionally resistant that you can keep the design going forward.
Design is useful as a tool to sketch out a plan but it can definitely become thinkism if you always hand off your design to someone else to action.
Also, the author is talking specifically about research type problems where there’s a lot of unknowns and the problem space is not well trodden and understood.
Also, I don’t know when the last time you built anything is, but talking with mechanical engineers and construction engineers, it’s all the same. You have your designs and then you open up a building to do repairs or start construction and some surprise presents you that you weren’t expecting. That’s what the research phase is for where they learn all they can about the problem space before they start designing (in this discussion ignoring all the years of trial and error that built a base of experience already) and even still in practice problems arise.
You have your designs and then you open up a building to do repairs or start construction and some surprise presents you that you weren’t expecting.
And yet, the field of architecture is still highly valuable. Without those designs, there would be a lot more problems. Try building something complex without a design. You're going to waste a lot of time and materials. My point there was that it's not like the digital world where mistakes and trial and error and rethinking things can sometimes cost almost nothing.
Theoretical design is pretty worthless until you try it out. Until then it’s just an idea or hunch you have....ignoring all the years of trial and error that built a base of experience already
Lots of (smart) people have tried lots of things, there is a ton to learn from that, you're calling it worthless and taking it as an assumption at the same time.
I've had a research-heavy career in computer science. A central problem with the academic research is it commonly ignores real-world constraints. Or less commonly, it imagines constraints that don't exist. As someone who went deep in a few domains, academic literature is usually disappointing[0] once you've read and understand the entirety of it.
The valuable thing about hands-on experience is that the real world doesn't let you ignore constraints. You get that feedback quickly if you are paying attention. This in turn allows you to build a more accurate mental model of the true nature of the problem you are solving and where the hidden limitations and leverage points are. A lot of academic literature tacitly works from a set of assumptions that don't map to any real-world environment.
Once you have that hands-on mental model, the flaws and limitations of much of what is in the academic literature becomes obvious from first principles. Most of the insights might be academically interesting in a theoretical sense but they often aren't reducible to useful practice in real systems. Non-academic careers require implementations that actually work well.
[0] The computer science literature from the 1970s and earlier is much better in this regard than what came later. Many early papers were written by people that clearly had concrete experience in the trenches with the problems they were writing about. Those papers are both more readable and more applicable. This awareness of constraints is lacking from a lot of modern computer science papers.
I agree that it's a flawed comparison but it does touch on a very real issue in the workplace. The difference between employees who I can send away on a problem and they'll encounter issues and try to solve them, vs those who come back to the senior at any problem, presents a serious ceiling on the level those employees can work at.
Perhaps the example should be a homework assignment, not an exam.
Even then I would call it bad teaching, as the usual mode is:
New stuff gets explained in class by the teacher - the details are for self study. This is how it was communicated to me in school, so I would call it part of the "contract".
But giving a homework on a fundamental new subject without explicitely telling that this is what it is about, will rightfully confuse, as the normal mode is different.
And when it happened to me, it always happened because the teacher messed up and got confused themself what they had and have not teached before.
Teaching to self study is a valuable skill in itself that should not be taught by accident, but with a clear pedagogic concept.
As a leader, I can just tell people literally that: if you can't go away and work independently on a problem, it puts a ceiling on how well you're going to do. Then I ask people what they've tried before asking me.
Just being upfront with people can break low performance patterns of behavior.
I think it’s largely there to set up the point that comes after, which is that it would be absurd in a professional setting to pronounce a problem unsolvable because the entirety of your university education doesn’t provide enough information to solve the problem.
I took an exam once and there was a question on it that wasn't covered. I had vaguely remembered something from somewhere else and was able to piece together the formula and actually got the correct answer. When the test marks came back, the prof apologized about the question and didn't count it in the grades. Myself and two other people in the class who got the answer were held after class and accused of cheating. That is the school system. They do not reward creativity or ingenuity.
Sounds like the synthesis problem could be the difference between a 8-9/10 and a 10/10 score. It would work in an accuracy grading system instead of a hedge against stupidity system.
I grew up poor. I thought my ticket to not being poor was an education. By the time I was applying for colleges (I had good grades) I realized that the education system was a joke.
I got through college, a very well-regarded engineering program, putting in the bare minimum effort to graduate.
I make a lot of money now, 20 years later. I attribute none of my success to public education or college, I didn’t learn anything besides our education system (in the US) is not what they tell kids it is growing up.
We beat the drum of education because it seems like a nice simple solution to fix a lot of perceived problems. “Oh you mean if we just throw money at this problem it’ll fix it?!”
Baltimore city gets some of the highest per-student federal funding in the country. My daughter is in a public middle school there, living with her mom. I’m in the burbs with my wife and kids.
She told me just yesterday she wants to go to high school in my district because “well it just seems more normal than city schools”
She also thought California was a country up until yesterday. Straight A student.
At some point, probably too late, we’ll realize we did it a wrong. Floating kids through high school, dropping an anvil of debt around their neck for a college degree they won’t use. Met a fellow the other day, he has his masters in social work. Making very, very good money, as a regional manager of a car sales group, I forget the specifics. Said he never used his degree and he is very grateful he has a clear path to pay the debt off.
I don’t know what the fix is, but I’ve seen enough to know what we do now, ain’t it.
> We beat the drum of education because it seems like a nice simple solution to fix a lot of perceived problems. “Oh you mean if we just throw money at this problem it’ll fix it?!”
Its strange to say, but its actually thanks to black racism is why we're here.
It started with https://en.wikipedia.org/wiki/Griggs_v._Duke_Power_Co.
The result was that the courts determined that "aptitude/IQ tests" unfairly discriminated against black people. However, education requirements are not seen as such.
So, we get "decent paying job" requiring 'a degree'. What degree? Doesnt matter. The degree is the legal form of 'we dont hire blacks'. It also has the side effect of 'we dont hire the poors' as well.
Obviously there are positions that require advanced schooling. Im not talking about those.
It is an article of faith on message boards that IQ tests are somehow unlawful in employment in the US because of Griggs. That's patently not the case. Plenty of large corporations with deep pockets do general cognitive testing (and even IQ testing) of incoming applicants. The companies that administer these tests have logo crawls, like every other product company, and the names are companies your mom will recognize.
Griggs says that you can't employ IQ tests to discriminate for jobs where the IQ threshold you established is immaterial to the job. For modern knowledge work, that's a trivial bar to clear. It is essentially the case that Griggs is a nonfactor in white collar employment.
My motivation for calling this out isn't that I'm OK with discriminating against Black people; quite the opposite, in fact. IQ tests aren't used more widely because they aren't effective at qualifying employees. IQ is an idee fixee among A Certain Part Of The Internet, and part of their pitch is that IQ tests are a secret cheat code that would enable hiring purely on merit, if only the woke wouldn't prevent it. No, false.
> Myself and two other people in the class who got the answer were held after class and accused of cheating.
Wow. I was never able to escalate situation to the direct accusation of cheating. I just got my marks lowered a bit because I used methods that were "above my abilities", as my math teacher believed. Though really I thought school math was boring and tried hard to make it more interesting, less routine, less relying on memorizing things, and more clever shortcuts from general principles.
But she never accused me directly. I know her explanation for lowered marks just because my mother visited her once with a specific test results and asked why marks are so low. I wanted a direct accusation, I was trying to provoke her to doubt my methods, but either I tried too hard and she knew deep inside of her the real situation, or I needed to do something truly exceptional and failed.
Yeah, thinking about it now, I should probably try some existing and very known methods but far above the school level, something she felt herself familiar with, so she could hope to prove that I've used methods without understanding them. And then it would come to my ability to prove, that I understand my methods. Something like raising e^(ix) into n-th power, to derive a formula sin(nx) = f(sin(x), cos(x)). Though, it is might be too weird for her.
> They do not reward creativity or ingenuity.
You can stimulate yourself if you look at it as a rebellion against the system. Teenagers are good at rebelling. Though, rebellion can make matters worse. I rebelled against history trying to get the lowest marks possible. It didn't end well, and I not just didn't learn history, I've got a life-long trauma, I start to regress to my teenager mind level every time I face history.
That's infuriating to me, and I wasn't even the victim!
I'm so lucky such a thing never happened to me. The closest thing was a math teacher in middle school asking me about a problem solution of mine in a test, that he had marked "mysterious" (I had to come up with my own solution path during the test because I hadn't memorized the canonical one we were taught). He asked me to explain my reasoning, and when he was satisfied that it was sound he gave me full marks for the question.
Nope, and nope. I've had tons of such questions during university in various courses, so what? Are they supposed to to babysit you and handle you only exact stuff they went over and over, to not upset some fragile minds? Such stuff is part of educational life. They expected to synthesize the knowledge out of other, related knowledge that was very directly covered, and use a bit of if-this-then-that. All in trivial manner if you actually grokked the subject well enough.
I failed some, did well on others, felt a bit of unfairness emotions completely understandable for that young mind which just wanted to do the test and move on. But when looking back, they were good questions, testing my actual skills in topic I was studying.
Absolutely. I did this just last week. I couldn't figure out why I wasn't able to get out application to use a fileserver in a new test environment. I brought it to the team and got help from someone else.
> As a teacher, I can tell you that students get really angry if you put a question on an exam that requires a concept not explicitly covered in class. Of course, if you work as an engineer and you’re stuck on a problem and you tell your boss it cannot be solved with the ideas you learned in college… you’re going to look like a fool.
I really hope this kind of thinking dies away because the longer you wait the more money you waste, not just your own salary but the opportunity cost of time. And this person is a teacher?
You have a deadline and scope (of solution) handed to you from product/management. It’s the same issue now at work.
Maybe the impression cones from a different understanding what "concept not explicitly covered in class" means.
Of course you would not test a completely unrelated concept but where I grew up "Transfer of Learning" is supposed to be part of every test, with a proportion that increases as students progress through school.
To test pupils ability of transfer of taught concepts means to let them apply information, strategies, and skills they have learned to a new situation or context.
My problem with my country of origin’s school system (Hungary) was that this “transfer of learning” was used solely during exams. It would have been really great, that after doing only simple, straightforward exercises during classes or home work, not the exam would have been the first time when students had encountered real world, way more complex exercises. (Btw the other really annoying exam type was when they tested your raw speed of recollection, in other words, whether you did that particular type of exercise a thousand times, and not that you understood or could you do it, like we would have practiced to ER or something where muscle memory can be life or death)
supposed to test for what? Your ability to synthesize knowledge out of thin air.
Yes, actually. Synthesis is the penultimate level of Bloom's taxonomy [1]. Perhaps not appropriate for a first quiz on new material, but absolutely in scope for a final exam.
I had many exams in university that challenged me to synthesize a proof to a hypothesis I'd never seen before. I had to bring together my knowledge of the material I'd learned in class with the bit of new material presented in the problem statement, and then devise a proof with one or more steps I'd never done before.
Many of my fellow students in first year struggled mightily with questions like this. There was much wailing and gnashing of teeth after the exams. In later years we'd all gotten used to it, and it was to be expected. That students in high school or younger years have been sheltered from synthesis-type problems on exams is a great disservice to them.
Edit: I should also note that outside of upper-year pure mathematics courses, synthesis-type problems rarely accounted for more than a third of the grade on an exam.
[1] https://en.wikipedia.org/wiki/Bloom's_taxonomy
Synthesis problems and time pressure don't feel fair together.
I'd be furious to get some lofty theoretical synthesis problem on an engineering exam where the other problems are grindy analysis of "looks like concept you know but expand into a form you haven't seen and then will take 1/n time on your n question exam". I suppose these are already 1 extension in, so 1 more logical extension isn't that far off
They're absolutely fair if you're given plenty of practice at them. If the only synthesis problem you see all year is on the final exam then you have a point. However, if you're doing novel (to you) proofs on assignments every week throughout the course, you should be used to the novelty by the time you reach the exam.
Expecting a final exam consisting only of calculation and memorization type questions is wholly inappropriate in a mathematics course full of proofs.
I think you are both right and it really depends on the leap that has to be done during the time constraints of the exam and how much practise was included in the curriculum.
Off-topic, I know, but I always disliked the time pressure of exams. All the bad scores I got were down to not feeling quite right at the time of the exam: maybe I didn't sleep well, the food I ate did not quite sit with me, had a fever, etc. None of these things actually impair too badly but if the time is short, there is an impact.
I suppose it's similar to sports where you have to, at least somewhat, adapt on the go. Many teams botch a game here and there. But... your final exam? Botch it and you're screwed. It just does not feel right.
it depends on what you mean by fair, i think. is the purpose of the test to gauge how well a person performs on tests? the difficult problems i encounter in the real world involve collecting a bunch of context on it, sitting with it for a bit, then putting it down and going for a walk. when i come up with something to try it's usually when i am outside and looking at trees.
maybe this is good for weeding out students who would not be a good fit for high stakes speed math.
It's really not high speed math. You'll have 2.5 hours to solve maybe 5 problems on the exam. A math grad student who TAs the course can typically complete the exam in 45 minutes or less, without any preparation whatsoever. An undergraduate student who can't solve the exam in 2.5 hours (without external aid) is not going to solve it in 10 or even 20 hours. They simply do not know what they're doing at that point.
It's really no different from a software dev interviewee who can't solve basic programming questions without being handheld and coached through the entire process.
I think if TA no prep completion time to total time ratio is above 2.5 it's not the time pressure based exams I'm referring to.
I didn't get those exams in math courses. Engineering courses I did get those where some of the point of the test is the speed of your pattern matching ability (did you use design skills to reason about it or did you brute force a solution).
Math you're already extrapolating theory and you need ample time for that.
If you can't do synthesis problems on a exam, you are not prepared for it.
I quite fondly remember a situation with a two question three hour open everything exam where I stared at the 1st problem and relevant equation for 20 minutes because I had no idea what I was looking at and was wondering if I should just leave and drop out.
Then I realized that a bunch of terms could cancel out and the problem became 3 lines trivial. The professor later admitted he wanted to give a "easy" question that you couldn't just copy out of your notes and had to think and to spend the rest of the exam on the actually difficult one.
This seems like you proved my point by staring at it for 20 minutes before starting. I don't think it's fair to grade with extreme time pressure and also expect deep thought.
Being able to do deep thought quickly isn't a valuable skill, and you don't pick it up on the way ti mastery with lots of practice like you do the speed for rote problems
No the literal oppposite. The exam was "no deep thought required. Need to know the subject and its trivial"
> Synthesis problems and time pressure don't feel fair together.
If you really want to grade the students its fair to put very hard problems on the exam to separate out mastery from merely very good understanding or recital.
Theres is more than ine dimension to mastery.
What skill are you grading? Can you expect some students to be able to churn out the problems? Are they doing design work? Is it more inventive or more creative? Is there more than one solution? How rigorous is the solution?
My point is that you can design an exam to test for speed and ability to do the day to day well. You prepare for this exam with lots of practice. There are a few problem types but you might not have seen every one. There's a basic level of extrapolation you're expecting.
You can design an exam for abstracting theory, and you prep for this exam by doing the hardest problems you can find, reading more theory, etc. You might not have seen anything like this but the test is to use the tools you're given in a class well. The test is the extrapolation.
I had theory courses where I expected the latter and practical courses where I expected the former.
There's more than one type of mastery; I bet your math professors would get destroyed by gifted 14 year olds in algebra competitions that involved arithmetic and timing. Do those 14 year olds not have more mastery?
Hard problems introduce noise to the grades. A student who gets a hard problem right could be good or lucky. In order to grade the students properly, you need multiple problems of similar difficulty. If you have both ordinary and difficult problems in the exam, it's probably long enough that it should take the entire day.
Undergraduate exams tend to be short. Which means that a perfect grade should be interpreted as "meets the expectations".
Which means that a perfect grade should be interpreted as "meets the expectations".
None of the mathematics exams I wrote during undergrad at Waterloo (as a math major) would fit that description. Nearly every single one of them had midterm grades with unimodal distributions centred below 70%, tending toward 60%. Typically, only 1-5 people in the class (of 100-200) would score a perfect grade. In upper year pure mathematics courses it was common to not have any perfect grades (in a class of about 20-30).
Mathematics is notorious for exams like that. But if you look at the reasons why people fail to get a perfect grade at undergraduate level, it's almost always due to honest mistakes or because they didn't learn what they were supposed to learn.
In my experience, studying mathematics is a bit weird. If you are ready to learn a topic, it's probably the field where you can get top grades with the least effort. But if you can't learn something with reasonable effort, hard work is unlikely to help. Doing something else and trying again after a few months might help.
Not of thin air, but of what you've studied so far. Of course some (many) teachers are imbeciles and just use the same exam they've used in the previous year, even if they haven't covered all the topics yet. But in general, it's a very good idea to have a small question that tests for real understanding, i.e. applying your knowledge to a problem not previously presented, versus mindless reproduction of memorized facts.
> Your ability to synthesize knowledge out of thin air.
The skill the class is teaching is the ad hoc synthesis of knowledge from other building blocks. That kind of reasoning is most of the purpose of the class; the concepts they teach you are in some sense just a foil for teaching this. The derivation from the concepts you were explicitly taught is pretty trivial and you've had many opportunities to practice.
This is common for any kind of systems-y domain. These types of questions were standard on my (physical) engineering exams.
As a kid I noticed that repairing things is the perfect way to combine experiential learning and "thinkism" - you have to develop a mental model of how something should work, what's broken, and how to fix it. Then you combine that with the physical sensations of how tight the nut is, or how hard you need to turn that wrench - which in turn feeds into the mental model and determination of next steps.
> how hard you need to turn that wrench
And learning what stripped threads are....
Part of learning to repair things is also learning how to not make them more broken.
it helps to back the model with the theory.
Screws and bolts have to be stretched to work. torque wrenches are used to enforce "just enough" tension without too much and that's why we use them. Over tensioning can break the bolt by snapping or strip the threads.
What's frustrating to me is that I learned to repair things, but there is SO MUCH practical knowledge to fix your iphone font size, but it is much harder for you to run into the real practical knowledge of life, like how to turn a nut/bolt/screw.
Wish there was a university of this stuff. how to use a wrench and when power tools. how to ask someone on a date. how to cook easily. how to choose cool clothes to wear.
The author is talking about two orthogonal problems.
1. "Thinkism": As described, over-engineering before writing code for a complex system and seeing where it takes you. Maybe decision by committee, or just overthinking. But its like one form of replacing on-the-ground adaptable, creative thinking, with a dumber process.
2. Which should be completely separate, it's saying that students are mad if they're forced to think for themselves. This is a complaint about underthinking and the tendency of inexperienced coders not to come up with a grand plan before writing a line of code.
So which one is the problem? I'd say the problem is not knowing when to over or under-think something.
I call this, the way to learn stuff is by doin' stuff.
Also buildin' stuff! (Which is the best type of doin'.)
Did not know of the "thinkism" expression. When I was studying in France eng. school, I called that "the mythe du cerveau" (literaly "the brain myth", though does not roll on your tongue as well).
It is guaranteed failure mode of large orgs. Curious to hear about more references on how to fight this at an organization level, besides the one given in the OT.
Yea, we just name things that we want to see destroyed...
Not everything need to be made so easy to refer, like using three or four of words instead of one..
try replacing the word with 'thinking'
> Stop expecting an AI to cure all diseases or solve all problems just because it can read all the scholarship and “think” for a very long time. No matter how much an AI “knows,” it is always too little.
AI improves our problem solving in another way though! E.g. I can now quickly spin up frameworks to test/eval my hypotheses. This speeds up our ability to trial-and-error towards progress.
Game developers are the best at this sort of stuff (especially valve and puzzle game designers)
Portal (a puzzle game by valve) had levels built in such a way that it introduced the player to new mechanic, and only then building on top of that
Some programming teachers think students should learn all the abstract concepts of programming and then write perfect programs. This is wrong. They should start with hello world, and then hello myname, and they should experiment and write programs that don't work, and after being frustrated for a little while with something just beyond their ability to solve, they should be shown the trick that solves it. Repeat until expertise.
Similar to the ideas in ‘On Practice’. A dialectic is required between practice and theory. https://archive.org/details/onpractice00maoz/mode/1up
Working experiments constrain our thinking from spiralling out. This is the true hindsight bais, a good one at that. When we have a working prototype, we can try to break it in many and find out lot about the underlying theory. Theoretical Physicists do this all the time with toy models.
I liked the article and the term thinkism (which I hadn’t heard before). I think education should be radically changed to be about doism instead. I think it’s likely we have more engaged kids learning more valuable life skills.
I agree that doism should done more in all honesty
Cuz in real life also its more about "doing" you're physically fixing a clock, or writing code, or designing a building
Doism shouldn't be 100% but it certainly should be more
Reminds me of https://xkcd.com/927/ - and to avoid confusions: everybody has good intentions and think they know better.
We definitely should try to improve and experiment with any system, including education, but I really doubt it is that easy to improve education and it will depend on objective, culture and political environment more than doing A or B.
I think it was Hamming who said that some of the best researchers can only think well when they have a bench full of equipment in front of them.
> As a teacher, I can tell you that students get really angry if you put a question on an exam that requires a concept not explicitly covered in class.
Well then I think you omitted a rather important topic in your teaching: that the purpose of teaching is to provide a toolkit with which the student can extend their abilities.
Yes. There's an adage in medicine. See one, do one, teach one. I think it applies generally.
We humans have a really good system to do this in practice. We use _imagination_ to simulate how things work and ... then we understand it.
> thinkism
Thank you for this term. In my view, the belief that AGI singularly will rapidly destroy us because it will think 10,000 times faster than us is a form of thinkism.
Like everything there’s always a balance. Sometimes building something and seeing how it works might have a higher cost to “correct” once built. Other times, it’s much faster to build.
this misunderstands whats thinking is ..
> Thinkism sets aside practice and experience
thinking succeeds experience & precedes practise, its not apart from it
Nobody said it better than von Neumann: "Young man, in mathematics you don't understand things. You just get used to them"