Excellent! Thanks. I followed the links on Github, btw. I'm looking into some of the trainings you're offering.
Last August, I took the Advanced Windows Exploitation course from Offensive Security (for the OSEE), and then I followed it up with the awesome Advanced Fuzzing and Crash Analysis course taught by Richard Johnson. Both were incredible courses, but my RE sucks.
I'm looking at the Advanced Browser Exploitation course next, but I'd really like to get better with reversing in order to get more out of these classes.
I'll admit I haven't had the chance to look through it at all yet.. in fact, I'm typing this comment just I took a glance at the table of contents. It's a small thing, but it kind of irks me that "Final Notes" following "0x509 ImplementingPlayer" is indexed as 0x510 rather than 0x50a
Anyway, I'm going to save this for later viewing.. very interesting post.
Always thought reversing is fun. Back in high school and college I would spend days or weeks trying to reverse random software packages I had installed on my system. It was pretty surprising how, with a few tools like OllyDBG, 90% of the software I attempted to crack was actually crackable. Most of Autodesk's software was ridiculously easy to crack. (Don't worry, I owned all that software already! I was studying animation at the time.)
I didn't realize this until later, but I got a skill out of reversing that's actually pretty useful as a software engineer. Granted, I work with JavaScript, but reversing actually taught me how to persevere with debugging. I've been able to fix some bugs that other people couldn't in a reasonable timeframe because I treated the problem like I would when reversing a program; it can be more expedient to perform a process of elimination by disabling or inverting parts of the code, observing what happens, keeping detailed notes on observed behavior, and lead yourself down to the key change you need to make. A lot of people debug problems by trying to look at the code and reason about it while setting breakpoints, but there can be a lot of mental overhead in doing this. When the problem is mysterious, I find that a process of elimination through experimentation can be more effective.
This actually happened recently, where we essentially had an infinite feedback look happening that was obfuscated by a lot of framework code. It's possible that the root cause could have been identified by debugging, but a few people tried to do so but couldn't figure it out. This is because setting a breakpoint somewhere in the process chain didn't reveal anything obvious, and you'd be lucky to actually spot the problem in the application code just by looking at it. I treated the problem like reverse engineering, and did exactly what I described above. It took me a few hours, but I did succeed in narrowing down exactly where in the code things were going wrong.
Sure, someone could have stepped through the app and framework code enough times that they might have figured it out. Perhaps there are also more advanced debugging techniques that none of us were aware of that would have helped. But the mindset of a reverser definitely works.
Probably closer to "mutation analysis". (or "creative tampering"?) Someone smarter than I am could probably reverse engineer while doing less of that, but I've found that it can be effective when knowledge of a system is poor. (and there's urgency that warrants it over traditional learning)
Yes, I have used it at my previous job (recently got fired) and it was really helpful. Of course, you need physical access to the application to make this work, e.g. it's much harder to do for some oddly behaving REST Api.
It seems the majority of RE-related stuff these days focuses on malware but the skills are very useful for other things, particularly debugging in general. I didn't specifically learn RE, but since I started long ago with Asm, I guess it came naturally. You can tell the author of this one probably comes from an HLL background because of the "0x" prefixes; those with an Asm background are more likely to use a "h" suffix instead.
For Windows debuggers I recommend Windbg, which is free and comes from Microsoft. Using VS for Asm-level debugging is only mildly better than using gdb for it --- i.e. very awkward and not what they were designed for.
I'll take this as an opportunity to plug some other RE tools:
IDA is considered the holy grail. A base version is free, but its most popular feature, the disassembler, is $$$. Does debugging and static analysis.
NSA's Ghidra is great too: https://ghidra-sre.org/. last time I checked I don't think it did debugging but they were adding it. It's great for static analysis though, it has a great free disassembler (traditionally IDA's domain).
Cheat Engine is pretty amazing too, despite the goofy name: https://www.cheatengine.org/. It's the best tool for pointer scans imo, but also does debugging and is scriptable. Popular for making video game trainers, but it can be used for anything.
I used to dabble in Cheat Engine, Tsearch, and IDA Pro, for reversing games. Recently, a nice guy randomly showed me a demo of a modern tool called radare[1]. I was floored by the IDA-like flow-diagrams that it spit out in the terminal[2], and the short commands for navigating/bookmarking/documenting. Nice to see hobbyists don't need to buy IDA anymore!
SoftICE still holds a super special place in my heart, but WinDbg is infinitely more powerful these days than SI ever was, especially if you're using it from another box (or from your host with a Windows guest VM).
> If 0x12345678 was loaded into a 64 bit register such as RAX, then RAX refers to 0x12345678, EAX refers to 0x5678, AX refers to 0x78, AH refers to 0x7, AL refers to 0x8.
I made a PR to correct this example which equated single hexadecimal digits with whole bytes.
Other than that what I read of the course is rather nice, targeted at a mostly beginner audience with some correct insight.
There also some weird insight:
> You can think of computers as trains, they don't stop and only go in a very specific and direct path as designated by the tracks. If there's a child on the tracks it's up to the people controlling the track to divert the train. This is why Windows gives you the Blue Screen of Death (BSOD) when there is a kernel error. If the OS doesn't stop that error, catastrophic damage could occur.
This isn't really wrong, it's just a non-obvious metaphor formulated strangely in my opinion. A BSOD would be more akin to the fully automated train just blowing up in sight of the children or something? It's weird.
I think the author is referring to catastrophic damage in the sense of file/data corruption and loss. In that light a BSOD makes sense as a way to "stop the error" since it halts execution rather than silently continue. I agree that it's strange, though.
Hmm, I've been itching to dive into some security thing again (I'm getting into the habit to dedicate 1 to 2 months per year for it, this would be year 4). This person's effort seems so awesome and genuine that I'm up for helping out.
Though, I don't want to study it completely by myself, it gets a bit lonely and there's no accountability. Does anyone want to set up a study group with me?
My background: I know how to reverse Linux binaries, albeit I'm a bit rusty. I followed a course called binary and malware analysis at the Vrije Universiteit Amsterdam, and two other related security courses (shout out to Herbert Bos and his team for giving such an awesome course).
Would love to have someone to study with - I'm currently going through Reverse Engineering 4 Beginners by Dennis Yurichev but will happily try this one out too.
>Learning your first calling convention is like learning your first programming language. It seems complex and daunting at first, but it's really quite simple.
I will definitely read this as I'm also doing the Reverse Engineering for beginners, as well as getting sidetracked to learn Assembly with Programming From the Ground Up first.
Will Wright defined the "Simulator Effect" as how game players imagine a simulation is vastly more detailed, deep, rich, and complex than it actually is: a magical misunderstanding that you shouldn’t talk them out of. He designs games to run on two computers at once: the electronic one on the player’s desk, running his shallow tame simulation, and the biological one in the player’s head, running their deep wild imagination.
"Reverse Over-Engineering" is a desirable outcome of the Simulator Effect: what game players (and game developers trying to clone the game) do when they use their imagination to extrapolate how a game works, and totally overestimate how much work and modeling the simulator is actually doing, because they filled in the gaps with their imagination and preconceptions and assumptions, instead of realizing how many simplifications and shortcuts and illusions it actually used.
>There's a name for what Wright calls "the simulator effect" in the video: apophenia. There's a good GDC video on YouTube where Tynan Sylvester (the creator of RimWorld) talks about using this effect in game design.
>Apophenia (/æpoʊˈfiːniə/) is the tendency to mistakenly perceive connections and meaning between unrelated things. The term (German: Apophänie) was coined by psychiatrist Klaus Conrad in his 1958 publication on the beginning stages of schizophrenia. He defined it as "unmotivated seeing of connections [accompanied by] a specific feeling of abnormal meaningfulness". He described the early stages of delusional thought as self-referential, over-interpretations of actual sensory perceptions, as opposed to hallucinations.
RimWorld: Contrarian, Ridiculous, and Impossible Game Design Methods
>Tip 5: On world building. As you know by now, Will's approach to creating games is all about building a coherent and compelling player experience. His games are comprised of layered systems that engage players creatively, and lead to personalized, some times unexpected outcomes. In these types of games, players will often assume that the underlying system is smarter than it actually is. This happens because there's a strong mental model in place, guiding the game design, and enhancing the player's ability to imagine a coherent context that explains all the myriad details and dynamics happening within that game experience.
>Now let's apply this to your project: What mental model are you building, and what story are you causing to unfold between your player's ears? And how does the feature set in your game or product support that story? Once you start approaching your product design that way, you'll be set up to get your customers to buy into the microworld that you're building, and start to imagine that it's richer and more detailed than it actually is.
This is good, keep up the good work.
Also, from the creator:
>I'm working on adding more content, more detail, more examples, and really just more everything. Please give me any and all feedback, I've already followed some advice given in the comments on this post. It's really appreciated
How would you get started reverse engineering an old PS1 game? Say you wanted to try and extract the model data or textures and the disc contains only 3 or 4 files, what's the starting point?
I really like the sections on assembly. They're well written and a pretty good introduction or refresher. There's plenty out there on the subject but few that are as approachable.
A few posts up, melvinroest suggested starting a study group for the (Windows-based) course in the original post -- would anyone be interested in doing the same for this Linux-based course?
Ok, awesome! I'm pretty much an exclusively Linux guy, so that's the path I'm planning on going down -- my email's in my profile. I'd love to discuss further!
RPISEC RE/VR courses (a little scattered due to the passage of time):
Secure Software Principles - CSCI 4971, Spring 2010:
http://www.cs.rpi.edu/academics/courses/spring10/csci4971/
Malware Analysis - CSCI 4972/6963, Spring 2013:
http://security.cs.rpi.edu/courses/malware-spring2013/
Advanced Exploitation and Rootkit Development, Spring 2013:
http://security.cs.rpi.edu/~candej2/
Program Obfuscation, Fall 2013:
http://security.cs.rpi.edu/courses/obfuscation-fall2013/syll...
Windows Exploitation, Spring 2014:
http://security.cs.rpi.edu/~gaasem/winexp/IndependentStudy.p...
http://gaasedelen.blogspot.com/2014/02/windows-exploitation-...
Modern Binary Exploitation - CSCI 4968, Spring 2015:
https://github.com/RPISEC/MBE
Malware Analysis - CSCI 4976, Fall 2015:
https://github.com/RPISEC/Malware
Forgot this one:
Hardware Reverse Engineering - CSCI 4974, Spring 2014:
http://security.cs.rpi.edu/courses/hwre-spring2014/
And put the full list on Github:
https://github.com/JeremyBlackthorne/RPISEC-Courses
Wow, great list, thank you for taking the time to compile it!
Thank you for putting this together!
This is great! Do you have other related curated lists that you're willing to share?
Yes! That's like half my job as a teacher. Let me scrounge around, polish what I got, and put it on Github.
Excellent! Thanks. I followed the links on Github, btw. I'm looking into some of the trainings you're offering.
Last August, I took the Advanced Windows Exploitation course from Offensive Security (for the OSEE), and then I followed it up with the awesome Advanced Fuzzing and Crash Analysis course taught by Richard Johnson. Both were incredible courses, but my RE sucks.
I'm looking at the Advanced Browser Exploitation course next, but I'd really like to get better with reversing in order to get more out of these classes.
I also find the lectures on here helpful http://opensecuritytraining.info/
I'll admit I haven't had the chance to look through it at all yet.. in fact, I'm typing this comment just I took a glance at the table of contents. It's a small thing, but it kind of irks me that "Final Notes" following "0x509 ImplementingPlayer" is indexed as 0x510 rather than 0x50a
Anyway, I'm going to save this for later viewing.. very interesting post.
Always thought reversing is fun. Back in high school and college I would spend days or weeks trying to reverse random software packages I had installed on my system. It was pretty surprising how, with a few tools like OllyDBG, 90% of the software I attempted to crack was actually crackable. Most of Autodesk's software was ridiculously easy to crack. (Don't worry, I owned all that software already! I was studying animation at the time.)
I didn't realize this until later, but I got a skill out of reversing that's actually pretty useful as a software engineer. Granted, I work with JavaScript, but reversing actually taught me how to persevere with debugging. I've been able to fix some bugs that other people couldn't in a reasonable timeframe because I treated the problem like I would when reversing a program; it can be more expedient to perform a process of elimination by disabling or inverting parts of the code, observing what happens, keeping detailed notes on observed behavior, and lead yourself down to the key change you need to make. A lot of people debug problems by trying to look at the code and reason about it while setting breakpoints, but there can be a lot of mental overhead in doing this. When the problem is mysterious, I find that a process of elimination through experimentation can be more effective.
This actually happened recently, where we essentially had an infinite feedback look happening that was obfuscated by a lot of framework code. It's possible that the root cause could have been identified by debugging, but a few people tried to do so but couldn't figure it out. This is because setting a breakpoint somewhere in the process chain didn't reveal anything obvious, and you'd be lucky to actually spot the problem in the application code just by looking at it. I treated the problem like reverse engineering, and did exactly what I described above. It took me a few hours, but I did succeed in narrowing down exactly where in the code things were going wrong.
Sure, someone could have stepped through the app and framework code enough times that they might have figured it out. Perhaps there are also more advanced debugging techniques that none of us were aware of that would have helped. But the mindset of a reverser definitely works.
Someone correct me if I’m wrong but it sounds like you just described “fault injection” or “mutation analysis.”
Probably closer to "mutation analysis". (or "creative tampering"?) Someone smarter than I am could probably reverse engineer while doing less of that, but I've found that it can be effective when knowledge of a system is poor. (and there's urgency that warrants it over traditional learning)
Yes, I have used it at my previous job (recently got fired) and it was really helpful. Of course, you need physical access to the application to make this work, e.g. it's much harder to do for some oddly behaving REST Api.
It seems the majority of RE-related stuff these days focuses on malware but the skills are very useful for other things, particularly debugging in general. I didn't specifically learn RE, but since I started long ago with Asm, I guess it came naturally. You can tell the author of this one probably comes from an HLL background because of the "0x" prefixes; those with an Asm background are more likely to use a "h" suffix instead.
For Windows debuggers I recommend Windbg, which is free and comes from Microsoft. Using VS for Asm-level debugging is only mildly better than using gdb for it --- i.e. very awkward and not what they were designed for.
For Windows I would rather recommend OllyDbg spiritual successor - x32dbg/x64dbg[1]. It is open source and easily scriptable and extendable.
[1] https://x64dbg.com/
https://wade.be/w32dasm/
I'll take this as an opportunity to plug some other RE tools:
IDA is considered the holy grail. A base version is free, but its most popular feature, the disassembler, is $$$. Does debugging and static analysis.
NSA's Ghidra is great too: https://ghidra-sre.org/. last time I checked I don't think it did debugging but they were adding it. It's great for static analysis though, it has a great free disassembler (traditionally IDA's domain).
Cheat Engine is pretty amazing too, despite the goofy name: https://www.cheatengine.org/. It's the best tool for pointer scans imo, but also does debugging and is scriptable. Popular for making video game trainers, but it can be used for anything.
I used to dabble in Cheat Engine, Tsearch, and IDA Pro, for reversing games. Recently, a nice guy randomly showed me a demo of a modern tool called radare[1]. I was floored by the IDA-like flow-diagrams that it spit out in the terminal[2], and the short commands for navigating/bookmarking/documenting. Nice to see hobbyists don't need to buy IDA anymore!
[1]:https://www.radare.org/r/
[2]:https://www.radare.org/r/img/r2cg.png
There is also a GUI called Cutter that is powerd by radare2.
https://cutter.re
Dude, look at Ghidra. The reverse engineering world has massively changed in the last year.
Radare is great and I use it as well, but Ghidra brings a whole lot of new capability and its straight up free.
Thanks! An NSA project, named after Godzilla’s archenemy. Also:
> Ghidra's existence was originally revealed to the public via WikiLeaks in March 2017
[1]:https://en.wikipedia.org/wiki/Ghidra
There is a useful IRC channel on freenode called ##re, for those who might care.
This title made me feel nostalgic, and search for +ORC [1] and +Fravia [2].
[1]: https://en.m.wikipedia.org/wiki/Old_Red_Cracker [2]: https://en.m.wikipedia.org/wiki/Fravia
I always feel bad software like SoftIce will never be a thing again. Felt so much power using it.
SoftICE still holds a super special place in my heart, but WinDbg is infinitely more powerful these days than SI ever was, especially if you're using it from another box (or from your host with a Windows guest VM).
Really wish +HCU were still around today.
> If 0x12345678 was loaded into a 64 bit register such as RAX, then RAX refers to 0x12345678, EAX refers to 0x5678, AX refers to 0x78, AH refers to 0x7, AL refers to 0x8.
I made a PR to correct this example which equated single hexadecimal digits with whole bytes.
Other than that what I read of the course is rather nice, targeted at a mostly beginner audience with some correct insight.
There also some weird insight:
> You can think of computers as trains, they don't stop and only go in a very specific and direct path as designated by the tracks. If there's a child on the tracks it's up to the people controlling the track to divert the train. This is why Windows gives you the Blue Screen of Death (BSOD) when there is a kernel error. If the OS doesn't stop that error, catastrophic damage could occur.
This isn't really wrong, it's just a non-obvious metaphor formulated strangely in my opinion. A BSOD would be more akin to the fully automated train just blowing up in sight of the children or something? It's weird.
I think the author is referring to catastrophic damage in the sense of file/data corruption and loss. In that light a BSOD makes sense as a way to "stop the error" since it halts execution rather than silently continue. I agree that it's strange, though.
I started with these series of tutorials:
https://tuts4you.com/e107_plugins/download/download.php?list...
Purely hands-on. Old school. Use virtual box.
Hmm, I've been itching to dive into some security thing again (I'm getting into the habit to dedicate 1 to 2 months per year for it, this would be year 4). This person's effort seems so awesome and genuine that I'm up for helping out.
Though, I don't want to study it completely by myself, it gets a bit lonely and there's no accountability. Does anyone want to set up a study group with me?
My background: I know how to reverse Linux binaries, albeit I'm a bit rusty. I followed a course called binary and malware analysis at the Vrije Universiteit Amsterdam, and two other related security courses (shout out to Herbert Bos and his team for giving such an awesome course).
My email is in my profile.
Would love to have someone to study with - I'm currently going through Reverse Engineering 4 Beginners by Dennis Yurichev but will happily try this one out too.
Will email you later.
I do not have a security background but I would love to study with you!
I received a couple of emails. This is gonna be fun! :D
Same here, I’ve been doing RE challenges (most recently, microcorruption.com).
I enjoy the little nuggets of wisdom like:
>Learning your first calling convention is like learning your first programming language. It seems complex and daunting at first, but it's really quite simple.
I will definitely read this as I'm also doing the Reverse Engineering for beginners, as well as getting sidetracked to learn Assembly with Programming From the Ground Up first.
BCI teaches some hands on courses in Boston if you like to travel. https://www.eventbrite.com/o/boston-cybernetics-institute-18...
Will Wright defined the "Simulator Effect" as how game players imagine a simulation is vastly more detailed, deep, rich, and complex than it actually is: a magical misunderstanding that you shouldn’t talk them out of. He designs games to run on two computers at once: the electronic one on the player’s desk, running his shallow tame simulation, and the biological one in the player’s head, running their deep wild imagination.
"Reverse Over-Engineering" is a desirable outcome of the Simulator Effect: what game players (and game developers trying to clone the game) do when they use their imagination to extrapolate how a game works, and totally overestimate how much work and modeling the simulator is actually doing, because they filled in the gaps with their imagination and preconceptions and assumptions, instead of realizing how many simplifications and shortcuts and illusions it actually used.
https://www.masterclass.com/classes/will-wright-teaches-game...
>There's a name for what Wright calls "the simulator effect" in the video: apophenia. There's a good GDC video on YouTube where Tynan Sylvester (the creator of RimWorld) talks about using this effect in game design.
https://en.wikipedia.org/wiki/Apophenia
>Apophenia (/æpoʊˈfiːniə/) is the tendency to mistakenly perceive connections and meaning between unrelated things. The term (German: Apophänie) was coined by psychiatrist Klaus Conrad in his 1958 publication on the beginning stages of schizophrenia. He defined it as "unmotivated seeing of connections [accompanied by] a specific feeling of abnormal meaningfulness". He described the early stages of delusional thought as self-referential, over-interpretations of actual sensory perceptions, as opposed to hallucinations.
RimWorld: Contrarian, Ridiculous, and Impossible Game Design Methods
https://www.youtube.com/watch?v=VdqhHKjepiE
5 game design tips from Sims creator Will Wright
https://www.youtube.com/watch?v=scS3f_YSYO0
>Tip 5: On world building. As you know by now, Will's approach to creating games is all about building a coherent and compelling player experience. His games are comprised of layered systems that engage players creatively, and lead to personalized, some times unexpected outcomes. In these types of games, players will often assume that the underlying system is smarter than it actually is. This happens because there's a strong mental model in place, guiding the game design, and enhancing the player's ability to imagine a coherent context that explains all the myriad details and dynamics happening within that game experience.
>Now let's apply this to your project: What mental model are you building, and what story are you causing to unfold between your player's ears? And how does the feature set in your game or product support that story? Once you start approaching your product design that way, you'll be set up to get your customers to buy into the microworld that you're building, and start to imagine that it's richer and more detailed than it actually is.
This is good, keep up the good work. Also, from the creator:
>I'm working on adding more content, more detail, more examples, and really just more everything. Please give me any and all feedback, I've already followed some advice given in the comments on this post. It's really appreciated
How would you get started reverse engineering an old PS1 game? Say you wanted to try and extract the model data or textures and the disc contains only 3 or 4 files, what's the starting point?
I really like the sections on assembly. They're well written and a pretty good introduction or refresher. There's plenty out there on the subject but few that are as approachable.
Would love to see this pop up again once it's more fleshed out.
Seems like this should be titled “reverse engineering software products course”.
Mechanical RE is equally important for many applications as well.
Does anyone remember SoftIce here? 🧓
I do! I remember my first project with it was cracking WinRAR to stop it from showing the nag screen.
Anyone have links to similar resources, but for Linux instead of Windows?
Self plug: https://www.reversinghero.com Learn reverse engineering by solving 15 challenges.
This is excellent. Does anyone know a similar resource for Mac or Linux?
I haven't compared them, but another reverse engineering resource:
https://beginners.re/
Wow, I was just about to go looking for something like this. Much more comprehensive than the OP. Really excellent, thanks!
This looks amazing, thank you!
A few posts up, melvinroest suggested starting a study group for the (Windows-based) course in the original post -- would anyone be interested in doing the same for this Linux-based course?
I am all in to form a peer group to study RE (Windows or Linux)
I am more interested in memory patching and binary modification of software (including games) than malware analysis
Yes I'm interested into Linux systems too, I work for embedded sw dev and sometimes I struggle with core dumps in gdb, this would help me :)
My email's in my profile -- shoot me a line :)
Ok, awesome! I'm pretty much an exclusively Linux guy, so that's the path I'm planning on going down -- my email's in my profile. I'd love to discuss further!
Anyone has suggestions for iOS/Android reverse engineering?