Show HN: HackerNews-new-jobs – insights into fresh and recurring job ads
github.comThe website is meant for people that look for jobs on HackerNews "Who's Hiring" threads and want to focus more on fresh ads and companies, or to quickly look up ad history of any company.
Github repository: https://github.com/nemanjam/hn-new-jobs
Demo website: https://hackernews-new-jobs.arm1.nemanjamitic.com
I used Algolia API as a data source, along with scheduled task that parses new threads few times at the beginning of each month. The extracted data is then stored in SQLite database for fast querying, and the results are cached with Keyv for faster page responses. I will see in the future how much traffic the website receives and if this stack is performant enough. For the website I used Next.js app with default ShadcnUI components and charts. I just wanted a quick functional prototype to test how much public interest is there for an app with functionality like this.
If you are interested in more implementation details you can find them in the Readme file on Github.
The project is free and open source. Feel free to use, self-host, fork and modify, and contribute. I would love to hear your impressions and suggestions and look forward to discussing features and technical details.
Nice to see the trend confirming my suspicion, the market has been complicated for the last year and it's not really improving (whether you are hiring or looking for a job)
I found the same - market is actually getting worse (if HN is representative of the market). This month there were more job seekers on HN than jobs for the first time since 2014.
https://bilbof.com/posts/tech-hiring-is-bad-right-now
This site I think counts only top-level comments, and the streams have been crossed since ~April 2023: https://www.hnhiringtrends.com/ . The overall trend is the same though - the market seems pretty historically bad.
There's also Indeed postings data, which unfortunately only goes back to 2020 but is similarly bleak: https://fred.stlouisfed.org/series/IHLIDXUSTPSOFTDEVE
This is a rather depressing graph, what happened in 2022-2024?
Collapse, unfortunately.
end of ZIRP
This and over-hiring.
And to put a more depressing angle to the graph, HN and Who's hiring were not that popular pre-2017, and now 2024 numbers are lower than the pre-2017 data
> HN [...] were not that popular pre-2017
Is that true? What happened to change that in 2017?
I've been around longer than that and I admit I haven't seen that much of a difference in engagement around here. But over such a long period it could easily have gone unnoticed by me!
The only thing I've noticed is the increase in activity during the recent Reddit shenanigans (which resulted in a drop of quality of the conversation on HN, but it seems back to normal now).
The tax change that causes companies to have to weirdly treat developer salaries as some sort of asset such that they can only write off 20% of it per year. Outcome of this is killer for startups and causing a huge issue everywhere.
Bottom line: If a company makes $1MM in revenue and pays $1MM in salary, they owe taxes on $800k profit. Yes, this is actually the law now.
do you have a link to the tax change?
surely tax changes come in the form of congressional+presidential bills and amendments
Tax Cuts and Jobs Act of 2017 moved domestic R&E expenditures (including salaries) under IRS Sec 174 from a same-year credit to a five-year amortized expense (similar to capital expenditures). It also amended 174(c)(3) to ensure that software dev is unequivocally an R&E expense[^1].
1: https://irc.bloombergtax.com/public/uscode/doc/irc/section_1...
What the hell? This seems almost purposefully written to destroy the one industry where ordinary people could get good salaries. It doesn't even make any sense whatsoever, just a straight "fuck you" regulation?
It almost surely was either a slap at large tech companies or was meant to generate short term revenue to cover some other cut.
Same thing happened when they restructured tax code to interpret withholdings differently. Everyone saw more on their paycheck temporarily (and they gave speeches about it!) but owed more later if they didn't change their withholdings.
Big tech, ie, large companies who are expected to exist for 5 years with about the same workforce, actually will go back to having the about same benefit in 2027. (They claim 20% of 2022 in 2022, then 20% of 2022 and 20% of 2023 in 2023…) It also covers only development expenses, so sales and legal (a large part of big tech!) are still taken as wage expenses.
So it’s likely the second — and was likely used in part to pay for those very same withholding changes! Or the then-new tax-exemption for lobbying expenses.
Note: these changes were signed into law in 2017 but came into effect in 2022
https://www.resourcefulfinancepro.com/news/irs-section-174-c...
> The tax change that causes companies to have to weirdly treat developer salaries as some sort of asset
The salaries are not an asset, they are the cost of creating the asset. They are capitalizable similar to (but on a different schedule than) costs of acquiring software that is not developed in house.
Has anyone heard opinions from the incoming administration on getting this changed?
I guess the one area this tax law particularly affects are bootstrapped revenue-generating (non-VC funded) startups with high dev costs? i.e., actual running businesses not playing with monopoly money.... which maybe Elon doesn't care about....
The change was introduced in Trump's tax cuts, so I doubt he will reverse it.
Did the change take effect during his administration or after the tax cuts expired or rather after the income brackets also increased? That change was intentional so that the following administration if it wasn't the GOP would feel the heat of and get the blame fortaxes going up.
It took place during the administration.
Another poster had a good summary: https://news.ycombinator.com/item?id=42379288
I think this is the more important comment in that thread: https://news.ycombinator.com/item?id=42380764
"Note: these changes were signed into law in 2017 but came into effect in 2022"
Like I thought, changes were meant to be a time bomb if the GOP wasn't in control.
Thank you, I missed that and it’s important context. I’ve upvoted your comment for more visibility.
Do you have a reference where I can read more about this, specifically the timelines?
Another poster had a nice summary: https://news.ycombinator.com/item?id=42379288
The "Section 174" tax change always gets brought up, but wouldn't that be kind of a "tail wagging the dog" explanation? Surely, if it's worth it for a business to hire talent, then it's still worth it regardless of some esoteric tax rule. Are there actually companies sitting there in their Hiring meetings saying "Gee, we really need to expand our business and hire some engineers--if it wasn't for this tax law, we'd be hiring!"
If it becomes meaningfully more expensive to hire talent, it needs to be meaningfully more worth it to hire.
It's more like if before the law was passed they'd hire 5 new devs, now they can only afford 3 and have to make do until more revenue comes in
Cashflow problems regularly kill business. Now you need a much higher cash buffer than before (in parent example where is that 800k coming from?). Combine with much tighter VC market and there are definitely many startups closing up shop because of this.
How do non-developer salaries affect those taxes?
(In 2024, if a company paid salaries for a software developer, a novel writer, and a cook, does each of those 3 positions affect taxes in the same way?)
No, just software devs. Thanks so called "Tax Cuts and Jobs" act.
They're unaffected - the law specifically applies to software development:
(c.1 and c.2 are the opposite - carveouts for land acquisition and fossil fuel and mining exploration)
Interest rates
Interest rates were raised from nearly zero, where they were for a number of years. When the interest rates are that low, high-risk investments like VC are more attractive then alternative (having a low chance of high returns is better than guaranteed no returns at all).
Location / remote / hybrid would be interesting to visualize.
True, I thought about it, technologies too, and other filters. Currently website has clear focus new/repeated job ads. I will see how much public interest is there for the website and if it is worth adding more features.
This looks lovely! Good work.
On my wishlist are some fuzzier categories:
Thank you, I highly appreciate it. As is website has clear and strong focus, it's "fresh" vs "repeated" job ads. Naturally, you would always prefer to give a shot with company that just posted a fresh ad, then to apply for a role that is repeated in 10 months of previous 12 but company still hasn't found a candidate that satisfies criteria.
For the additional filters, by technologies and role types it would be of great help if I could find some high level indexing and fuzzy search tools/libraries. I would probably need to migrate from SQLite to Postgres and when I am already there probably use ORM too.
Certainly, I would need to do serious research, if there is enough public interest in the current website and I manage to find some contributors with data science and information retrieval experience maybe we can add many interesting filters like tech, roles, location, visas, remote, etc.
This is great! I had long ago started a similar project to parse HN job ads month after month looking for the ones that just stay the same each month, which might suggest the company is probably "fishing" instead of "hiring". But it never really got off the ground. This platform seems more than capable of doing that kind of analysis and more. Great work.
Thank you very much, very much appreciated. I was curious to see how much people would be interested in such website.
It would appear you have a non-trivial "company" parse problem, based on eyeballing the "First Time Companies" section. I do appreciate the problem of trying to do semantic analysis on HN comment text, but several of them follow the "X@Y" nomenclature and a few others are "X-Y" so it doesn't seem to be insurmountable to try and fix
This is just a first iteration of a quick prototype to gauge public interest in the idea. It just looks for the "|" separator in the comment title, which is something that majority of posters enforce.
https://github.com/nemanjam/hn-new-jobs/blob/main/constants/...
If people show constant interest I can iterate it further, enhance it, ad features, etc.
Well, if that's your attitude then I wish you all the luck with your project
I was also going to say $(git commit -a) is evil based on <https://github.com/nemanjam/hn-new-jobs/blob/main/data/datab...> but it seems that you just want an always changing binary blob to make your git repo grow without bound :-( https://github.com/nemanjam/hn-new-jobs/blob/main/.gitignore...
Possible bug report, I noticed a company that posts regularly but I didn’t see it listed: https://news.ycombinator.com/threads?id=spark_CM
Here it is:
https://hackernews-new-jobs.arm1.nemanjamitic.com/search?com...
Something still looks off, the earliest ad is from 2024-07, but the company posted more recent ads too.
I guess this is because they changed their ad’s headline from:
to:
Other companies are similarly affected, e.g. Medusa:
- listed: https://hackernews-new-jobs.arm1.nemanjamitic.com/search?com...
- earliest ad listed: https://news.ycombinator.com/item?id=34222858
- more recent ad on hn: https://news.ycombinator.com/item?id=42315828
Perhaps it’s the “)|” bit causing problems with some regexps.
Yes, there is a lot of room to improve parsing.
This is nice, but I find this one more detailed: https://hnhiring.com/trends
This is awesome! But it's missing at least one company from 2024-08. Maybe a parsing error? Or does it exclude hiring freelance/contract posts?
Thank you very much, I am glad you like it.
I don't understand what is missing in 2024-08? You can link month by slug, I see nothing unusual here:
https://hackernews-new-jobs.arm1.nemanjamitic.com/2024-08
But as a side note, yes, this is not meant as an exact analytic tool, rather just a best effort website that gives some interesting insights.
To clarify, only the "Who's hiring" thread is parsed, you can see it clearly in this constants file:
https://github.com/nemanjam/hn-new-jobs/blob/main/constants/...
Also, in there you can see how simple the parsing regex is, it just looks for "|" separator in the comment title.
Another thing I noticed, some companies used different letter casing for their name in some comments, and the company name is part of the primary key, so same company is perceived as different, I should probably handle this better.
https://github.com/nemanjam/hn-new-jobs/blob/main/modules/da...
For example you can search for "ConsenSys" on the Search page:
https://hackernews-new-jobs.arm1.nemanjamitic.com/search?com...
I didn't realize freelancer jobs go to a different HN post `Ask HN: Freelancer? Seeking freelancer?` Makes sense now, because the job I noticed missing was a freelance post.
You can try searching directly on Algolia, my website uses it as a data source:
https://hn.algolia.com/
https://hn.algolia.com/api
If it's there my website has it also. But Algolia does pretty good job parsing HackerNews, I am pretty confident 99.9% of comments are included.
I can maybe make a PR later - but would love to see the companies sorted alphabetically so I could just glance for specific ones!
Really shows the hiring slowdown over the last 2 years.
Really terrific contribution.
Thank you, much appreciated.
Me: hmm what’s the big dip in the middle?
Spouse: uh, COVID?
Me: ohhhh
"HN Who's hiring" thread is just a random sample that pretty accurately describes trends in the entire industry.
I don't know who's to thank for it, but someone set up a constantly updated ClickHouse database of all data from the Hacker News API at https://play.clickhouse.com/play?user=play#U0VMRUNUIG1heCh0a...
It gets you 80% of the way there on any HN data project.
This very is useful, thank you. Algolia API is also pretty generous, it made my work a lot easier. It returns even 1k comments in a single response. I started with low level Axios parser but I soon discovered that HackerNews webserver has very strict rate limiting. Good to know that there are already available data sources so you can focus on querying logic and data presentation.
This is very useful, thank you!
Nice work for Central Defender /s.
Thank you very much. He is Chelsea legend that messes up my SEO.