Knowing the input text was two words separated by a space, I was able to use hashcat and the unix wordlist (/usr/share/dict/words) to find the solution almost immediately. It's a shame that Alex didn't find it this way on his first attempt as the two words are fairly common.
I worked on a puzzle like this roughly 2 years ago from Anthropic. I did the first half, the easier part of the CTF, and my friend did the second half, the more technical ML stuff. We both got interviews at Anthropic, which was cool -
I wasn't anywhere close to nailing an interview at Anthropic but it gave me a lot of confidence to end up going all in on tech, which paid off greatly.
My friend's short write up:
https://x.com/samlakig/status/1797464904703910084
I was one of the solvers. It took me about a week to figure out. This is what I wrote out in my submission with the answer:
> After looking at the final two layers I was somewhat quick to intuit that this was some sort of password check, but wasn’t entirely sure where to go from there. I tried to reverse it, but it was proving to be difficult, and the model was far too deep. I started evaluating the structure and saw the 64 repeated sections of 84 layers that each process 4 characters at a time. Eventually I saw the addition and XOR operations, and the constants that were loaded in every cycle, and the shift amounts that differed between these otherwise identical sections.
> I thought it was an elaborate CTF cryptography challenge, where the algorithm was purposely weak and I had to figure out how to exploit it. But I repeatedly was getting very stuck in my reverse-engineering efforts. After reconsidering the structure and the format of the ‘header' I decided to take another look at existing algorithms...
Basically it took a lot of trial and error, and a lot of clever ways to look at and find patterns in the layers. Now that Jane Street has posted this dissection and 'ended' this contest I might post my notebooks and do a fuller post on it.
The trickiest part, to me, is that for about 5 of the days was spent trying to reverse-engineer the algorithm... but they did in fact use a irreversible hash function, so all that time was in vain. Basically my condensed 'solution' was to explore it enough to be able to explain it to ChatGPT, then confirm that it was the algorithm that ChatGPT suggested (hashing known works and seeing if the output matched) and then running brute force on the hash function, which was ~1000x faster to compute than the model.
TFA details a solution, it's pretty interesting. Basically the problem was to reverse engineer an absurdly obfuscated and slightly defect MD5 algorithm.
> Alex had actually tried to brute force the hash earlier, but had downloaded a list of the top 10,000 most popular words to do it, which turned out not to be big enough to find it. Once he had a big enough word list, he got the answer.
Ah dang. When I did this I also thought the length bug was intentional but I didn't figure it out before I started my new job, so I dropped the puzzle.
Another classic Jane Street puzzle. Boy this was a good one. Sometimes I look back at my childhood and how quick I was to solve some difficult integrals and so on and now I’d struggle at that. This is far beyond that but the leaps of intuition required here sort of have that property that they need you to stay in the game. Step away a few years and try to come back and there’s just a wall.
I don’t think I’m close to making progress on stuff like this. Interesting to note. Glad they wrote out this behind the scenes thing.
I was curious to see if I could crack the MD5 hash so I managed to write the following python code to extract the expected hash from the model:
https://gist.github.com/alexspurling/598366d5a5cf5565043b8cd...
Knowing the input text was two words separated by a space, I was able to use hashcat and the unix wordlist (/usr/share/dict/words) to find the solution almost immediately. It's a shame that Alex didn't find it this way on his first attempt as the two words are fairly common.
I worked on a puzzle like this roughly 2 years ago from Anthropic. I did the first half, the easier part of the CTF, and my friend did the second half, the more technical ML stuff. We both got interviews at Anthropic, which was cool - I wasn't anywhere close to nailing an interview at Anthropic but it gave me a lot of confidence to end up going all in on tech, which paid off greatly. My friend's short write up: https://x.com/samlakig/status/1797464904703910084
Model interpretability is going to be the final frontier of software. You used to need to debug the code. Now you'll need to debug the AI.
With the number of operations and the error rate in GPUs this is going to be interesting in SOTA models.
Don't forget quantization..
This is pretty cool, I wasn’t aware of these types of challenges. How does one even approach this?
Feels to me like it’s similar to dumping a binary with an image, the format being entirely custom.
And/or trying to decode a language or cipher, trying to recognize patterns.
I was one of the solvers. It took me about a week to figure out. This is what I wrote out in my submission with the answer:
> After looking at the final two layers I was somewhat quick to intuit that this was some sort of password check, but wasn’t entirely sure where to go from there. I tried to reverse it, but it was proving to be difficult, and the model was far too deep. I started evaluating the structure and saw the 64 repeated sections of 84 layers that each process 4 characters at a time. Eventually I saw the addition and XOR operations, and the constants that were loaded in every cycle, and the shift amounts that differed between these otherwise identical sections.
> I thought it was an elaborate CTF cryptography challenge, where the algorithm was purposely weak and I had to figure out how to exploit it. But I repeatedly was getting very stuck in my reverse-engineering efforts. After reconsidering the structure and the format of the ‘header' I decided to take another look at existing algorithms...
Basically it took a lot of trial and error, and a lot of clever ways to look at and find patterns in the layers. Now that Jane Street has posted this dissection and 'ended' this contest I might post my notebooks and do a fuller post on it.
The trickiest part, to me, is that for about 5 of the days was spent trying to reverse-engineer the algorithm... but they did in fact use a irreversible hash function, so all that time was in vain. Basically my condensed 'solution' was to explore it enough to be able to explain it to ChatGPT, then confirm that it was the algorithm that ChatGPT suggested (hashing known works and seeing if the output matched) and then running brute force on the hash function, which was ~1000x faster to compute than the model.
Did you get an interview with Jane Street?
Yeah, they put me on the quant researcher track. Made it to the final round onsite, but they did not extend an offer T_T
Study math/statistics/ML at a graduate level, to start.
TFA details a solution, it's pretty interesting. Basically the problem was to reverse engineer an absurdly obfuscated and slightly defect MD5 algorithm.
I'm really curious what were the magic words.
> Alex had actually tried to brute force the hash earlier, but had downloaded a list of the top 10,000 most popular words to do it, which turned out not to be big enough to find it. Once he had a big enough word list, he got the answer.
They don't reveal the answer.
If I had to guess, “hot dog” would be the first thing I’d try. “Vegetable dog” was given as 0, and it may be alluding to a Silicon Valley episode.
If you want, you can enter your guess in the huggingface page for the puzzle:
https://huggingface.co/spaces/jane-street/puzzle
It's not "hot dog". I wrote in another comment how I found the solution but to give you a clue it is AI related.
not hot dog
Ah dang. When I did this I also thought the length bug was intentional but I didn't figure it out before I started my new job, so I dropped the puzzle.
Another classic Jane Street puzzle. Boy this was a good one. Sometimes I look back at my childhood and how quick I was to solve some difficult integrals and so on and now I’d struggle at that. This is far beyond that but the leaps of intuition required here sort of have that property that they need you to stay in the game. Step away a few years and try to come back and there’s just a wall.
I don’t think I’m close to making progress on stuff like this. Interesting to note. Glad they wrote out this behind the scenes thing.
Give me unlimited API access maybe I can distill it
[stub for offtopicness]
Seems like a thinly-veiled recruiting ad...
Where is the veil...?
I was being polite... :-D