Looking at [1], it has several red flags.
The replay protection is overly complicated and doesn't kick in after the message is decrypted. This makes it possible to DoS the server with forged messages.
Key derivation uses a custom scheme. Typically there's no reason NOT to piggy-back on existing schemes and there's plenty to choose from - from TLS to IKE.
Also, as already mentioned, there's again NO reason not to use TLS in Anonymous DH mode with an app-level authentication of the session handshake.
Designing your own crypto protocols is a very interesting challenge, but for practical purposes you just have to recycle existing designs. There's really no other way about it. A custom crypto doesn't make any difference for those who doesn't know/care about it, but it certainly will not make you any friends between those who does. Unless, of course, you can explain and prove why your design is better than those that exist already, and these guys don't do this.
The reason for designing something new was this: in order for really secure messaging to catch on among the masses today — it needs not only be secure. When you want to compete with the likes of WA, you also need be fast and reliable on weak mobile connections.
TLS/HTTPS is slow and takes a lot of time to restore connection on fragile networks.
MTProto was born as a result of an experiment: whether it would be possible to create something that is both secure and fast.
Disclaimer: Unfortunately, I'm not Nikolay. But I do work for Telegram.
>MTProto was born as a result of an experiment: whether it would be possible to create something that is both secure and fast.
I'd like to see some papers/surveys/case studies on the matter. Yes, it's an "experiment", doesn't mean it's a good idea to roll out your own implementation before performing (and publishing) extensive tests.
Not saying you're doing something wrong, just that security-oriented people (usually who'd be interested in your product) may want to know more than just "yes guys it's secure, trust us".
This is all great, but it reads like a marketing speak. There was this imperfect world full of idiots, slow, ugly and dim lit and then we came along and lifted the gloom.
Sorry, you have to do better than "TLS/HTTPS is slow". Anything over TCP is slow on congested and lossy networks. That's given, but that's hardly a reason to reinvent the whole crypto stack.
How is your protocol superior / faster / better than DTLS with session resuming? Is it faster than IKE in Quick Mode or IKE v2, which are both datagram protocols designed to work over lossy networks? What are the inherent problems with adopting DTLS or IKE to your purposes? This is dead easy to do without touching original crypto design and it instantly removes all questions pertaining to the quality of your crypto design. So why not?
I'm not a professional cryptographer, but I know applied crypto well and I've seen my share of custom crypto designs. Virtually ALL of them are is a result of thinking that it's better and easier to invent something new than to diligently learn what exists and understand how it works. It's fine for some areas, but it is decidedly not a way to go in the cryptography domain.
Even WA doesn't roll their own key deriviation scheme, they use PBKDF2+SHA1.
make sure to check out their structure definition language:
http://core.telegram.org/mtproto/TL-dependent
look like a great parody to me? can't tell if they are serious with these 'dependent types'
Is anything wrong with it?
it's ridiculous, pseudo-mathematical, and a severe case of NIH syndrome :-) Peano numbers for data definition language? are you serious?
You can drop the first two. And the third can have different motivations: I think here we just have a mathematician with sufficient time on their hands who enjoyed coming up with this. I think the primary valid complaint is: overly complex for its task.
it's not just overly complex, the use of 'dependent types' for this indicates that the author doesn't understand how they are really useful. neither he seems to know what serialization is, he apparently wants to serialize linked lists.
From their page describing TL language.
> TL Language (Type Language or Time Limit)
So it is a partially applied acronym with a branch in it!
There is something seriously messed up in this project.