I'm one of the people behind the Metal Gear Solid 3: Subsistence server and I've spent a very significant amount of time trying to recreate DNAS. If someone wants to make a stab at this though I figured I'd give a bit more light on the core problem
As TheDominator said "One of the biggest problems you probably will run in to is encrypted traffic." and as Viscosity said:
The main issue is that sony own the key to their SSL connection. Using that would be illegal unless we reverse engineered it which is pretty much cracking it.
For what its worth, using the key wouldn't actually be illegal, the illegal thing would (likely) be getting access to that key since SSL/TLS is built on public key cryptography, the private key never leaves the server. It cannot be reverse engineered and even with modern computing it is still not feasible to generate the private key from the public key.
I figured I'd add an explanation of the problem for anyone who wants to try and pick up where I left off:
SSL/TLS (the protocol that provides encryption for TCP connections, DNAS is over TCP) is based on public key cryptography. There is a certificate that is provided when you connect to the server and try to establish a secure connection, this certificate includes stuff like who you're connecting to (the domain name, company name, etc) and it includes the key you can use to encrypt a message to the server(there is a separate key for decryption only the server knows). This certificate is then 'signed' using the private key. This signature can be verified using the public key but can only be generated using the private key. If you change any details in the certificate(like the domain name) then verification will fail unless you update the signature using the private key...so basically this prevents tampering of the certificate.
So one question someone might then ask is: What is stopping me from creating a certificate with my own keys and just saying it's from Sony, this is prevented by a sort of 'chain' of signatures. Not only does the private key sign it but you need to get this certificate signed by a 'Certificate Authority' (CA for short). When you request a CA sign your certificate they will verify certain aspects of the certificate, at the cheapest level they just verify you control the domain name the certificate is for and more expensive ones will verify the company and personal information also. Once they are satisfied they sign your certificate.
Part of the verification of a certificate the PS2 receives when connecting to DNAS is to check for this CA's signature and that the CA is one that the PS2 trusts. You can create your own CA and sign a certificate (called self-signing) but that CA won't be trusted by the PS2.
All this to say when the PS2 attempts to connect to the DNAS server If the signature doesn't match the content, or if the CA's signature is not from a trusted source, or a number of other reasons the certificate is rejected and the connection won't go forward.
Preventing someone from masquerading as another server is one of the primary purposes of using SSL/TLS in the first place, so it is not surprising that this is a major roadblock. The three classic methods of dealing with this are to either add your own CA to the trusted list of CAs, or to change the domain it tries to contact to one you can get a certificate for, or disable certificate verification.
All of those are applicable on the PS2, however if you have access to do any of these on your console, you'd also have access to simply disable DNAS altogether by patching it out. Why waste time building a DNAS clone when it can't be used unless you already skip DNAS?
If someone wants to take up this challenge there are a few possibilities:
I don't remember if the PS2 does revocation checking but while writing I had the thought of using a stolen CA key. Occasionally some random CA will have their key stolen. Usually, this key won't be publicly released and I can't think of any time they have been if it ever happens that one is publicly released that belongs to a CA the PS2 trusts, and the PS2 doesn't do revocation checking then you could use that key to generate an acceptable certificate for DNAS. I'll admit this is a pretty unlikely case since releasing the key is a stupid move after someone has stolen it.
Occasionally CAs will have a bug that results in mistakenly signed certificates, abuse one of those to get a valid certificate
Recover the DNAS private key. Sounds crazy but this might be the most likely case. It is possible to discover the private key information from the public key it is just computationally difficult and likely not feasible with current computing power.
Noticing a trend? These all rely on somehow getting a valid certificate which shouldn't happen.
The other option is the ability to patch the game, this can be accomplished with something like a Code Breaker, Action Replay, Gameshark or a homebrew program to patch out DNAS. Without using something 'extra,' the only option is to find an exploit that can be leveraged to get code execution. This exploit would need to be in code that precedes DNAS, so likely candidates are in DNS response handling, SSL/TLS verification or in the TCP and UDP processing code. If the game has downloadable content that is another possibility. Other locations like save management are viable but would be roughly as accessible as using a cheat disk or homebrew.
If someone here does decide to try and pick up this challenge please feel free to contact me with any questions on here or on email if I don't respond. I've spent a lot of time working on DNAS and it has annoyed me that I haven't managed to solve this problem over the years.
Email: my username at savemgo.com