Syncing HAK contents
ThriKreen
Member Posts: 4
Background if you don't know me, was on DLA as one of the modelers, where we ended up making WyvernCrown of Cormyr. Made the horse system, as well as a bunch of other features. Ended up getting hired at Bioware to handle content for the Premium Modules as well as the last patches.
Had an idea for keeping the client and server in sync for HAK contents that could have kept it going, but sadly, any further development on NWN was shut down and those of us on the Live Team got moved onto another projects.
For the connection, the idea was to have the NWN client look for and run an batch file, passing the IP of the remote server to it. The purpose is to have it run something like rsync in said batch file that would connect to the server's IP, since it's also running an rsyncd server, and sync any HAK files prior to actually launching the client connection. This saves having to hunt for the right version HAKs to install, and it would always match whatever the server is using.
However, HAK packs sit in your /nwn/hak/ folder and is a container format that packs a number of other files into one, to ease sharing as well as use up less files on the HDD from large allocation units. The big problem is that even if one file in the HAK changes, it often means you have to download the whole thing again, and we know HAKs can get quite big - Especially with the push for higher resolution textures and such.
One solution was to replicate the method we used for development on WyvernCrown of Cormyr (and also testing the 1.69 patch contents) was to use rsync to sync the individual files into a folder, then if there were any updates, use an ERF tool to pack the folder's contents into an updated HAK of the same name as the folder. Drastically reduced the download time as you're only updating a handful of changes files instead of the whole thing again, and just need some time for the packing step.
The other idea on top of this, was to basically allow NWN to read a folder of unpacked content as if it was a HAK file, saving the step of repacking. This could also come in handy for mod authors as they can speed up their development time as they'd only have to make the actual HAK pack at the end prior to distribution.
Obviously there's security issues in regards to running the external batch file, as it would be much better if the download and sync features were built into the client and server. It's been awhile since I've had to think about this obviously, but I'll share what we were thinking and hopefully the people at Beamdog can maybe make something like this a reality!
Had an idea for keeping the client and server in sync for HAK contents that could have kept it going, but sadly, any further development on NWN was shut down and those of us on the Live Team got moved onto another projects.
For the connection, the idea was to have the NWN client look for and run an batch file, passing the IP of the remote server to it. The purpose is to have it run something like rsync in said batch file that would connect to the server's IP, since it's also running an rsyncd server, and sync any HAK files prior to actually launching the client connection. This saves having to hunt for the right version HAKs to install, and it would always match whatever the server is using.
However, HAK packs sit in your /nwn/hak/ folder and is a container format that packs a number of other files into one, to ease sharing as well as use up less files on the HDD from large allocation units. The big problem is that even if one file in the HAK changes, it often means you have to download the whole thing again, and we know HAKs can get quite big - Especially with the push for higher resolution textures and such.
One solution was to replicate the method we used for development on WyvernCrown of Cormyr (and also testing the 1.69 patch contents) was to use rsync to sync the individual files into a folder, then if there were any updates, use an ERF tool to pack the folder's contents into an updated HAK of the same name as the folder. Drastically reduced the download time as you're only updating a handful of changes files instead of the whole thing again, and just need some time for the packing step.
The other idea on top of this, was to basically allow NWN to read a folder of unpacked content as if it was a HAK file, saving the step of repacking. This could also come in handy for mod authors as they can speed up their development time as they'd only have to make the actual HAK pack at the end prior to distribution.
Obviously there's security issues in regards to running the external batch file, as it would be much better if the download and sync features were built into the client and server. It's been awhile since I've had to think about this obviously, but I'll share what we were thinking and hopefully the people at Beamdog can maybe make something like this a reality!
13
Comments
Does anyone even read the server descriptions anymore ?
Anyway, good to see you. Are you planing on doing anything with the new EE?
Also, a hak sync would be good. I recall we were trying out the bat update system back then but there were some problems with it.
With this being released on steam, having access to the steam workshop will make getting haks much more simple I think, at least, but something that auto detects what you need for the server you are joining and downloading it all for you would be sweet.