The Make haks and tlk's easier to download for persistent worlds card discussion
JuliusBorisov
Member, Administrator, Moderator, Developer Posts: 22,754
Greatings, NWN:EE enthusiasts!
Please, provide more details and feedback for
https://trello.com/c/Jk7UmlSa/33-make-haks-and-tlks-easier-to-download-for-persistent-worlds
in this thread.
Please, provide more details and feedback for
https://trello.com/c/Jk7UmlSa/33-make-haks-and-tlks-easier-to-download-for-persistent-worlds
in this thread.
0
This discussion has been closed.
Comments
On one end of the spectrum, we have a server broadcasting a set of links, client checks if it has the haks at the right version, and if not downloads them (compressed).
Big issues here are:
- If PWs repackage the same content in PW-specific haks, that content is duplicated on player system
- If PWs are using different versions of the same hak (i.e. same name), player will have to download every time they switch servers
- If only a small part of the hak changes, player needs to re-download the whole thing.
On the other end, there's a full rework of the resource manager, where each revision of each resource is tracked separately, and is synced when needed. This means no duplication on the player PC, no large downloads, no collisions between versions. It does however mean BD will have to lock up one dev in a basement and we likely won't hear from them until July.I'm curious which point on the spectrum people consider Good Enough.
Sometimes this is countered with "storage is cheap/internet is fast!" arguments but I had to play this game on dial-up for years. Even now my rural-ass internet is not great, and we've also got data-capped people that would at least want to know when there's going to be a multiple-download problem so they can make a decision on where to spend their data. It'd be so awesome for that not to be a thing, anymore, and redownload only when those assets have been altered (eg texture overrides etc).
Apologies for the mini-rant, I just wanted to say I am all for locking up a dev in a basement for a year. Especially niv! *shakes fist*
If those files already exist in hak/override/tlk/music etc, then they are used as is. For any matching files not found the server points to a file server IP and the player downloads the new files to folder called persistent world/servername.
This way CEP/ProjectQ dont have to be redownloaded, unique hak sets/portraits etc wont overwrite each other causing conflicts and if the player wants to avoid bloat across multiple servers they can just add the files to their master hak/portrait/music/tlk folder.
Code wise this solution is really just a glorified ftp server so development zots should be fairly light.
This seems like it would be very difficult to tackle if you only one single copies of each unique file in a client folder. It would be easier to allow duplicates if Beamdog does not have a system of verified reserved 2da lines and asset file names, which would likely break some existing things.
Auto-downloading would work better if it simply does the checks others mentioned, and verifies that the client and server have the same files. After the checks it would send the compressed files one at a time so partial download is possible, and install them in the proper directory. Fixing custom server tlk as more straight forward which can be downloaded to a PW tlk folder would be handy too.
If Beamdog simply supported downloading files if they do not exist or their hash changed, this would be viable IMO. The issues mentioned above could be addressed as follow:
- Duplication: Hard disk space is very cheap now compared to hak size & with autodownload, players could just wipe their hak directory if it filled up and re-download what they are currently using automatically.
- Filename clash: This is a problem with the current nwn server landscape, but it happens far less frequently than you'd think, as almost everyone re-packages content anyway. I don't know that a better download system must solve it ( though it'd be bonus).
- Downloading the whole file. I don't think this is that big of a problem. At least compared to a player not understanding which directory to put a file in or trying to remember the website for the server, etc. By the time they did that, the entire large file would be downloaded in the new system. PW owners can also add more smaller haks if they want to cater to this use case.
Optional add ons:An option to clear all haks out via the UI for players who cannot find hak/tlk directories & run out of disk space.
Creating unique namespaces for each server's hak list could also get around some of the issues with re-downloading haks or having file mismatches, but this seems like a lot of work for not that much return on investment.
It's tempting to fight for the end to re-packaging content by revamping how resources are managed to avoid the need for everyone to make their own hak. This would make it tons easier to try new servers that really do have similar content under the covers. This (seems) like a really far reaching effort though, so given the low cost of hard disk space, I question if it is really worth it.
Auto-download of the HAK and TLK, and perhaps even Portraits of players in the server
If the server you want to try repackaged CEP2 into his own haks you will have to download those 2gb++ whether you like it or not if you want to play there. I don't see how some automatic hak downloader changes that.
If the downloader informs player that the total size of haks required to download is 2gb and gives him choice to start downloading or cancel then I see no problem with that.
Resource bloat cannot be fixed itself without NWN2 style of custom content.
It's not gonna be worse than what we got and it worked fine in NWN2. It would be awesome if they'd come up with some nice ressource-identification-system but i'm not sure i want them to spend too much time on it.
#1 Player connects to server.
#2 Server checks if it has the files needed when trying to create character or join with one in the vault.
#3 Message is shown to the player. Haks required. List of haks, size, download yes or no, and progress bar.
#4 If downloaded this way. Haks will be located in a subfolder inside hak folder. With the [module_name]_hak. So players can identify and erase those haks once they decide wont be needed anymore. Of course those haks should be protected to preserve any kind of piracy (dont know if it can be done somehow).
#5 Serverside configuration for upload bandwidth. Maxclients downloading at the same time, etc.
Related to same haks being repackaged. I have done this, for two reasons. I serve in already packaged files the stuff I use, so players only have to unwrap and connect. The other one is to ensure players have the same content as the module so they see the content it uses at the version that content were packaged.
For this reason I think my #4 step is important. So you will not override your own haks when connecting to a PW. Also when the client is checking haks, should look first in [module_name]_hak subfolders to use that custom hak and if its not found, then check in hak folder.
IMHO this should not only be to haks: tlks, haks and overrides should have one subfolder per PW. THe standard tlks, overrides and etc (meaning the ones that are not on any subfolder) should only be used for running local modules.
The tlk is also only a single one, so no point in having it in subfolders.
Is there perhaps some sort of master list of all these changes?
And how up to date is nwn.wikia.com these days?
Maybe I've been spoiled with modern tools, but the data required to make small changes to the game seem unreasonable. It shouldn't take external downloads to make simple data changes, audio editions, or texture edits.
What would be ideal is if the module format was expanded so that it could be unpacked and added to. Think something like unzipping the mod file, then dropping in an /audio, /texture, and /data folder that can override texture content and can have scripts pointing to the resources in the folder. Then when a player joins any given server with low level custom content/changes, it can be downloaded to a module directory that the server checks each time it's connected to ensure it doesn't need to redownload those assets.
My PW, uses the set as Worm released, so not modified by later persons - my version works with EE and I have no issues - however if a hak consolidator, updater, distribution were to occure, my working tileset might be replaced with some "Updated" version that breaks my world.
I know the materials I have, well almost all, work and are not a problem but I have no idea what other versions of the same material might cause - this scheme is dangerous.
The ability to have a button for players to use and get the required haks - which might require more than one site - would be helpful. An auto install of the required haks and tlk, custom music and portraits would be nice if it is NOT from the server -- One hak we use for our scripts is empty at distributed keeping our scripts private.
Also - downloading from the server would place an undue strain on my internet - I have a high speed commercial website for posting the haks to download - so the downloads are not on the server site for download.
I acknowledge there are several possible scenarios for PWs proving their haks currently and not everyone has the solution I use.
Recap:
1) haks at the server cannot all be distributed - due to script security in custom hak with scripts - distributed version is empty.
2) Server location downloads would greatly increase internet load and data cap
3) Resource differences - versions that are needed for a world cannot necessary be updated or changed to other versions.
CEP 2.4 differs from TAD updated version of the CEP and can and does break some modules made with CEP 2.4 - SO - there it is - one can not just assume the latest is the greatest. CEP is supposed to be fully backward compatible - the only exception is when Bioware (in the past) required changes - No such changes have been made since 1.69 so CEP must be fully backward compatible and the updates since 2.4 are not.
This is an example - the CEP 2.4 vs updates - why an auto update system can not be forced.
If you go over that limit - BOOM - module will not load anymore - all broken - boo hooo. I have been there and done that - not fun. So this suggestion is not going to work - even if Beamdog expans the module container limit so it will load with more content there are still max content limitations imposed by windows - don't know about Mac and Linux. Some of the CEP haks are pushing that MAX container limit already which is why there are so many CEP haks.
My main point, which may not have been clear, is that I think the best direction to take is to expand the .mod format to include some of the responsibilities previously held by the .hak
Some custom content will require more extensive downloads, and there's no way around that. But things like scripts, custom texture (for server-side overrides) and audio source, etc, is all lightweight enough that there's no reason to avoid transferring it over modern internet on the fly. Multiple downloads of the same custom content seems like a worry for module builders, not players. The simple solution for that is good mod management and only overriding rather than editing community based hak material.
NWN2's solution was good too, though I don't think it went far enough in modernizing capabilities.
There are many ways to avoid these file size limitations in the modern age of computers. I wonder how they could make the change and retain backwards compatibility though. Unless they would make some kind of auto-upgrader for old modules, with old haks.
I'm not sure why a program would not be able to handle a file with less than the maximum or why an older file that could only have less would not work with an improved program that can handle more. I think the limits seem to be in the program as opposed to somehow built into the file but I might be wrong.
If the idea is just to lift the limit, then that is completely backwards compatible. (Assuming there isn't some kind of 32-bit pointer/memory register problem with the new limit size
Anyway, the idea that I was unpacking, was the idea of changing HAKS and TLKS to be more efficient, and to adopt a more readable and efficient structure. There are many design patterns that work better on modern PC's, that weren't an option back in the days of NWN1 development. It would be interesting to see if we could squeeze more performance out of NWN1. That being said, the proposed changes would likely be difficult to achieve while maintaining true backwards compatibility. Unless, I mentioned that they would provide some sort of module/hak/tlk/erf upgrader.
I think I see what you're saying now. It seems like a similar issue to what someone posted about new graphics formats, something that is smaller and quickly readable by a program. I assume if you could squeeze performance out of resource files a conversion tool would be the best option on the player side. I think to download files automatically it would probably be better to just stick to compression formats like NWN2.
It would be particularly fantastic if the work Sinfar has done with (presumably) a roided up resource manager plugin for NWNX + some ingenious coding to allow for their "ERFs" system was combined with the hak/tlk problem.
Allow module builders to separate content from the module into "packages", treated like sub-modules / other projects within NWN. Packages are saved and opened/closed separately from the module and each other, and when open module resources can be dragged between them, allowing the module to be separated into multiple smaller pieces such that the entire module doesn't need to be loaded into memory at any one given time (by the Toolset, or by the server/client), thereby allowing these packages to be loaded/unloaded as specified by scripting during run-time to free up the space of, for example, zones of areas not currently being visited and any creatures, sounds, placeables, and unique scripts or other content that won't be used by the module when those areas aren't being used.
Hak/tlk can then subsequently also be loaded by the server as packages also, communicated to the client on connection/load of the module, before character creation. Where a client has the package already (up to date with the server, or the server uses unmodified community haks) the checksum matches a full compatability checksum and they proceed. If the client doesn't have any/some of the haks content at all, it can communicate to the client where to download (if auto-download is a thing) or send a message. If the server also modifies some of the haks (doesn't use all of the resources from a community hak, or has made module-specific modifications to hak-provided resources) then that can be communicated also and checked for as well, so that new players can be directed to a download containing only the required content if they have none of it, and so that changes to hak assets by the module builders can be updated to clients.
The changes to hak resources I'm talking about are not overwriting content, but additional data that modifies provided hak resources. Changes colour, applies a new texture, scales the hak asset, or something similar. I'm not sure if this is feasible or a easy, but it would be fantastic if persistent worlds could modify community resources to the extent possible without providing additional bulky, duplicate assets to download.
The changes to how 2da's and their files are structured that this would necessitate would also make it possible for current haks with duplicate resources to simply include one "universal" of such duplicated content to be referenced by other hak resources as appropriate. For example with creatures with many variations, many of which are based on a single model with tiny aesthetic changes -- sometimes a model from the original game, no less. This would not end redundancy in old haks, but new ones and rebuilds could certainly do so and decrease their size at least to some extent (compensating for the additional bytes taken up by a new, more dynamic and modern resource file format and structure).
It seems to me that the changes I've suggested to haks/resource management between the server/client and Toolset would be many of the same changes required to facilitate integrating the "ERF"(Sinfar)/resource manager into the Toolset/server as projects, hence why I introduced both in this... massive... rambling... wall of text...