Skip to content

The Make haks and tlk's easier to download for persistent worlds card discussion

JuliusBorisovJuliusBorisov Member, Administrator, Moderator, Developer Posts: 22,714
Greatings, NWN:EE enthusiasts!

Please, provide more details and feedback for

https://trello.com/c/Jk7UmlSa/33-make-haks-and-tlks-easier-to-download-for-persistent-worlds

in this thread.
«134

Comments

  • SherincallSherincall Member Posts: 387
    Depending on how it's architectured, this can be a weeklong project or a yearlong one.

    On one end of the spectrum, we have a server broadcasting a set of links, client checks if it has the haks at the right version, and if not downloads them (compressed).
    Big issues here are:
    • If PWs repackage the same content in PW-specific haks, that content is duplicated on player system
    • If PWs are using different versions of the same hak (i.e. same name), player will have to download every time they switch servers
    • If only a small part of the hak changes, player needs to re-download the whole thing.
    On the other end, there's a full rework of the resource manager, where each revision of each resource is tracked separately, and is synced when needed. This means no duplication on the player PC, no large downloads, no collisions between versions. It does however mean BD will have to lock up one dev in a basement and we likely won't hear from them until July.

    I'm curious which point on the spectrum people consider Good Enough.
    Flashburn
  • thirdmousethirdmouse Member Posts: 67
    The duplication of resources across modules/PWs is one of my biggest pet peeves. It's still fairly common for PWs to use repackaged haks, even of the entire CEP or Project Q instead of using a top hak for organizational purposes -- if only because in some cases a decision was made years ago that they've been stuck with ever since. As more people are drawn back to NWN, you can already hear the distant, "this mega-project is not done right, I want to do mega-project different!" about the same content, drawing closer ;)

    Sometimes this is countered with "storage is cheap/internet is fast!" arguments but I had to play this game on dial-up for years. Even now my rural-ass internet is not great, and we've also got data-capped people that would at least want to know when there's going to be a multiple-download problem so they can make a decision on where to spend their data. It'd be so awesome for that not to be a thing, anymore, and redownload only when those assets have been altered (eg texture overrides etc).

    Apologies for the mini-rant, I just wanted to say I am all for locking up a dev in a basement for a year. Especially niv! *shakes fist*
    Zwerkulesflyinghtcher
  • voidofopinionvoidofopinion Member, Moderator Posts: 1,248
    edited December 2017
    It would be nice if players could connect to a server and have the server check for the required modified files against some kind of checksum.

    If those files already exist in hak/override/tlk/music etc, then they are used as is. For any matching files not found the server points to a file server IP and the player downloads the new files to folder called persistent world/servername.

    This way CEP/ProjectQ dont have to be redownloaded, unique hak sets/portraits etc wont overwrite each other causing conflicts and if the player wants to avoid bloat across multiple servers they can just add the files to their master hak/portrait/music/tlk folder.

    Code wise this solution is really just a glorified ftp server so development zots should be fairly light.
    Avenger_teambgAtrophiedericAnthar
  • FreshLemonBunFreshLemonBun Member Posts: 909
    There are reasons a PW would want to have their own unique repackaging. Even if you use common haks like CEP the references in various 2das might not be the same to ensure that they can make it compatible with all the other content including their own custom additions and alterations. Other concerns might be wanting to trim out unused assets or modifying the same assets that are different in the standard packages.

    This seems like it would be very difficult to tackle if you only one single copies of each unique file in a client folder. It would be easier to allow duplicates if Beamdog does not have a system of verified reserved 2da lines and asset file names, which would likely break some existing things.

    Auto-downloading would work better if it simply does the checks others mentioned, and verifies that the client and server have the same files. After the checks it would send the compressed files one at a time so partial download is possible, and install them in the proper directory. Fixing custom server tlk as more straight forward which can be downloaded to a PW tlk folder would be handy too.
    Maloney
  • farhorizonfarhorizon Member Posts: 4
    edited December 2017
    I think that the ability to automatically download haks/tlk would be such a huge win for the community that I'd go for the simplest implementation to ensure that it is actually prioritized. The rough edges can be smoothed out with some best practices for handling file names (mostly, just have each server use their own repackaged files with unique names), which largely already exist today.

    If Beamdog simply supported downloading files if they do not exist or their hash changed, this would be viable IMO. The issues mentioned above could be addressed as follow:
    1. Duplication: Hard disk space is very cheap now compared to hak size & with autodownload, players could just wipe their hak directory if it filled up and re-download what they are currently using automatically.
    2. Filename clash: This is a problem with the current nwn server landscape, but it happens far less frequently than you'd think, as almost everyone re-packages content anyway. I don't know that a better download system must solve it ( though it'd be bonus).
    3. Downloading the whole file. I don't think this is that big of a problem. At least compared to a player not understanding which directory to put a file in or trying to remember the website for the server, etc. By the time they did that, the entire large file would be downloaded in the new system. PW owners can also add more smaller haks if they want to cater to this use case.
    Optional add ons:

    An option to clear all haks out via the UI for players who cannot find hak/tlk directories & run out of disk space.

    Creating unique namespaces for each server's hak list could also get around some of the issues with re-downloading haks or having file mismatches, but this seems like a lot of work for not that much return on investment.

    It's tempting to fight for the end to re-packaging content by revamping how resources are managed to avoid the need for everyone to make their own hak. This would make it tons easier to try new servers that really do have similar content under the covers. This (seems) like a really far reaching effort though, so given the low cost of hard disk space, I question if it is really worth it.
    Anthar
  • RifkinRifkin Member Posts: 141
    This has to be #1... The burden on new players to servers is already too high, having to go through all the trouble of finding, downloading unknown files that could be harmful/malware...

    Auto-download of the HAK and TLK, and perhaps even Portraits of players in the server
    Cablefish
  • MalisharaMalishara Member Posts: 16
    As long you could specify where to download from (separate website instead of the server's own hak files), it might be a good thing.
    ZwerkulesAtrophiederic
  • ShadooowShadooow Member Posts: 402
    I don't understand what is the problem here with the "resource bloat".

    If the server you want to try repackaged CEP2 into his own haks you will have to download those 2gb++ whether you like it or not if you want to play there. I don't see how some automatic hak downloader changes that.

    If the downloader informs player that the total size of haks required to download is 2gb and gives him choice to start downloading or cancel then I see no problem with that.

    Resource bloat cannot be fixed itself without NWN2 style of custom content.
    CablefishSoren
  • CablefishCablefish Member Posts: 37
    edited December 2017
    I'm happy with the top end of the spectrum (Edit: the simple server-send-links-sollution requiring less work by the devs).

    It's not gonna be worse than what we got and it worked fine in NWN2. It would be awesome if they'd come up with some nice ressource-identification-system but i'm not sure i want them to spend too much time on it.
    BelleSorciere
  • fot1fot1 Member Posts: 74
    I second everything hda_nwn said. Having a subfolder per server name/module name/some unique identifier for a PW sounds the optimal solution, and should't be the hardest part to implement of this feature.

    IMHO this should not only be to haks: tlks, haks and overrides should have one subfolder per PW. THe standard tlks, overrides and etc (meaning the ones that are not on any subfolder) should only be used for running local modules.
    AtrophiedericMaloney
  • SherincallSherincall Member Posts: 387
    Just a side note: There should be no reason for servers to have their own overrides any more, now that haks are loaded before character creation.
    The tlk is also only a single one, so no point in having it in subfolders.
  • voidofopinionvoidofopinion Member, Moderator Posts: 1,248

    Just a side note: There should be no reason for servers to have their own overrides any more, now that haks are loaded before character creation.
    The tlk is also only a single one, so no point in having it in subfolders.

    Good reminder.

    Is there perhaps some sort of master list of all these changes?

    And how up to date is nwn.wikia.com these days?
  • TheCapuletTheCapulet Member Posts: 43
    Personally, I think NWN needs to move away from hak additions and tlk edits to make "some" of the changes we see them usually associated with.

    Maybe I've been spoiled with modern tools, but the data required to make small changes to the game seem unreasonable. It shouldn't take external downloads to make simple data changes, audio editions, or texture edits.

    What would be ideal is if the module format was expanded so that it could be unpacked and added to. Think something like unzipping the mod file, then dropping in an /audio, /texture, and /data folder that can override texture content and can have scripts pointing to the resources in the folder. Then when a player joins any given server with low level custom content/changes, it can be downloaded to a module directory that the server checks each time it's connected to ensure it doesn't need to redownload those assets.
    Mizzajl
  • BelleSorciereBelleSorciere Member Posts: 2,108
    So if haks are done away with, how do the module creators integrate what is now hak content into their modules?
  • ThorssonThorsson Member Posts: 190
    NWN2 has this. Everything is saved in My Documents. There is one hak directory and common haks are used by any PW - if you change one you rename it, usually with shorthand for your PW as the start characters. Other files are mostly in the modules sub-folder. It works with few problems.
  • Barry_1066Barry_1066 Member Posts: 77
    There exists certain dangers in not maintaining exact hak content that a server requires - for Example: There are a number of various versions of a tileset - I was following a discussion of Worm's 4 season tileset - seems there has been changes made to that and it is not all working with the EE but was under 1.69.

    My PW, uses the set as Worm released, so not modified by later persons - my version works with EE and I have no issues - however if a hak consolidator, updater, distribution were to occure, my working tileset might be replaced with some "Updated" version that breaks my world.

    I know the materials I have, well almost all, work and are not a problem but I have no idea what other versions of the same material might cause - this scheme is dangerous.

    The ability to have a button for players to use and get the required haks - which might require more than one site - would be helpful. An auto install of the required haks and tlk, custom music and portraits would be nice if it is NOT from the server -- One hak we use for our scripts is empty at distributed keeping our scripts private.

    Also - downloading from the server would place an undue strain on my internet - I have a high speed commercial website for posting the haks to download - so the downloads are not on the server site for download.

    I acknowledge there are several possible scenarios for PWs proving their haks currently and not everyone has the solution I use.

    Recap:
    1) haks at the server cannot all be distributed - due to script security in custom hak with scripts - distributed version is empty.
    2) Server location downloads would greatly increase internet load and data cap
    3) Resource differences - versions that are needed for a world cannot necessary be updated or changed to other versions.

    CEP 2.4 differs from TAD updated version of the CEP and can and does break some modules made with CEP 2.4 - SO - there it is - one can not just assume the latest is the greatest. CEP is supposed to be fully backward compatible - the only exception is when Bioware (in the past) required changes - No such changes have been made since 1.69 so CEP must be fully backward compatible and the updates since 2.4 are not.

    This is an example - the CEP 2.4 vs updates - why an auto update system can not be forced.
    Maloney
  • Barry_1066Barry_1066 Member Posts: 77
    edited December 2017

    Personally, I think NWN needs to move away from hak additions and tlk edits to make "some" of the changes we see them usually associated with.

    Maybe I've been spoiled with modern tools, but the data required to make small changes to the game seem unreasonable. It shouldn't take external downloads to make simple data changes, audio editions, or texture edits.

    What would be ideal is if the module format was expanded so that it could be unpacked and added to. Think something like unzipping the mod file, then dropping in an /audio, /texture, and /data folder that can override texture content and can have scripts pointing to the resources in the folder. Then when a player joins any given server with low level custom content/changes, it can be downloaded to a module directory that the server checks each time it's connected to ensure it doesn't need to redownload those assets.

    I wish to remind you that one of the reasons PWs are using haks is to get the resources OUT of the module. There is a serious resource limitation in the module - this is why on my world, we have most scripts, many creatures, and other resources in haks. IT IS A ROYAL PAIN - I would love to have them in the module - we have to remove them from the hak and import them to the module to work with them, then export them and put them back in the hak - the ones in the module must be deleted. It is a pain but necessary to keep the module container from going over the limit.

    If you go over that limit - BOOM - module will not load anymore - all broken - boo hooo. I have been there and done that - not fun. So this suggestion is not going to work - even if Beamdog expans the module container limit so it will load with more content there are still max content limitations imposed by windows - don't know about Mac and Linux. Some of the CEP haks are pushing that MAX container limit already which is why there are so many CEP haks.
    Post edited by Barry_1066 on
    NeverwinterWights
  • TrentOsterTrentOster Administrator, Developer Posts: 433
    The original concept of a .hak file was to contain resources that a module would require, but which could not be included in a .mod. .hak files were envisioned as where custom art, sounds, music and other assets would be contained. The resource limitation in the module was not envisioned as a use for .hak files, but I'm glad it was a useable workaround. We'll have to think about the best means to solve all the issues.
    ZwerkulesNeverwinterWightsPhenomenMaloney
  • TheCapuletTheCapulet Member Posts: 43
    edited December 2017
    I wasn't indicating that I think haks should be done away with. Just that they're being used too much in scenarios where a lot of the content that a hak contains could be transfered on join time instead.

    My main point, which may not have been clear, is that I think the best direction to take is to expand the .mod format to include some of the responsibilities previously held by the .hak

    Some custom content will require more extensive downloads, and there's no way around that. But things like scripts, custom texture (for server-side overrides) and audio source, etc, is all lightweight enough that there's no reason to avoid transferring it over modern internet on the fly. Multiple downloads of the same custom content seems like a worry for module builders, not players. The simple solution for that is good mod management and only overriding rather than editing community based hak material.

    NWN2's solution was good too, though I don't think it went far enough in modernizing capabilities.
    Post edited by TheCapulet on
  • ThorssonThorsson Member Posts: 190

    My PW, uses the set as Worm released, so not modified by later persons - my version works with EE and I have no issues - however if a hak consolidator, updater, distribution were to occure, my working tileset might be replaced with some "Updated" version that breaks my world.

    This would simply be poor practice - new versions should be called something slightly different so they didn't over-write existing files. Equally if one server adapted a file it would also re-name it. It's not really difficult.
    Soren
  • RifkinRifkin Member Posts: 141

    Personally, I think NWN needs to move away from hak additions and tlk edits to make "some" of the changes we see them usually associated with.

    Maybe I've been spoiled with modern tools, but the data required to make small changes to the game seem unreasonable. It shouldn't take external downloads to make simple data changes, audio editions, or texture edits.

    What would be ideal is if the module format was expanded so that it could be unpacked and added to. Think something like unzipping the mod file, then dropping in an /audio, /texture, and /data folder that can override texture content and can have scripts pointing to the resources in the folder. Then when a player joins any given server with low level custom content/changes, it can be downloaded to a module directory that the server checks each time it's connected to ensure it doesn't need to redownload those assets.

    I wish to remind you that one of the reasons PWs are using haks is to get the resources OUT of the module. There is a serious resource limitation in the module - this is why on my world, we have most scripts, many creatures, and other resources in haks. IT IS A ROYAL PAIN - I would love to have them in the module - we have to remove them from the hak and import them to the module to work with them, then export them and put them back in the hak - the ones in the module must be deleted. It is a pain but necessary to keep the module container from going over the limit.

    If you go over that limit - BOOM - module will not load anymore - all broken - boo hooo. I have been there and done that - not fun. So this suggestion is not going to work - even if Beamdog expans the module container limit so it will load with more content there are still max content limitations imposed by windows - don't know about Mac and Linux. Some of the CEP haks are pushing that MAX container limit already which is why there are so many CEP haks.
    Changing the way the files are loaded, and how the data is stored might be worth looking at.

    There are many ways to avoid these file size limitations in the modern age of computers. I wonder how they could make the change and retain backwards compatibility though. Unless they would make some kind of auto-upgrader for old modules, with old haks.
  • FreshLemonBunFreshLemonBun Member Posts: 909
    I assume it would work that if the imaginary limit is 10 and you have 5 and it upgrades the limit to 20 your module still works with 5. If you have 10 and it upgrades to 20 you still have 10. If you have 10 and 5 in a hak and the module upgrades to 20 you have 10 and 5 in a hak. If you have 10 and 5 in a hak and the new limit is 20 but instead you add them to the module you will now have 15.

    I'm not sure why a program would not be able to handle a file with less than the maximum or why an older file that could only have less would not work with an improved program that can handle more. I think the limits seem to be in the program as opposed to somehow built into the file but I might be wrong.
  • RifkinRifkin Member Posts: 141

    I assume it would work that if the imaginary limit is 10 and you have 5 and it upgrades the limit to 20 your module still works with 5. If you have 10 and it upgrades to 20 you still have 10. If you have 10 and 5 in a hak and the module upgrades to 20 you have 10 and 5 in a hak. If you have 10 and 5 in a hak and the new limit is 20 but instead you add them to the module you will now have 15.

    I'm not sure why a program would not be able to handle a file with less than the maximum or why an older file that could only have less would not work with an improved program that can handle more. I think the limits seem to be in the program as opposed to somehow built into the file but I might be wrong.

    Sorry, you've mistaken my message towards the upgrade of the file limit. I'm specifically talking about changing the way haks, tlks and mods are structured, while attempting to retain backwards compatibility.

    If the idea is just to lift the limit, then that is completely backwards compatible. (Assuming there isn't some kind of 32-bit pointer/memory register problem with the new limit size ;) )

    Anyway, the idea that I was unpacking, was the idea of changing HAKS and TLKS to be more efficient, and to adopt a more readable and efficient structure. There are many design patterns that work better on modern PC's, that weren't an option back in the days of NWN1 development. It would be interesting to see if we could squeeze more performance out of NWN1. That being said, the proposed changes would likely be difficult to achieve while maintaining true backwards compatibility. Unless, I mentioned that they would provide some sort of module/hak/tlk/erf upgrader.
  • FreshLemonBunFreshLemonBun Member Posts: 909
    Sorry I mistook what you were getting at because you were replying to a post that seemed to be about getting around the limits on how many assets the modules can store.

    I think I see what you're saying now. It seems like a similar issue to what someone posted about new graphics formats, something that is smaller and quickly readable by a program. I assume if you could squeeze performance out of resource files a conversion tool would be the best option on the player side. I think to download files automatically it would probably be better to just stick to compression formats like NWN2.
  • PhenomenPhenomen Member Posts: 33
    I think ability to edit and compile scripts inside haks would help a lot.
    Barry_1066GM_ODA
  • voidofopinionvoidofopinion Member, Moderator Posts: 1,248
    Phenomen said:

    I think ability to edit and compile scripts inside haks would help a lot.

    The ability to easily edit any file in a hak from the toolset would be amazing!
    Barry_1066GM_ODA
  • OmnipsiOmnipsi Member Posts: 31
    The Toolset being able to compile module-specific haks/tlks, with unused content dropped, would be cool. It would be cooler if it could also specify module-specific assets to overwrite/modify the underlying resource. This would entail changing the hak (2da) format entirely though.

    It would be particularly fantastic if the work Sinfar has done with (presumably) a roided up resource manager plugin for NWNX + some ingenious coding to allow for their "ERFs" system was combined with the hak/tlk problem.

    Allow module builders to separate content from the module into "packages", treated like sub-modules / other projects within NWN. Packages are saved and opened/closed separately from the module and each other, and when open module resources can be dragged between them, allowing the module to be separated into multiple smaller pieces such that the entire module doesn't need to be loaded into memory at any one given time (by the Toolset, or by the server/client), thereby allowing these packages to be loaded/unloaded as specified by scripting during run-time to free up the space of, for example, zones of areas not currently being visited and any creatures, sounds, placeables, and unique scripts or other content that won't be used by the module when those areas aren't being used.

    Hak/tlk can then subsequently also be loaded by the server as packages also, communicated to the client on connection/load of the module, before character creation. Where a client has the package already (up to date with the server, or the server uses unmodified community haks) the checksum matches a full compatability checksum and they proceed. If the client doesn't have any/some of the haks content at all, it can communicate to the client where to download (if auto-download is a thing) or send a message. If the server also modifies some of the haks (doesn't use all of the resources from a community hak, or has made module-specific modifications to hak-provided resources) then that can be communicated also and checked for as well, so that new players can be directed to a download containing only the required content if they have none of it, and so that changes to hak assets by the module builders can be updated to clients.

    The changes to hak resources I'm talking about are not overwriting content, but additional data that modifies provided hak resources. Changes colour, applies a new texture, scales the hak asset, or something similar. I'm not sure if this is feasible or a easy, but it would be fantastic if persistent worlds could modify community resources to the extent possible without providing additional bulky, duplicate assets to download.

    The changes to how 2da's and their files are structured that this would necessitate would also make it possible for current haks with duplicate resources to simply include one "universal" of such duplicated content to be referenced by other hak resources as appropriate. For example with creatures with many variations, many of which are based on a single model with tiny aesthetic changes -- sometimes a model from the original game, no less. This would not end redundancy in old haks, but new ones and rebuilds could certainly do so and decrease their size at least to some extent (compensating for the additional bytes taken up by a new, more dynamic and modern resource file format and structure).

    It seems to me that the changes I've suggested to haks/resource management between the server/client and Toolset would be many of the same changes required to facilitate integrating the "ERF"(Sinfar)/resource manager into the Toolset/server as projects, hence why I introduced both in this... massive... rambling... wall of text...
  • AzimnAzimn Member Posts: 24
    I'm not the most versed in knowledge of this but didn't they already fix this in NWN2? Could they copy what they did?
  • FreshLemonBunFreshLemonBun Member Posts: 909
    @Azimn A community modder fixed it for Obsidian Entertainment a long time after the NWN2 launch.
This discussion has been closed.