@jpsweeney94
re: Manifest folder - That's normal. All the actual data is in the data>sha1 folder, and then right at the end of the process it will create the Manifest (literally one of the last things it does before finishing)
Good to hear the process is still working, but that is exceptionally slow. Without knowing the ins and outs, I can only broadly mention those things (IO/network/etc). You don't perhaps have anti-virus software that is being very aggressive and scanning every single file as it's created? That would easily massively slow down the process. I read windows search Indexing can do the same.
As the server host benefiting from William Dragonfang's work's, you also need to be sure the files are NOT in use either by having the toolset open or running nwserver with the haks/tlk. It WILL slow it down.
Thanks for the insights! @WilliamDraco@Daijin_Heartlight
So i restarted my PC with windows search indexing turned off, server is offline, and no anti-virus is running. I've let it run for the last 4-5+ hours and it's only at 220mb+... If it stayed at the same rate i would be fine with just leaving it run for say 24hrs, but it progressively gets worse and starts to crawl. I suspect i would have to leave it for weeks for it to finish.
I'll try a few more things but any other suggestions are welcome!
Update: I let it run for a while after this post. Ran for 12+ hours again and it was at around 400mb...
How strange. You do have more than 25% HD space remaining right? In my experience, sometimes HD's with more than 75% full tend to slow down or just not function for big programs correctly. Also your disk if not a SDD, but a standard HD is defragmented?
@Daijin_Heartlight SSD and it's about 8gb shy of 25% remaining. Would that really cause an issue? Perhaps i'll try and remove some items on it.
Another thing i am contemplating is trying to run it on another PC. Just need to borrow a friends or something i suppose.
I know nothing of SSD's other than they are faster, and are bad to defragment. Thats about all i know of SSD's. The slow speed of your updating nwsync with your hak/tlk files could be a result of something so simple that its not even given thought. Let us know if a friend's computer works, then we definitely know its something easily overlooked.
@Balanor@Grizzled_Dwarflord@jpsweeney94
It's no video, but I've laid out a complete walkthrough for using nwsync - This doesn't include it's whole potential, but it covers everything needed for getting you up and running.
Thanks, Sir Draco! I'll be forwarding this to our DM to get this ball rolling. I'm sure he'll be checking in for some questions. Just looking at the 4 page manifesto my eyes rolled into the back of my head. But I'm sure they'll figure it out. Cheers!
Finally, it's ashame that no one from Beamdog was able to provide any kind of support or tutorial regarding this.
@Daijin_Heartlight Well i let it run over thanksgiving break for roughly 3 days and it finished. I periodically checked on it with remote desktop and it seemed to be going at the same speed. I didn't take into account the compression it must be doing.
Either way, i've got it working in game now with the server! Thanks a bunch for the guide @WilliamDraco
Just some follow up issues. What exactly is not included with NWSync? I've noticed it didnt bring music or portraits with it... Any way to force NWSync to grab these? Any other custom content it might not be bringing with?
EDIT: Cheers to beamdog for the new update, good stuff:
Music and ambient sounds are now loaded through ResMan and can thus be in hak and nwsync.
I tried re-running NWSync write but it came back with "We wrote 0B of new data"... Anything we need to do to get this working?
Portraits could already be a part of NWSync, with music and ambient sound as you saw have been added with the new stable 1.79. One Caveat with Portraits: If included in NWSync, I don't think(??) they appear in character creation.
As to why it didn't write anything new - I need to continue some work on NWSync GUI. NWSync itself allows you to pick multiple source files/folders and add them all together, so you could pick the .mod file and the /music folder and it'd create one repo of them together. I haven't yet got around to making NWSync GUI allow multiple source file/folders.
Since it only accepts one source ATM, it figures out the Hak/Tlk files automatically based on the .mod file, but it can't know which portraits or music files it's meant to include automatically.
Solution: Put the portraits, music and ambient sounds you want into a hak file. add the hak file to your module, then re-run nwsync with the mod as source. It should pickup the new hak contents and include them.
If you have paid hosting for nwserver and need to watch storage limits, it doesn't need the real portraits/music/ambients and can be fed a fake-empty hak if you need I.E:
1. Real extras.hak has 100mb of music inside of it. Associate extras.hak with the module and run nwsync_write. should pick it up.
2. create a fake extras.hak that is 0mb (no contents, just the same name) and upload this to the host.
3. nwserver only checks that the hak name is there, and doesn't check or use any contents (only clients need the contents). You're all good and have saved yourself 100mb of uploading/hosting
Note this trick only works with certain content types. I don't know a comprehensive list of which are or are not required by the server.
Thanks for the reply @WilliamDraco. Could you just change the source folder to the music/portrait folder after you're done with the initial write from the .mod source and select the same nwsyncdata folder as destination? or would that overwrite files or cause issues?
Also, do you need to place "Override" files into a hak as well? Or does NWSync grab those from the override folder?
Unfortunately, simply changing the source folder would not work, as nwsync bundles collections of files together into a single manifest as it runs (So that, for example, the same nwsync server could serve two or more completely separate modules/haklists without providing all clients with all data). It has it be done all in one-run to be considered a single package together.
NWSync does not automatically go through /override (or /dev, the new folder with 1.79). Note that Override can contain all sorts of files, some of which would not work if delivered by NWSync. Keep in mind that it's primary job is a hak-serving-system. If the file does not work when put in a hak, it probably wouldn't work via NWSync (hence why music/ambiant was made hak-compatible at the same time as nwsync-compatible)
Thanks for all the help, much appreciated. Moving these files into Haks, hopefully not too many more issues...
One issue i've discovered is the Hak Pak editior complaining about .bmu files being added to a Hak. Something I'm missing? https://www.dropbox.com/s/6khbs7mpp1kpp9q/Hak Pak Editor BMU error.png?dl=0
My personal assumption is the hak pak editor is ancient and no developer has even sneezed in its direction in years - the 'about' page still has all the bioware copyright stuff and doesn't even mention beamdog at all.
Suggest you report that one via official channels: https://beamdog.atlassian.net/servicedesk/customer/portal/1
Thank you for the resource @shadguy!
Is there a guide/instructions on how to pack files into a Hak using those tools? After downloading the latest release there are tons of EXE and i have no idea which one to pick to package BMU into a hak and what cmd lines to run once opened.
@jpsweeney94 - use nwn_erf to build .mod, .erf, or .hak files (they're all the same file format). I can share some sample commands when I'm not at work (ie it ain't in front of me), or you can try and see if --help (or similar) helps produce some advice to get you started, if you're comfy hacking away at stuff on your own.
Seems i am prone to errors Sorry for question spam, appreciate everyone's help.
I can't open any of the EXE. The cmd window flashes and closes right away. Both 'neverwinter.windows.amd64.zip' and 'neverwinter.windows.i386.zip' versions. Also tried restarting PC with same result and running as administrator.
Open a command terminal - in windows explorer, select the address bar, and type cmd. That will bring up a command line interface for you. For a more powerful thing, google powershell, but maybe another day.
Once you have the command line up, you can try typing in a command like the one I did.
If you do a lot of hak maintenance, you can save frequently used commands to regenerate your HAKLs in a batch file (.bat). But first, get the command line up and try creating a hak with a simple command line.
Thanks @shadguy, quickly discovered i needed to just enter the entire cmd in a blank CMD window. Seems i am more lack-of-knowledge prone than error prone
So, think I've finally got NWSync all setup and tested on my server. Works great! Thanks everyone for the assistance.
Only nit picky thing I'm curious about is portraits. I loaded them into a Hak and it works great on a clean install to show existing characters custom portraits in the PW... But none are available at character creation.
Is this just a limitation of NWSync or is there someway to get these loaded before the module at character creation? Seems like NWSync should be able to handle all PW files so users dont have to do additional downloading and messing around with files (Isn't that the whole point?)
It's a limitation with how the game reads the portraits, yeah. If they want to pick the portrait on character creation, they have to have it downloaded the old-fashioned way (have it in their portraits folder).
If desired, you can include all portraits in the nwsync and then also provide them via extra download. All players can SEE the portraits, dedicated players who really want the special portrait can go to the special effort for it.
(I think the reason it's not done is because it's hard to know which portraits should be pickable and which not)
Looking for advice on uploading files for NWSync to web server. Every time we make changes to hak, tlk, and other mod files i need to run NWSync again, which is no problem. The process is fairly quick after the initial NWSync write is done.
But, after i do this i need to re-upload the entire directory again to my web server, which takes a considerable amount of time since it needs to go through so many folders and files. So, is there some way to upload changes to NWSync files without having to re-upload the entire repo?
Comments
re: Manifest folder - That's normal. All the actual data is in the data>sha1 folder, and then right at the end of the process it will create the Manifest (literally one of the last things it does before finishing)
Good to hear the process is still working, but that is exceptionally slow. Without knowing the ins and outs, I can only broadly mention those things (IO/network/etc). You don't perhaps have anti-virus software that is being very aggressive and scanning every single file as it's created? That would easily massively slow down the process. I read windows search Indexing can do the same.
So i restarted my PC with windows search indexing turned off, server is offline, and no anti-virus is running. I've let it run for the last 4-5+ hours and it's only at 220mb+... If it stayed at the same rate i would be fine with just leaving it run for say 24hrs, but it progressively gets worse and starts to crawl. I suspect i would have to leave it for weeks for it to finish.
I'll try a few more things but any other suggestions are welcome!
Update: I let it run for a while after this post. Ran for 12+ hours again and it was at around 400mb...
Another thing i am contemplating is trying to run it on another PC. Just need to borrow a friends or something i suppose.
Thanks, Sir Draco! I'll be forwarding this to our DM to get this ball rolling. I'm sure he'll be checking in for some questions. Just looking at the 4 page manifesto my eyes rolled into the back of my head. But I'm sure they'll figure it out. Cheers!
Finally, it's ashame that no one from Beamdog was able to provide any kind of support or tutorial regarding this.
Either way, i've got it working in game now with the server! Thanks a bunch for the guide @WilliamDraco
Just some follow up issues. What exactly is not included with NWSync? I've noticed it didnt bring music or portraits with it... Any way to force NWSync to grab these? Any other custom content it might not be bringing with?
EDIT: Cheers to beamdog for the new update, good stuff: I tried re-running NWSync write but it came back with "We wrote 0B of new data"... Anything we need to do to get this working?
As to why it didn't write anything new - I need to continue some work on NWSync GUI. NWSync itself allows you to pick multiple source files/folders and add them all together, so you could pick the .mod file and the /music folder and it'd create one repo of them together. I haven't yet got around to making NWSync GUI allow multiple source file/folders.
Since it only accepts one source ATM, it figures out the Hak/Tlk files automatically based on the .mod file, but it can't know which portraits or music files it's meant to include automatically.
Solution: Put the portraits, music and ambient sounds you want into a hak file. add the hak file to your module, then re-run nwsync with the mod as source. It should pickup the new hak contents and include them.
If you have paid hosting for nwserver and need to watch storage limits, it doesn't need the real portraits/music/ambients and can be fed a fake-empty hak if you need I.E:
1. Real extras.hak has 100mb of music inside of it. Associate extras.hak with the module and run nwsync_write. should pick it up.
2. create a fake extras.hak that is 0mb (no contents, just the same name) and upload this to the host.
3. nwserver only checks that the hak name is there, and doesn't check or use any contents (only clients need the contents). You're all good and have saved yourself 100mb of uploading/hosting
Note this trick only works with certain content types. I don't know a comprehensive list of which are or are not required by the server.
Also, do you need to place "Override" files into a hak as well? Or does NWSync grab those from the override folder?
NWSync does not automatically go through /override (or /dev, the new folder with 1.79). Note that Override can contain all sorts of files, some of which would not work if delivered by NWSync. Keep in mind that it's primary job is a hak-serving-system. If the file does not work when put in a hak, it probably wouldn't work via NWSync (hence why music/ambiant was made hak-compatible at the same time as nwsync-compatible)
One issue i've discovered is the Hak Pak editior complaining about .bmu files being added to a Hak. Something I'm missing?
https://www.dropbox.com/s/6khbs7mpp1kpp9q/Hak Pak Editor BMU error.png?dl=0
Suggest you report that one via official channels: https://beamdog.atlassian.net/servicedesk/customer/portal/1
A more immediate solution might be to find a nwhak.exe alternative? I've never looked at them but a google reveals https://neverwintervault.org/project/nwn2/other/tool/nwn-packer-v20
https://github.com/niv/neverwinter.nim
-Dave
Is there a guide/instructions on how to pack files into a Hak using those tools? After downloading the latest release there are tons of EXE and i have no idea which one to pick to package BMU into a hak and what cmd lines to run once opened.
-Dave
> C:\Games\NWN\utils\nimtools\nwn_erf.exe -e hak --quiet -c -f C:\Games\NWN\hak\dnee_parts.hak .\dnee_parts_helm .\dnee_parts_mstaff .\dnee_parts_race .\dnee_parts_shield .\dnee_parts_wpn
This sample takes all files it finds from several listed subdirs and places them in my dnee_parts.hak.
-Dave
I can't open any of the EXE. The cmd window flashes and closes right away. Both 'neverwinter.windows.amd64.zip' and 'neverwinter.windows.i386.zip' versions. Also tried restarting PC with same result and running as administrator.
Any ideas why?
Once you have the command line up, you can try typing in a command like the one I did.
If you do a lot of hak maintenance, you can save frequently used commands to regenerate your HAKLs in a batch file (.bat). But first, get the command line up and try creating a hak with a simple command line.
-Dave
Got it all sorted. Thanks for the help!
Only nit picky thing I'm curious about is portraits. I loaded them into a Hak and it works great on a clean install to show existing characters custom portraits in the PW... But none are available at character creation.
Is this just a limitation of NWSync or is there someway to get these loaded before the module at character creation? Seems like NWSync should be able to handle all PW files so users dont have to do additional downloading and messing around with files (Isn't that the whole point?)
Thanks!
If desired, you can include all portraits in the nwsync and then also provide them via extra download. All players can SEE the portraits, dedicated players who really want the special portrait can go to the special effort for it.
(I think the reason it's not done is because it's hard to know which portraits should be pickable and which not)
Will this work with portraits.2da in a Hak? or does that need to go in the override folder?
Should work in a hak same as override.
But, after i do this i need to re-upload the entire directory again to my web server, which takes a considerable amount of time since it needs to go through so many folders and files. So, is there some way to upload changes to NWSync files without having to re-upload the entire repo?
-Dave
I'm not sure what you mean. How would i generate the NWSync folder on the web server? I use an FTP