Having some issues again with NWSync since new update to module Hoping for some help.
I ran NWSync and upload files to my webserver via FTP. FTP gives me this prompt about 5 times: https://www.dropbox.com/s/c8jq1sznlebrojk/nwsync_overwrite.png?dl=0 (I'm pretty sure i tried keeping either files and same error... If I'm remembering incorrectly, I atleast know i did the "overwrite" option to keep the source file)
Also made some changes like adding music to a hak and what not, but those worked before this newly added hak.
Any idea what could be the problem?
Thanks!
EDIT: Re-ran NWSync write and saw these in the log as well, could this be it?
W, [2019-12-31T18:20:44] -- nwsync_write: Duplicate resref entry in erf, but differing offset/size: resref=m_crowishwizard_.tga offset=39323109 size=24620 (idx=1172) would collide with offset=39224761 size=98348; renamed to __erfdup__1172.tga
W, [2019-12-31T18:20:44] -- nwsync_write: Duplicate resref entry in erf, but differing offset/size: resref=m_crowishwizard_.tga offset=39347729 size=6188 (idx=1173) would collide with offset=39224761 size=98348; renamed to __erfdup__1173.tga
W, [2019-12-31T18:20:44] -- nwsync_write: Duplicate resref entry in erf, but differing offset/size: resref=m_crowishwizard_.tga offset=39353917 size=1580 (idx=1174) would collide with offset=39224761 size=98348; renamed to __erfdup__1174.tga
W, [2019-12-31T18:20:44] -- nwsync_write: Duplicate resref entry in erf, but differing offset/size: resref=m_wisemalewizard.tga offset=44722885 size=24620 (idx=1336) would collide with offset=44624537 size=98348; renamed to __erfdup__1336.tga
W, [2019-12-31T18:20:44] -- nwsync_write: Duplicate resref entry in erf, but differing offset/size: resref=m_wisemalewizard.tga offset=44747505 size=6188 (idx=1337) would collide with offset=44624537 size=98348; renamed to __erfdup__1337.tga
W, [2019-12-31T18:20:44] -- nwsync_write: Duplicate resref entry in erf, but differing offset/size: resref=m_wisemalewizard.tga offset=44753693 size=1580 (idx=1338) would collide with offset=44624537 size=98348; renamed to __erfdup__1338.tga
W, [2019-12-31T18:20:44] -- nwsync_write: Duplicate resref entry in erf, but differing offset/size: resref=po_hu_m_oarcher_.tga offset=102731633 size=24620 (idx=3111) would collide with offset=102633285 size=98348; renamed to __erfdup__3111.tga
W, [2019-12-31T18:20:44] -- nwsync_write: Duplicate resref entry in erf, but differing offset/size: resref=po_hu_m_oarcher_.tga offset=102756253 size=6188 (idx=3112) would collide with offset=102633285 size=98348; renamed to __erfdup__3112.tga
W, [2019-12-31T18:20:44] -- nwsync_write: Duplicate resref entry in erf, but differing offset/size: resref=po_hu_m_oarcher_.tga offset=102762441 size=1580 (idx=3113) would collide with offset=102633285 size=98348; renamed to __erfdup__3113.tga
EDIT: Re-ran NWSync on old working-version as well (keeping separate NWSync data folders) and those lines appear on it as well. So don't think that's the issue?
Are you sure you are copying everything that nwsync creates to the server? Specifically are you also updating the new 'latest' or the new manifests subfolder? Something on the server isn't matching up with what NWSync is creating locally. It might even be the case where nwsync has deleted things in it's process and the server hasn't removed those - I thought that wouldn't be an issue but maybe.
As to the 'Duplicate resref entry' notices: They are just telling you that your source files (haks) have the same file multiple times. And not only are they there multiple times, but they are DIFFERENT each time. The game has always handled this by just using the 'newest' one according to hak-ordering rules. NWSync is throwing those warnings so that you know, but it's not a 'problem' unless you were expecting to use both of the originals separately.
You should go see about cleaning those up though, by removing all the unused duplicates that it mentions.
As to Shadguy's comment: It's possible to run nwsync directly on the server if it allows any kind of remote terminal control. If yours is just ftp, probably no such luck.
Using your NWSync tool, i target my "nwsyncarda" folder and i upload that entire folder to my web-server. I even tried a fresh NWSync write to see if it was old files or something, and same thing.
I'm assuming it's something to do with the 1st screen shot i posted? Im not sure why it would be replacing files on a new folder upload. Or why the same file would be uploaded twice. Or why one has 0 bytes. Wonder if it's my FTP? Maybe ill try a different one.
As to Shadguy's comment: It's possible to run nwsync directly on the server if it allows any kind of remote terminal control. If yours is just ftp, probably no such luck.
Oh, cool. Ill have to look into this, would be nice. Would i be able to see the web-server directory in windows explorer? Thinking so i can target it with your tool.
Is https support still planned on being added to NWSync?? If so, would google drive be supported? lol would be awesome to not have to upload entire repo every time and just the changes with google drive on desktop...
https support was added to NWSync with 8193 - Both http and https are currently supported.
As to Google drive... I doubt it that will ever happen. Maybe try to search for a ftp client that does differential uploads?
@WilliamDraco Thanks for the suggestion. I finally managed to "mount" my FTP/server folder to my desktop in windows 10 through "add a network location" when right clicking on "This PC" in windows explorer. I did some small tests and it does work for adding/deleting files.
However, when i try to select the network folder in your NWSync GUI program, it doesn't show up in the choose destination folder dialogue. Any ideas why? It does show up in choose source folder dialogue but not the destination one for some reason.
Probably a question more well suited to the github page for the GUI, rather than this thread about the CLI program itself.
While here though: It must be a bug/issue in the library I'm using to build the GUI. Difference being that the Source is a file, while Destination is a folder, so they are different elements. I'll follow-up but it's not likely a matter I can directly resolve.
I have this bug, tryed almost everything, dowloading the game again, deleting the folder, tryed configuring the firewall, copy the file from another player's PC, but does not solve!
neverwinter@host:~/nwsync$ ./nwsync_write -v --description="HOST Server Data V20200329" ~/www/ ~/.local/share/Neverwinter\ Nights/hak/host_v3.0.0.hak
I, [2020-03-29T21:44:59] -- nwsync_write: Reindexing
I, [2020-03-29T21:44:59] -- nwsync_write: Adding hak: /home/neverwinter/.local/share/Neverwinter Nights/hak/host_v3.0.0.hak
I, [2020-03-29T21:44:59] -- nwsync_write: Preparing data set to expose
E, [2020-03-29T21:44:59] -- nwsync_write: ResRef README.0 is not resolvable (we don't know the file type); origin: Erf:/home/neverwinter/.local/share/Neverwinter Nights/hak/host_v3.0.0.hak
A readme is useless to deliver as part of nwsync. I'd therefore recommend removing it from the hak.
If you are super desperate to keep it together for some reason, you can try using a text-like format that is supported, like .nss. Better to find an alternative method of storing the documentation, though.
A readme is useless to deliver as part of nwsync. I'd therefore recommend removing it from the hak.
If you are super desperate to keep it together for some reason, you can try using a text-like format that is supported, like .nss. Better to find an alternative method of storing the documentation, though.
Thank You, I was doing just that when I read you.
I had to remove README.res
Now that my world is on EE finally, I tried to set up nwsync. With the GUI that provides @WilliamDraco it is makeable.
I stumbled upon some things, that can be a little cumbersome... I opened a issue at github for the help message window and a message window for errors.
Because, at my first use, it didn't do its job and I didn't realized anything... Okay, later I took a look in the destination folder and saw the log with the error. I suggested to show a message, that something didn't work and to take a look at the log.
The other thing I mentioned is, the help message window is a bit oversized. How about resizing or make it resizable?
Next, two things I want to ask. Before I start, I run the task not from my server, i took my working machine.
First, I set up NWN:EE to use my old hak/tlk folders so I don't have to copy them to my user folder (I use more than one user on my Win10 laptop). The sync didn't realized that. I think that's how it should work?
Second one: I have a huge amount of haks (mostly the CEP) and the write process is extremely slow. I think 10% after an hour.
If I stop the process, would the sync begin at the start, or when I stopped it?
Thanks for that program and the GUI. And of course, hopefully the answers to my questions.
I've responded to your github issue there - Thanks
As to the other matters.
1) My understanding was that nwsync should follow alias' in its resman the same way that the game does, although I've never tested that myself. That'd be something for Niv to comment on/resolve if so, as the GUI simply prepares the command and hands it off to nwsync to follow.
2) There's some combination of system components that does cause a really slow write process, especially using a hard-drive (rather than SSD) but that doesn't seem to always be the core problem. For instance, I've prepared 12GB of haks->nwsync in ~30 minutes, but some report 1GB taking hours and hours...
EDIT: I also remember: Any kind of virus-scanning program that might be actively scanning every file NWsync creates will make it excruciatingly slow, as it creates a *LOT* of files to scan. Temporarily disable any such 'real-time scanners'
On the positive side, you can stop and resume the process and it should (mostly) pick up where it left off.
I think it is good to talk about the suggestions on github. I also gave you more info there
For my question one, I will wait till niv can answer it. For now I'm using my user folder and delete those haks later, when the sync is finished.
For number two, I think I'll do more tests. The most users will have CEP, so I will use a module with only the haks we provide... Then the down-/upload should be smaller.
For the other reasons of a slowdown...
Hmm, I use it on my HDD (only one). So it reads and writes at once (I had yet no success, to get win10 connected with my webspace).
The only virus scanner I use, is the built in from win10. Don't know, if it is scanning the whole time...
At least, I tried to start over where I was last night, but it began from the beginning. Okay, that could be a typical beginners problem Won't stop me! I got World Gate to run, then that sync will be possible, too.
Edit:
At least with that reduced hak-mod, about 3GB, it was fast. About one hour only. So indexing the CEP slowed down the process. Now it's time to upload the roundabout 45.000 files o.O XD
I am getting a 503 time out error for my server, 'Town of Luce'. *update, it only seems to do it on some computers. It might be an error on the http server side as well.
@niv I activated NwSync with my PW but most of my players do have problems with it. Some say, it even won't start to download or like me, I got this 503 timeout error...
How does NwSync save the data actually? Doesn't it look at the hak-folder, if the data exists? I created the sync data on the same machine I develope and play. The data is the same as that on the server. Only, due file limit on my webserver, I removed CEP from sync. Could that be a problem?
The 503 timeout error is a http webserver thing, Not nwsync. Maybe your webserver is mis-understanding all of nwsync's requests as some sort of DOS attack and putting up temporary blocks or something. Whatever the case, it's something to sort out with your host and server, not nwsync itself.
As to the second part, publishing an nwsync repo REPLACES haks, it is not in addition to. If you use CEP content, you must include CEP content in the nwsync repo or else it will not be loaded by clients.
UNLESS
you use the additional server flag to also publish the list of haks. per the docs
-nwsyncpublishhaks
This optional switch is purely for advanced users. Normally, once a server is told to advertise a NWSync URL, it will stop sending the list of required haks and tlk to clients (otherwise, clients would attempt to load them in addition to the nwsync manifest). If you have a very special setup, you can override this behaviour with this switch.
For normal operation, it is strongly advised to not do this. Content should be in NWSync, otherwise it cannot deduplicate.
Again though, that's not recommended and is a pathway to strange results.
@WilliamDraco Ah , I understand. Okay, then I will have to switch to another option. Because my webspace is limited to a specified number of files and with CEP its run out of files...
Then yet, NwSync isn't working for our PW, without problems. Thank you for your explanations!
Now I think about using a old tool from 1.69 times, named WorldGate that worked relative simple and alike NwSync. Only disadvantage is, it has to run before the Client is started, not from within it.
Has anyone ever had this issue when setting up nwsync on Windows? For some reason it can't add my TLK file for this one module, but there's no such issue with other modules.
Error: unhandled exception: 'custom.tlk' is not a resolvable resref [ValueError]
I'm getting closer. Now, I actually only get a port 80 error. I used a few websites to check if port 80 is open and they all confirmed that it is. Stumped now.
I have used this before and am trying to setup it up on a linux server again. It becomes very vague on the linux stuff when you just say run the write command. To us people that don't code , it would be nice for examples or videos.
Comments
I ran NWSync and upload files to my webserver via FTP. FTP gives me this prompt about 5 times: https://www.dropbox.com/s/c8jq1sznlebrojk/nwsync_overwrite.png?dl=0 (I'm pretty sure i tried keeping either files and same error... If I'm remembering incorrectly, I atleast know i did the "overwrite" option to keep the source file)
After upload, i try to join the server and get this error: https://www.dropbox.com/s/o1105p3zy7cazvh/join_error.png?dl=0
Almost like NWSync has files named the same and they need to overwrite or something? Never had this issue until recently. Only recent change that *might* be causing it is a new hak (contents): https://www.dropbox.com/s/t0tfdl3397q6cjm/MEX10_TOP_2DA_hak.png?dl=0
Also made some changes like adding music to a hak and what not, but those worked before this newly added hak.
Any idea what could be the problem?
Thanks!
EDIT: Re-ran NWSync write and saw these in the log as well, could this be it?
EDIT: Re-ran NWSync on old working-version as well (keeping separate NWSync data folders) and those lines appear on it as well. So don't think that's the issue?
As to the 'Duplicate resref entry' notices: They are just telling you that your source files (haks) have the same file multiple times. And not only are they there multiple times, but they are DIFFERENT each time. The game has always handled this by just using the 'newest' one according to hak-ordering rules. NWSync is throwing those warnings so that you know, but it's not a 'problem' unless you were expecting to use both of the originals separately.
You should go see about cleaning those up though, by removing all the unused duplicates that it mentions.
As to Shadguy's comment: It's possible to run nwsync directly on the server if it allows any kind of remote terminal control. If yours is just ftp, probably no such luck.
Using your NWSync tool, i target my "nwsyncarda" folder and i upload that entire folder to my web-server. I even tried a fresh NWSync write to see if it was old files or something, and same thing.
I'm assuming it's something to do with the 1st screen shot i posted? Im not sure why it would be replacing files on a new folder upload. Or why the same file would be uploaded twice. Or why one has 0 bytes. Wonder if it's my FTP? Maybe ill try a different one.
Oh, cool. Ill have to look into this, would be nice. Would i be able to see the web-server directory in windows explorer? Thinking so i can target it with your tool.
As to Google drive... I doubt it that will ever happen. Maybe try to search for a ftp client that does differential uploads?
@WilliamDraco Thanks for the suggestion. I finally managed to "mount" my FTP/server folder to my desktop in windows 10 through "add a network location" when right clicking on "This PC" in windows explorer. I did some small tests and it does work for adding/deleting files.
However, when i try to select the network folder in your NWSync GUI program, it doesn't show up in the choose destination folder dialogue. Any ideas why? It does show up in choose source folder dialogue but not the destination one for some reason.
While here though: It must be a bug/issue in the library I'm using to build the GUI. Difference being that the Source is a file, while Destination is a folder, so they are different elements. I'll follow-up but it's not likely a matter I can directly resolve.
neverwinter@host:~/nwsync$ ./nwsync_write -v --description="HOST Server Data V20200329" ~/www/ ~/.local/share/Neverwinter\ Nights/hak/host_v3.0.0.hak
I, [2020-03-29T21:44:59] -- nwsync_write: Reindexing
I, [2020-03-29T21:44:59] -- nwsync_write: Adding hak: /home/neverwinter/.local/share/Neverwinter Nights/hak/host_v3.0.0.hak
I, [2020-03-29T21:44:59] -- nwsync_write: Preparing data set to expose
E, [2020-03-29T21:44:59] -- nwsync_write: ResRef README.0 is not resolvable (we don't know the file type); origin: Erf:/home/neverwinter/.local/share/Neverwinter Nights/hak/host_v3.0.0.hak
See this file for all the supported filetypes: https://github.com/niv/neverwinter.nim/blob/master/neverwinter/restype.nim
A readme is useless to deliver as part of nwsync. I'd therefore recommend removing it from the hak.
If you are super desperate to keep it together for some reason, you can try using a text-like format that is supported, like .nss. Better to find an alternative method of storing the documentation, though.
Thank You, I was doing just that when I read you.
I had to remove README.res
Now to continue with the guide ...
Can a LOG be activated?
I stumbled upon some things, that can be a little cumbersome... I opened a issue at github for the help message window and a message window for errors.
Because, at my first use, it didn't do its job and I didn't realized anything... Okay, later I took a look in the destination folder and saw the log with the error. I suggested to show a message, that something didn't work and to take a look at the log.
The other thing I mentioned is, the help message window is a bit oversized. How about resizing or make it resizable?
Next, two things I want to ask. Before I start, I run the task not from my server, i took my working machine.
First, I set up NWN:EE to use my old hak/tlk folders so I don't have to copy them to my user folder (I use more than one user on my Win10 laptop). The sync didn't realized that. I think that's how it should work?
Second one: I have a huge amount of haks (mostly the CEP) and the write process is extremely slow. I think 10% after an hour.
If I stop the process, would the sync begin at the start, or when I stopped it?
Thanks for that program and the GUI. And of course, hopefully the answers to my questions.
dunahan
As to the other matters.
1) My understanding was that nwsync should follow alias' in its resman the same way that the game does, although I've never tested that myself. That'd be something for Niv to comment on/resolve if so, as the GUI simply prepares the command and hands it off to nwsync to follow.
2) There's some combination of system components that does cause a really slow write process, especially using a hard-drive (rather than SSD) but that doesn't seem to always be the core problem. For instance, I've prepared 12GB of haks->nwsync in ~30 minutes, but some report 1GB taking hours and hours...
EDIT: I also remember: Any kind of virus-scanning program that might be actively scanning every file NWsync creates will make it excruciatingly slow, as it creates a *LOT* of files to scan. Temporarily disable any such 'real-time scanners'
On the positive side, you can stop and resume the process and it should (mostly) pick up where it left off.
For my question one, I will wait till niv can answer it. For now I'm using my user folder and delete those haks later, when the sync is finished.
For number two, I think I'll do more tests. The most users will have CEP, so I will use a module with only the haks we provide... Then the down-/upload should be smaller.
For the other reasons of a slowdown...
Hmm, I use it on my HDD (only one). So it reads and writes at once (I had yet no success, to get win10 connected with my webspace).
The only virus scanner I use, is the built in from win10. Don't know, if it is scanning the whole time...
At least, I tried to start over where I was last night, but it began from the beginning. Okay, that could be a typical beginners problem
Edit:
At least with that reduced hak-mod, about 3GB, it was fast. About one hour only. So indexing the CEP slowed down the process. Now it's time to upload the roundabout 45.000 files o.O XD
@niv I activated NwSync with my PW but most of my players do have problems with it. Some say, it even won't start to download or like me, I got this 503 timeout error...
How does NwSync save the data actually? Doesn't it look at the hak-folder, if the data exists? I created the sync data on the same machine I develope and play. The data is the same as that on the server. Only, due file limit on my webserver, I removed CEP from sync. Could that be a problem?
As to the second part, publishing an nwsync repo REPLACES haks, it is not in addition to. If you use CEP content, you must include CEP content in the nwsync repo or else it will not be loaded by clients.
UNLESS
you use the additional server flag to also publish the list of haks. per the docs Again though, that's not recommended and is a pathway to strange results.
Then yet, NwSync isn't working for our PW, without problems. Thank you for your explanations!
Now I think about using a old tool from 1.69 times, named WorldGate that worked relative simple and alike NwSync. Only disadvantage is, it has to run before the Client is started, not from within it.
Error: unhandled exception: 'custom.tlk' is not a resolvable resref [ValueError]
Anyone know what can cause this error?
Thanks for the stuff so far .
It created folders modules ,hak, tlk , sha1 ,manifest
manifest folder is empty? Sha folder is empty?
Edit : I watched the silent video and heres where I made it to.
EDIT : After 15 or so tries , I got the windows gui to start doing the process. It completed and now I'm loading files to apache /webmin server .
My setup
PC Windows 10 = mydesktop
VPS Ubuntu 18.04 LTS = gaming server + file server