Skip to content

nwsync feedback & discussion thread

245

Comments

  • AzanixAzanix Member Posts: 3
    edited November 2018
    It is on Windows and the latest release works fine, how come it does not check for existing files in our hak-folder? the server forces a full download of it all it seems even though some people got the relevent haks.

    Will that be possible in the future ?

    btw, i think we all appreciate all the hard work niv <3
  • JapualtahJapualtah Member Posts: 165
    niv said:

    How big is your nwsync/nwsyncmeta.sqlite3 file?

    Moving the NWN folder to an SSD solved the problem, I now get the same speed I had before 8186, so be aware that using a HDD is very penalizing, that's maybe worth a note in the documentation.
    If you wonder why I was still using a HDD, it's just being conservative. Working on NWN involves reading/writing a very large number of files frequently and I don't trust SSDs on the long term.
    niv
  • nivniv Member, Moderator, Developer Posts: 410
    edited November 2018
    Thanks, Japualtah. Yeah, SSDs help a lot. Unfortunately, if you have multiple GB of data, indexing them (even with the help of sqlite primary keys) takes a bit.

    I'll look into improving this (it's already on the list and there's definitely room to make it go faster on spinning rust) but we're limited here by the storage format and simply the amount of data that gets lumped up over time.

    Edit: Re the HAK question @Azanix. Yes, that will also be possible to do. Not too sure when and if there's a real benefit by then, but I'll give it some thought. It's definitely also on The List.
    JapualtahdTd
  • sknymicksknymick Member Posts: 25
    I have a question regarding "Standard" or "Common" resources like CEP or CTP. If I want to play on 2 or 7 servers that utilize CEP, am I going to have to download their resources (around 6gb) each time, or will NWSync recognize/associate that data? That might end up being a truckload of storage to enable me to login to multiple servers. Not to mention, it takes me an hour to download on my crappy connection :smile:
  • SherincallSherincall Member Posts: 387
    You will only download each asset (model, texture, etc) once - whether they are part of a 'standard' resource like CEP/CTP or if the server repackaged them. So there is no duplication of data at all.

    Kind of - Until all servers you play on have moved to nwsync, you will still need to have the .hak files. So you might end up with CEP in nwsync and CEP in hak/. Once all the CEP servers switch to nwsync, you can delete the haks.
    dunahan
  • MartwopiolaMartwopiola Member Posts: 2
    edited December 2018
    Hello everyone. I've ran into an issue trying to play NWN PW that use NwSync. Everytime I try to get into one of these servers, I get a message saying that NwSync Failed to connect.

    I've checked the error log and found that I'm missing an .sql.

    Here's the error:

    "nwsync: sqlite3 error [14]: os_win.c:45373: (3) winOpen(C:\Users\myuser\Documents\Neverwinter Nights\nwsync\nwsyncmeta.sqlite3) - The system can not find the path specified
    nwsync: sqlite3 error [14]: cannot open file at line 45374 of [fb90e7189a]
    nwsync: Failed to set up sqlite.
    Failed to configure NWSync in your user directory."

    What do? I don't have this "nwsyncmeta.sqlite3" file neither in my NWN user data folder in Documents, nor in the install folder.

    I own the Steam version, using the latest stable branch. I've already verified the integrity of the installation and everything was ok.

    My game is installed in my HDD but my user data is stored in my SSD. I don't think that's the problem because I've never had this kind of issue with any other game before.

    Thanks for your time.
    dunahan
  • ChassChass Member Posts: 31
    I dont mean to possibly necro this thread. But i'm running into a issue with the windows build of nwsync. Getting out of memory errors when i clearly have 16 GB of memory available This is just CEP Haks. Is there a way around this?
  • JapualtahJapualtah Member Posts: 165
    2 Gigabytes for 34 resrefs, holy Batman!
    I think I recall reading in the NWSYnc documentation that NWSync struggles with large files, you should probably reconsider your files size, just saying.

    @niv Any updates on the project mate? You had so much in the pipe, I wonder what you're up to atm ;-)
  • KeppKepp Member Posts: 44
    I'm getting this error when I do a test sync:

    NWsync: ?

    Cannot download, we only support http:// for now. This is likely a server-side repository issue. Please contact them for help.

    Current transfer:
    Receiving: ?

    Link to my original post for more details: https://forums.beamdog.com/discussion/74429/cep-and-nwsync-from-other-servers#latest
  • KeppKepp Member Posts: 44
    Someone figured it out for me on the other thread.  :)
    JuliusBorisov
  • NineHellsNineHells Member Posts: 1
    Hi there, i couldn't find a way to allow NwSync to use webspace with https when starting the server. It says https is not supported or disabled in libcurl. Is it possible to fix this? Or is there any know free filehosting webspace that doesn't use https?
  • Grizzled_DwarflordGrizzled_Dwarflord Member Posts: 168
    Does anyone know if this is still the definitive NWSync or is there something newer and more up to date? What is disconcerting is that I had to use google just to find it because I'll be damned if Beamdog has it posted anywhere on this forum.
  • JuliusBorisovJuliusBorisov Member, Administrator, Moderator, Developer Posts: 22,714
    Does anyone know if this is still the definitive NWSync or is there something newer and more up to date? What is disconcerting is that I had to use google just to find it because I'll be damned if Beamdog has it posted anywhere on this forum.

    Yes, this is the latest documentation.
  • JuliusBorisovJuliusBorisov Member, Administrator, Moderator, Developer Posts: 22,714
    NineHells wrote: »
    Hi there, i couldn't find a way to allow NwSync to use webspace with https when starting the server. It says https is not supported or disabled in libcurl. Is it possible to fix this? Or is there any know free filehosting webspace that doesn't use https?

    @NineHells Hello! Yes, it's not supported at this time. The team might think about it in the future. That said, NWSync doesn't depend on it for security and consistency.
    dunahan
  • Grizzled_DwarflordGrizzled_Dwarflord Member Posts: 168
    Thanks, Jules, for the confirmation. Unfortunately I'm still too dense to understand the ins and outs of setting up an https server through caddyserver.com. I'm kind of stuck once I run the caddy.exe. Not sure what I need to do after that. That part is not addressed in the User Manual (and they say as much in the manual), but I don't seem follow what transitional steps need to be taken in order to set up the public https site.
  • sknymicksknymick Member Posts: 25
    @Grizzled_Dwarflord the caddy server is just a stand in (a surrogate if you will) for whatever web service you opt to use (as I understand it). We (World of Amon) didn't use caddy at all. We just uploaded our files to our webserver, where we had other outward facing issues. But the guide, necessarily, begins to get fuzzy after NWSync generates your files (no way BD could help everyone interface with their individual web hosting decisions).

    Are "y'all" planning on hosting your own website on your own machine? Or are you using a professional/dedicated web hosting service? If you're running your own, it's worth digging into caddy more, I suppose. If not, then it's worth getting on a chat with your individual web hosting company for help if necessary (this is what we had to do).

    Sknymick
    dunahan
  • dTddTd Member Posts: 182
    I used Apache no problem ,was supplied with by my Linux distribution. Had to edit one config and make sure the directory was rwx by all. If you're familiar with web server software it shouldn't be too hard.
  • SnottlingSnottling Member Posts: 85
    Forgive my ignorance, but why can't Beamdog put up a server where server owners can upload their files and the players can get them from there? Just like Steam Workshop. Just as an option for those who aren't on Steam. This should be integrated in NWN.
    TorgrimmerBalanorRifleLeroy
  • TorgrimmerTorgrimmer Member Posts: 331
    edited March 2019
    Snottling wrote: »
    Forgive my ignorance, but why can't Beamdog put up a server where server owners can upload their files and the players can get them from there? Just like Steam Workshop. Just as an option for those who aren't on Steam. This should be integrated in NWN.

    Agree, this is what preventing me from playing on the Beamdog client over steam, the hassle of the PU mess.
  • sknymicksknymick Member Posts: 25
    Old NWN players have a bad taste in our mouths about putting our "intellectual property" on a company's server. When Atari/Bioware ceased supporting NWN, we had a very short time to move the "Neverwinter Vault" from a company platform to a private platform. In fact, much of it was lost and can't be recovered. A server owner is much better served having his files managed locally than having them stored by a corporation (regardless of their marvelous intentions).

    As it stands, a single server malfunctioning will never prevent a whole community from playing. It's inconvenient for the owner on the face, but I see decentralization as an asset.
    dTdSherincallGM_ODA
  • AspirinsmurfAspirinsmurf Member Posts: 9
    edited May 2019
    What exactly can be served to players with NWNSynch at this time? Can it mess with overrides, provide GUI updates and selectively replace game resources such as dialog.tlk? Or is it basically limited to what could be done with a hakpak?

    sknymick wrote: »
    Old NWN players have a bad taste in our mouths about putting our "intellectual property" on a company's server. When Atari/Bioware ceased supporting NWN, we had a very short time to move the "Neverwinter Vault" from a company platform to a private platform. In fact, much of it was lost and can't be recovered. A server owner is much better served having his files managed locally than having them stored by a corporation (regardless of their marvelous intentions).

    As it stands, a single server malfunctioning will never prevent a whole community from playing. It's inconvenient for the owner on the face, but I see decentralization as an asset.

    I don't get this. By your own reasoning, redundancy is good. If Beamdog wants to contribute server space to that effort, that's a net positive. You would still be free to host and mirror your files wherever you want.
  • sknymicksknymick Member Posts: 25
    edited May 2019
    To the best of my knowledge it's only the assets that the module requires to operate. So stuff in your .hak and custom .tlk files. I don't think it's advisable to have people Override their dialog.tlk file as it will change for the entire game, not just a particular module.

    My point wasn't to be contentious, only to explain the community's hesitance writ large to rely on a singular source for data dispersal that's not controlled by members of the community. Most contributors have put their content on a community managed site called Neverwinter Vault. It's where everything from the old bioware page moved to after the shutdown. If it's just a singular host you're looking for, that's what has been used for the past few years. I thought the question was "why doesn't Beamdog host all the NWSync files" though. Perhaps I should have lead with the information about the Vault. Sorry to confuse.
    dTd
  • AspirinsmurfAspirinsmurf Member Posts: 9
    edited May 2019
    Niv obviously has access to the game's source code, so I suppose the possibilities here are greater than what could be done with just haks and custom tlk files. Custom music has been mentioned, if the engine can be reworked. If I've understood the concept behind the tool correctly, this could potentially lead to hassle-free total conversions such as the Star Wars project or Modern D20.

    On the point of file hosts, I agree that vendor lock-in would be a bad thing. But the vault, as a community-driven project, is in some ways even more vulnerable than a company server. For maximum redundancy, I'd upload my stuff to as many places as I feasibly could. I've seen way too much great content fade away from the Internet over the years, including when IGN pulled the plug on the old NWVault. Nothing is forever, as they say.
  • mrtwondermrtwonder Member Posts: 4
    edited May 2019
    6ehoj7tck13s.jpeg
    I am getting a 503 time out error for my server, 'Town of Luce'. *update, it only seems to do it on some computers. It might be an error on the http server side as well.
    Post edited by mrtwonder on
  • TheCapuletTheCapulet Member Posts: 43
    @niv Any chance we can get some sort of compression for nwsync? Finally working on haks and the size is getting out of control.

    https://support.baldursgate.com/issues/40550
  • nivniv Member, Moderator, Developer Posts: 410
    edited May 2019
    @TheCapulet

    nwsync compresses with zlib by default on the server side repo and zstd in clients. both on the server side repo and in-client. Some hak content compresses really well, and some does not. A very average compression ratio would be about half size.

    If you are talking about removing stale clientside data from previous transfers on the same server, then yes - that's already done and just waiting for the next patch to ship.

    If not, please elaborate what kind of inefficiency you are looking at!
    Post edited by niv on
    dunahanJuliusBorisov
  • TheCapuletTheCapulet Member Posts: 43
    In that case, never mind. Lol. I had no idea it was already compressing the data. That's amazing news. Thanks niv.
  • TheCapuletTheCapulet Member Posts: 43
    Any specific reason why you're using zlib instead of LZMA? LZMA is public domain now, and is the difference of over 200mb in archived hak size for me. But I dont know the drawbacks of it when it comes to integrating it.
  • nivniv Member, Moderator, Developer Posts: 410
    edited May 2019
    Apologies, I mis-typed. Guess it was too early. ;)

    The client is actually using zstd exclusively. The web-server side tooling currently generates zlib compressed payloads, mostly because that was simpler at the time, but the game can ingest either variant transparently. lzma was a poor pick because we actually recompress content for maximum device-local efficiency as it arrives from external sources and it has to be fast and cpu-efficient (you may not care on a desktop cpu, but a mobile phone certainly will).

    In addition to compression, nwsync deduplicates content based on the (decompressed) hash, even across remote repositories/downloads.
    JuliusBorisov
  • TheCapuletTheCapulet Member Posts: 43
    So if I'm understanding right, the nwsync web server currently compresses stuff using the old zlib compression but the client can use both zlib and zstd, and I can simply put the files compressed with zstd on the nwsync server and it will still serve them properly?

    And that's understandable with lzma. It's actually what I assumed when I asked. But on the other hand, can't I use up just as much cpu time (if not more) with higher compression rates on zlib and zstd files?

    If it uses zstd, I have no room to complain though. That more than does the job. Should put things within a few percentage points of lzma.
Sign In or Register to comment.