NWNX

Liareth

We expect the initial release to be some time this weekend. The first release will be .so and .nss only. More on available functionality to come.

I've set up a Discord channels for discussions related to NWNX. Feel free to pop by if you want to know how you can contribute or how the changes effect you or if you have any other questions.

Discord: https://discord.gg/hxTt8Fr
Code: https://github.com/nwnxee/unified

Sherincall

Quick summary/FAQ of stuff discussed on the channel today:

• Initial release is just .nss and .so files.
  
• Code will be open sourced some time in the future
  
• Debugging symbols as such will not be available; however, the new NWNX API exposes all NWN functions and structures so there is very little to reverse engineer. Browsing code in IDA will be harder though.
  
• NWNX is Linux only for the time being, but may be ported to other platforms eventually - code is written such that allows this, but it is extra work that is for now better spent elsewhere. Once ported, all plugins will be available on all platforms.
  
• Server deployment via docker is currently being evaluated
  

List of plugins to be available at release:

• NWNX_Administration.so
  
• NWNX_BehaviourTree.so
  
• NWNX_Chat.so
  
• NWNX_Core.so
  
• NWNX_Creature.so
  
• NWNX_Data.so
  
• NWNX_Events.so
  
• NWNX_Metrics_InfluxDB.so
  
• NWNX_MoveSpeed.so
  
• NWNX_Names.so
  
• NWNX_Object.so
  
• NWNX_Optimizations.so
  
• NWNX_Player.so
  
• NWNX_Profiler.so
  
• NWNX_Repair.so
  
• NWNX_Ruby.so
  
• NWNX_SQL.so
  
• NWNX_ThreadWatchdog.so
  
• NWNX_Tracking.so
  

In the channel Liareth also posted a few code snippets that show how easy it is now to write plugins. This is just a random excerpt for the curious:
https://gist.github.com/mtijanic/3b36edd02df330b613fdfa15580ab9f3

metagamer

Hi everybody,

I have one question about NWNX and the new amazing NWN Enhanced.
Even if it seems that server side there will be no problems, what about client side?

Actually many featuress are unlocked by using NWNCX. Mainly NWNCX allows to visualize some new graphical aspects (examples: visual effects on the weapons), unlock the camera and, one of the most important, load the custom content at the creation of the character (it means that if you introduce new skills, new domains or new feat they can be choosen at the creation).

I am afraid that recompiling the client NWNCX will stop to work. Did you consider also this aspect?

Thank you so much guys for your work and let me know if I can contribute somehow!

Liareth

That's a great question.

We don't support NWNCX at the moment. There are a few reasons why we've made this decision:

- Security on the server side. On 1.69 we have seem existing servers crash (accidentally!) because of other server's alterations to the netcode through NWNCX plugins. We have hardened the server significantly since them, but exposing the netcode to modifications is still a risky proposition and we don't want to encourage it.

- Security on the client side. It would be trivial for any developers to create an infected plugin and distribute it. Even if, as a plugin consumer, you trusted the developer completely, there is still no infrastructure in place for signing plugins or ensuring their authenticity. Each server would have to do this on their own, and being practical here - very few would.

- (regarding both of the above:) It is certainly true that exploiters will find exploits with or without symbols or an API, but their job is made much more difficult if an API and/or symbols are not provided.

- Workload on the NWNX team. At launch we're shipping with only one target - Linux. We're doing this because we don't have the resources available to supports Windows right away. If we were to support NWNCX, it would increase the number of supported platforms to 4. We simply cannot accommodate this at the moment.

That isn't to say it isn't possible for you to do on your own - but we won't ship an API for it and the client symbols won't be released for NWN:EE.

I know this news might be disappointing to you, but I hope you can understand. I will say, however, that NWNX development is not a dictatorship. The project belongs to those who came before and those who will follow in the future as much as it does those who maintain it now. This post simply reflects what we will make available at launch, and it's possible that the state of affairs might change in the future.

I strongly encourage you to file feature requests so we can introduce the functionality you rely on into the main game and make it available to all developers, rather than those who use a specific NWNCX plugin. Can you imagine what the modding scene might look like in six months, if we roll all of the advanced custom functionality available on all servers, into the base game for everyone to use? Coming from a background working on a hak-less server, Arelith, the prospect really excites me, and I hope it excites you, too.

metagamer

Thank you so much Liareth for your answer.

I am glad to say that these changements looks amazing and they did not disappoint to me. Everything new is well accepted.
I am Admin in one of the biggest Italian server and I am using NWNX for many many things.
However, exactly how you said, I had to trust the developer in the past but I realized that many features were uncompleted and I had to finish them by myself.

In these days I will show you what I have done and how I used NWNX to develop systems that improve the RPG experience in multiplayer, and especially what I've tried to do to further improve.

_Cyan_

You have to change the functions signatures completely or try to find a method to disallow hook on syscall by the client, so, would be nice to have at least a minimum of support for some basic functionality.

Btw you are doing a great work guys.

NelaK

Sherincall said:

Code will be open sourced some time in the future

Just to clarify - is this just the NWNX code that we're talking about or is this the full NWN source code that will be open sourced? If its the later... That is beyond huge and probably the single biggest feature and expectation that I could have from NWNEE.

NobAkimoto

Pretty sure they mean the modified NWNX source - GPL requires its release as they distribute modifications to the original.

Sherincall

@NelaK yes, only NWNX sources will be released. I can't begin to fathom what kind of legal monstrosities open sourcing NWN would entail, even if it did make business sense for beamdog. Sorry to shoot down your hopes.

@NobAkimoto Actually, NWNX for EE was rewritten from scratch and doesn't have any original NWNX code in it, so it is "free" from the GPL. It will still be open sourced, but as far as I know, the actual license still hasn't been decided on.

Liareth

@NelaK The new NWNX source code. Not the base game!

@NobAkimoto GPL has nothing to do with the new NWNX - it shares no code whatsoever with the old NWNX. We haven't settled on a license yet for the new core.

Edit: Damn, beaten by a second!

Valgav

@Liareth any chances for some documentation for nwnx and new api? Old nwn doesn't give much help for beginners.

Also I assume that if we get all structures and methods there's no problem to move some logic outside nwn as we can call them from something completly external.

One more thing, does nwnx will still work on strings? After all this years getting back to nwnscript without objects, collections and everything that's normal for current languages sounds like a nightmare by it's own. I really don't want to get back to writing super complexed toString() methods in nwn and parsers for those strings in plugins.

_Cyan_

So basically, is not nwnx anymore but something else made by somebody around nwnx community for a new game and not nwn (the old nwn will be there so can be explained like this).

Something already happened before with Wyvern crown of cormyr and the old DLA team.

Liareth

@Valgav The code is much easier to read and follow (at least to my eyes ...) than the old NWNX. Many things are abstracted now, due to the nature of the new core, but you are provided with all the tools you need to do anything you need to do, so it should be easy enough to follow.

Regarding data to/from NWNX - this is handled with strings internally still, but it is stack-based rather than all in one local variable, and each string has type information embedded inside of it, so pushing or popping the wrong types will result in an exception (and an error thrown to stdout). It's a little slower this way but much safer and more user friendly, and in my experience the speed cost is irrelevant compared to other more important factors.

It might make more sense for me to show you the full implementation of a plugin, which is a port of nwnx_chat:

https://gist.github.com/ScholarMidnight/3fd7d008fff3676d32153a3abd3fab3b

_Cyan_

Now the source is fully c++ 11 stl ? Nice one.

Sherincall

@_Cyan_ Actually, it requires C++17. Which might be a pain point for some, as it requires gcc-6 or clang-3.8 to compile.

_Cyan_

@Sherincall Yeah but the code now is much more cool, no more mix of "Everything", and level of abstraction could be insane.

Will be support for multithreading too?

Liareth

@Cyan Yes, see here for a code example:

https://gist.github.com/ScholarMidnight/e2d2cdc77da67f0407b11a83de20123d

Though I will say that the Task service is the most poorly written service, and I want to rewrite it.

Sherincall

@_Cyan_ You can use std::thread if you want, but currently it's up to you to implement synchronization.

The SQL plugin may eventually support offloading the DB operations to another thread, and we may down the road add a nicer API to start a worker thread and signal a NWScript event from it

Looks like this already exists, see above. I hadn't noticed it.

_Cyan_

How you fetch from async thread ? A sig/slots system has been implemented or something like with boost signals 2 is needed?

Liareth

It's not that elaborate. It's just a way to fire one-off tasks onto an async thread, or run tasks on the main thread from an async thread. If you need return values or signals you'll have to handle that yourself.

Valgav

@Liareth It's really looking better and I'm saying this as a person who haven't written any line in c++ for like 6 years.

Are there any plans for wrappers for higher level languages? I see there's nwnx_ruby, but are there any chances for port or new implementation of nwnx_jvm or nwnx_lua? With new api and exposed methods and structures this could be game changer.

Balanor

Liareth said:

....
- Workload on the NWNX team. At launch we're shipping with only one target - Linux. We're doing this because we don't have the resources available to supports Windows right away.

Kind of a bummer here. While I know the hardcore servers often run on Linux, there are a number (including mine) that run on Windows and need NWNX to function as intended. Do you have any timeline as to when you are planning to work on and release NWNX for Windows server?

Liareth

@Balanor Unfortunately I don't have an ETA on the Windows version. I will say that we will release the Windows server API at the same time as the Linux server API, but it will be up to other contributors to make it work.

Liareth

The current state on NWNX is:

- I have updated all of the plugins to work with the latest head start build.

- Unfortunately, I cannot distribute this NWNX build, as the binaries contain symbol information which I cannot strip. That means that the binary release and the source release will need to be linked. Getting this done is my top priority, but the exact information we can release is something that needs to be decided by the higher-ups. I can't give an exact estimate on this, but I can say for certain that it won't be before Tuesday the 28th.

Balanor

(This posted showed up days later and was basically a duplicate of my post below)

Balanor

(It looks like my original reply got eaten somehow)

Thanks for the reply, Liareth. To narrow down my primary NWNX focus a bit: Would it be more worthwhile as a fan to suggest/encourage NWScript feature changes to the toolset that mimic NWNX behavior? In other words, from a purely business standpoint; is it less expensive or time consuming for you to port over what exists in NWNX to NWNX EE, or is it just as expensive or time consuming to add new NWScript functions to the toolset?

I am specifically thinking of the new SetTag function, which was great to see in EE. Previously that was only possible through NWNX, but now it's a NWScript function in the toolset. So as we're looking forward, pretty much everything the nwnx_funcs library is really useful (GetFirstArea, GetNextArea, SetAbilityScore, SetRace, etc, etc, etc). If they are moved to the toolset rather than ported to the EE version of NWNX, that would more quickly open up use for non-Linux servers and (maybe even better) would make them available to single player module creators who don't use NWNX at all.

BUT, the problem here is that, from a business standpoint, I'm guessing it is cheaper for Beamdog to port existing NWNX functions, rather than add them into the toolset. I'm hoping I'm wrong though, or I/we may be able to convince the company of the value in setting up previously NWNX-only scripting functionality in the EE toolset.

Liareth

@Balanor It's actually much easier, cleaner, and safer for me to port things to the base game. Some things are unsuitable for porting, however -things that modify internal data structures, for example. The thing is - I was only granted code access after I ported most of the plugins. In the future I foresee myself maintaining the NWNX core / API and gradually porting functionality into the base game, while the rest of the community handles extending the functionality covered by NWNX.

Balanor

It's very good to hear that it is easier to port things to the base game than to the EE version of NWNX. I have to admit I am definitely an end-user of NWNX, so I don't know much about the source code for NWNX, or what would be considered any functionality that might modify the internal data structures. I myself primarily use NWNX for the additional scripting functionality it provides, which is the biggest obstacle I foresee to getting my PW migrated to NWN EE. But from at least one fan's perspective (mine!), I'd be more than happy to update my scripts to use EE toolset functions rather than NWNX functions if it means I can move my PW to the EE version of NWN sooner rather than waiting for Windows NWNX plugins.

Might you have an idea if you would be porting NWNX functionality into the base game prior to the EE's official release (hopefully!), or would all that be done sometime after it becomes generally available?

niv

The biggest advantage of having them in the base game is that singleplayer modules can make use of them too.

We're adding things all the time - this is what this beta is for (among other things).

Glorwinger

Any Updates? ** Trying not to be impatient ** Looking forward to testing and building for NWN:EE

Liareth

@Glorwinger Still waiting for authorisation to release the API. I did however release the code to the core and plugins, so developers can have an early peek and start planning their migration efforts.

https://nwnx.io/storage/pre-releases/8149-prerelease.zip