@Sherincall Yup. Even said as much about the hooking everything up to make it work (it's B plus some quality of life code, basically). I was more concerned with composability, flexability and elegance when I was thinking up the approach.
Still, it's a bit dissapointing about the performance overhead of this. I mean, I knew there would be performance overhead (there's ALWAYS overhead with any kind of indirection like callbacks), but that still seems a bit much. A few small functions getting called about once per second (6 second round/6 attacks) per creature fighting (in the attack case) really doesn't seem like it should create *that* much performance overhead.
I don't really know much about NWNX, but could that itself be the cause of the performance problem? If it's a seperate process that uses RPC to communicate with NWN that would make for horrendous performance. Every time NWNX touches NWN it would have to completely trash all the CPU cache, regrab everything it needs from RAM (REALLY slow), do a bunch of other stuff to enforce process security, do its work then hand control back to NWN (which then has to go through all the steps again). While this is happening the CPU has "stalled" and can't do any work.
And on the subject of NWNScript, is it really that bad for performance? I'd been mentally thinking of it as being a nieve bytecode interpreter; they're on the order of 2-10 times slower than native code on an instruction-for-instruction basis (depends on the instruction). The big downside of them, performance-wise, is that they're usually cache-unfriendly - where native code would have a list of data entries the naive interpreter has a list of pointers to data. The pointers screw with prediction on the CPU (modern CPUs guess a lot to try and avoid stalls, if your memory access isn't predictable you get stalls).
NWNX is not a separate process. It injects itself into NWN, and modifies the NWN executable code to run additional instructions. There is no overhead to it, other than what extra code it adds, which is native. So effectively no overhead.
NWScript on its own is quite acceptable performance-wise. Mostly because you simply can't do anything complex with it. All the builtin functions are implemented in the native code, nwscript bytecode is just the glue. The VM escape / function call is also fairly fast, maybe 100ns overhead per call. However, setting up the VM to run the script and teardown, is on the order of 100us. A creature in combat could easily be generating up to 5 events per second, which is 500us per creature. At 100 creatures, that's 50ms pure overhead, even if you do nothing.
I don't have any concrete data about these numbers, but the orders of magnitude should be right. If anything, the estimates are pretty conservative. I wouldn't be surprised if stuff takes 2x or even 5x longer when you have a large server.
NWNX is not a separate process. It injects itself into NWN, and modifies the NWN executable code to run additional instructions. There is no overhead to it, other than what extra code it adds, which is native. So effectively no overhead.
NWScript on its own is quite acceptable performance-wise. Mostly because you simply can't do anything complex with it. All the builtin functions are implemented in the native code, nwscript bytecode is just the glue. The VM escape / function call is also fairly fast, maybe 100ns overhead per call. However, setting up the VM to run the script and teardown, is on the order of 100us. A creature in combat could easily be generating up to 5 events per second, which is 500us per creature. At 100 creatures, that's 50ms pure overhead, even if you do nothing.
I don't have any concrete data about these numbers, but the orders of magnitude should be right. If anything, the estimates are pretty conservative. I wouldn't be surprised if stuff takes 2x or even 5x longer when you have a large server.
which is why it would be so important for NWNX to be merged in the main script.
which is why it would be so important for NWNX to be merged in the main script.
I don't understand what you are saying here. The problem with NWNX is that it is server-only, so it isn't usable for singleplayer modules. A secondary problem is that it is hard to set up and to learn; but that is a NWNX problem, unrelated to beamdog. Should be handled by the NWNX devs.
NWNX is no slower or faster than core NWN game engine. It is an extension to it. Whatever BD does to the core game, the community can do the exact same thing with NWNX. We can, and have, unhardcoded many of the mechanics various ways, including many discussed on these forums. When it is done through nwscript callbacks, there is a noticeable performance hit. When it is done with native code (i.e. re-hardcoding to another behavior), there is no hit, but the ease of use is lost. Merging NWNX will not increase performance.
On the other hand, if you are saying that NWNX should just be part of the core game, I must disagree. Many of the functionalities it implements - those that are useful in singleplayer - could and should be ported. But having NWNX allows you to customize your server in ways that is simply not possible with nwscript, even if they go the A) route and reimplement and open source the entire game engine.
Anyway, since this is the place to list wishes regarding customization, I'll throw mine in the mix: BD should expose the client('s server) API like they do for the server API, so NWNX can be run on the client / in single player. This is problematic, as it is easy to embed viruses and whatnot into NWNX plugins that people would download, and there's no easy way to check this. But requiring all the code to be open source and properly signed would go a long way. This way, I could make a e.g. D&D 5e 'hak' that you'd download, and it would apply to all the modules you play. Or I could make it so all the mechanics are exposed to nwscript, so SP modules can script their own, but then they have to care about perf issues.
@Sherincall: your post in reply to mine implies that every time an object invokes a script it must instantiate a fresh nwscript virtual machine. That seems... not good. I would have thought that it would just be "set up a stack in the VM, add a stack frame to the stack for the invoked script, go". That's something that's measured in CPU instuctions, not microseconds.
For reference, and I could be completely wrong, this is a high-level pseudocode model of how I currently think NWN works:
InitialiseNWN();
while (true){ //1 iteration = 1 frame
doUserInput();
doGameLogicTick();
doPhysicsTick();
doScriptingTick();
doNetworkingTick();
//other stuff I haven't thought of
drawGraphics();
drawOverlayUI();
}
I am not a game programmer. This opinion is based on conversations I've had with a games programmer I used to hang around with. Apparently game frames usually do everything in phases like this to make the best use of branch prediction, memory prefetching and CPU caches (all features that have been in x86 since at least the original Pentium was released IIRC).
According to him it's all about making memory accesses as sequential and predictable as possible so the CPU can guess which chunk of memory you're going to access next. In doing so it avoids stalling while data is grabbed from RAM - 1 stall is on the order of hundreds of CPU cycles where the CPU doesn't do anything.
You're not far off with the main loop pattern. Except in NWN scripts can be executed either in the scripting tick or the game logic tick. There is a scripting tick that goes through all the lists that are in need of an update and runs the proper script. But when the logic (or network) code decides it needs to run a script, it will either do it immediately, or put it in that list, depending on... probably what the dev had for breakfast that day.
As for the overhead of running scripts, here's a quick rundown of what happens when RunScript is called - it is called for everything that needs a tick in doScriptingTick(), as well as the one-off scripts elsewhere:
Almost 10 instances of allocating/copying/freeing a 16 byte string.
Searching the list of already loaded resources for the compiled script If not found, loading it from disk. This is slow, but also relatively rare, so ignore this case
Sanity check the .ncs file
Allocate the VM stack, set up all the data
Actually interpret the instructions one by one
There's definitely several thousand instructions worth of staging here, not counting the new/delete calls and the list iteration. There's also a ton of room for improvement however. Quite possibly enough to make the perf issue I mentioned disappear altogether.
By the way @Plok , if you're interested in how NWN engine works exactly, feel free to send me a PM (or better yet catch me on discord) and I can give you some heavy reading material.
First change would be to adjust Combat Expertise. +5 AC is very unbalancing. On a MP server it results in DMs and builders having to balance all encounters with a dedicated expertise tank in mind, which leads to encounters being too lethal for any party that doesn't have one. Making it +3/-3 and removing Improved CE would be better.
First change would be to adjust Combat Expertise. +5 AC is very unbalancing. On a MP server it results in DMs and builders having to balance all encounters with a dedicated expertise tank in mind, which leads to encounters being too lethal for any party that doesn't have one. Making it +3/-3 and removing Improved CE would be better.
My view is that nothing should be removed. It should be unhardcoded so that modders can more easily change it on a case by case basis. I'd be opposed to Combat Expertise being nerfed in the base game.
I'd like to see Combat Expertise and Power Attack changed so that players can choose how much of a penalty they take. Such as +1/-1, +2/-2, +3/-3, +4/-4, and +5/-5. Ditto for the improved versions. Script hooks could be added so modders can limit the maximum bonus.
I think it would be acceptable for Combat Expertise and Power Attack to get disabled while casting a spell though.
I think there's merit in arguing the performance issues with something like a callback for combat but considering the script based alternative as running a 200-250 ms heartbeat script to intercept combat before a hit occurs I think that surely an engine based event would have better performance. You probably don't want to have a lot of individual calls but it's probably possible to have a single event of some kind that can be used to handle situational effects at better performance than purely script mimicry. At least add something that can be used for those that want to use it because the alternative is usually a lot uglier.
@FreshLemonBun When I was coming up with that approach, I wasn't really thinking explicitly about de-hardcoding things, I just posted it here because it seemed like the approach could also handle that.
The whole thing I was going for (and still am) was composability - people not always stepping on each others toes as they mod the game. No having to specially modify haks to work with other haks, no having diverging extensions of haks... just install a hak, install another hak and (excepting extreme circumstances) it all just work. As an added benefit, haks could be smaller, more modular and be easier to reuse. Imagine being able to cherry pick which bits of the the PRC compedium you want/don't want - I want Psionics and Epic Spellcasting but not Tome of Battle. That would be stupid easy if scripting was composable - split the different systems into seperate haks and don't install the ones you don't want.
The whole point of the approach was that there are LOTS of callback calls. A hak that adds a prestige class could be registering a dozen callbacks - most would only fire occasionally but things like OnHit effects could fire up to 15 times a second per object they're registered to (IIRC 15 is the hard upper limit on number of attacks the engine supports). Any callback that works with an AOE could potentially be fired dozens of times in quick succession.
I honestly think it should be go big or go home. A half-hearted approach just causes more problems. Taking your suggestion of bundling everything up into one or two functions - you lose the composability but also make it more likely people step on each others toes - there's now a few big functions that everyone has to edit.
Sadly, in spite of the elegance of the approach, I think it needs reworking. Beamdog haven't really expressed much interest in making the game more plug and play from a mod standpoint (besides adding Steam Workshop support) - we can't really expect them to spend a huge gob of time working on making an approach like this performant when it would only really benefit new mods.
@Dark_Ansem I'm thinking it's not a go-er mainly because Beamdog have announced they're releasing the EE so early. I know Beamdog have been very good custodians of the Infinity Engine games - adding new features and updates years after release - but I also think that a big change like my suggestion really needs to be out sooner rather than later. Putting it in later is either going to annoy people or stop people from working on projects while they wait for it to be put in.
If an approach like my suggestion is added a year after launch when the buzz has died down and everyone's already done a load of cool new mods it's just going to seem like a ton of extra work for little benefit. While composability is a very, very nice property to have in any system it's sooo much more work to modify a system to be composable than just make it so in the first place.
After the hypes died down people won't update their projects to use an approach like this. They're not programming geeks like me who will rewrite/refactor things just to make it cleaner, more elegant and more ameniable to change and extension. Working code is good enough. Even professional programmers struggle with this; we use euphemistic phrases like "Technical Debt" (we wrote a prototype and then shipped it) and "Legacy Code" (this thing is doing dozens of things it was never designed to do, really badly) when we're on this subject.
In my experience you only really get to truly appreciate the merits when you've had to try and cram new features into years old projects that have been made by many hands - often many hands at once and not all of them competent.
On top of all this it doesn't play nice with how NWN actually works. When you're putting food on the table you've gotta work with what's there, not what you want to be there. We need a more realistic, less idealistic approach to dealing with this.
TLDR; I don't think it's gonna work because people will (rightly) think "if it ain't broke, don't fix it". It's a lot of work for something that ain't gonna take off.
@Dark_Ansem Haven't really given it much thought; as my last post probably suggests, I'm a bit disheartened with this due to the announcement date.
Fundamentally it will need some manner of indirection. Indirection is programmer jargon for "replace this bit of code with some code that calls other code" - a middle-man basically. You usually want to make the indirection code accept some value to change the code it calls so you can customise program behaviour without resorting to a bunch of if/elseif spaghetti. There's tons of ways of handling this: pointers, objects, closures, lambdas, a string for a function name... of these nwnscript currently only sort-of supports one; a string containing a function name (or .nss filename in this case).
To simplify the above; you could imagine indirection as replacing things with placeholders and giving scripters the ability to fill those placeholders in (and having the standard nwn behaviour happen when the placeholder isn't filled in).
The approach I suggested in previous posts (and grudgingly abandonded) - event callbacks - is all just indirection, but turned up to 11. No matter what the approach is it is going to be bumping into the same performance bottlenecks of my previous approach. The trick will be to design it in such a way as to avoid/mitigate them as much as possible. I can't come up with something that fits that bill without being intimately familiar with how NWN's engine works.
Welp. Looks like I killed this discussion with the whole pragmatism thing. Sorry about that.
I think that perhaps getting into the nitty technical details of HOW to do this might not be the best approach; ultimately we're not Beamdog/Bioware developers with access to the source code so any approach we suggest is just speculation. Perhaps it would be better to list what we WANT from this; provide Beamdog with specifications rather than approaches.
Personally, the things I want from the end approach are:
Composability (as mentioned many times before) - the ability to stack changes to how the game works without touching other peoples' scripts.
Elegance - the result should be as simple and general as possible. Ideally should have a nice, easily memorable interface to work with rather than the standard "modify global state and then spend untold dozens of lines of code fixing up the unwanted consequences" of nwnscript.
Flexible enough to do things like add rider effects such as knockdown a target, give extra damage if a target is a given alignment or buff your character on use.
Performant - not much point in doing this if you can't actually use it. (My suggested approach falls down here)
Obviously, we can't always have what we want and having the ability to modify hard coded feats without these properties is favourable to not having the ability at all.
Never played got into elder scrolls so I can't comment. I tried Morrowind when I was a dumb teenager and never really got it and Oblivion bored me whitless when I was a college student. I've never played Skyrim. Judging by the number of youtube videos I've seen of Skyrim where the dragons have been replaced by Thomas the Tank Engine, I'm guessing Skyrim's modding is composable.
This is going to be a long post, so I've broken it up with some handy dandy html <hr> tags
Composability is basically "I can have more than one mod changing something at the same time". Take this example; imagine there's two haks. One adds a prestige class that grants Sneak Attack and the other creates a feat that gives you an extra 1d6 sneak attack damage.
The first hak, cautious of the idea that people can enter the Prestige class from Rogue, Blackguard or Assassin, decides against just giving you Sneak Attack 1d6 in cls_feat.2da (since this wouldn't work with Rogue base class). Instead, it modifies the levelup script to calculate the next feat in the sequence and gives you that instead. The second hak, also aware of this problem, also modifies the levelup script to do the same thing.
If you try and use these haks together, what happens? One or the other works, but not both. That sucks. It *really* sucks when your haks are more complicated than toy examples and you have files that are edited by one hak but not the other. You get all the scripts in one hak working but only some of the scripts in the other, which is worse than none running because things break in unfathomable ways.
There's no logical reason why these two haks should conflict - the only relation between these two haks is that they both add Sneak Attack damage. The difference in the logic of these two haks is literally a single if statement. The fact that they break when used together is purely an implementation detail in how modding in NWN works. Everyone has to step on each others' toes to do even the simplest things. Even ignoring scripting, just having two files that add a feat will step on each others toes because there can only be one feats.2da.
A recent game you may have played that has composable modding is Civilization 6. In Civ 6, I can just install as many mods as I want from the steam workshop, enable them in the "Additional Content" section of the main menu and everything just composes together and works. It's not perfect, any time your mod uses Lua scripting you can run into the same problems NWN has, but it works for 99% of what you'd want to do.
In Civ 6, almost all changes to the core game logic are stored in an Sqlite database (there's also Lua scripting but it mainly seems to be used for scenarios and map generation). Modding is mostly just writing SQL statements to modify the game (or SQL encoded in XML... don't ask). The core approach is to allow modders to declare rules based on conditions in this database; eg.:
if Leader "Ghandi" builds a Unit "Thermonuclear missile", give a 25% bonus to Yield "Production".
If I am Civilization "Aztec" and I kill a UnitType "Military" with a Unit "Eagle Warrior", spawn a Unit "Builder" where it died, 25% of the time.
If a city has a Building "Granary", add 1 to Yield "Food" to all worked tiles containing Feature "Wheat" or Feature "Rice"
If a Unit "Impi" attacks a Unit of UnitType "Gunpowder", add 5 to it's combat strength
If a District "Hansa" is next to a District "Commercial Hub", increase it's Yield "Commerce" by 2.
I do not think Civ 6's approach is even remotely a good idea for NWN. It works well in the domain of turn based 4x strategy games but probably won't in the domain of Real Time with Pause RPGs. However, there's lessons that can be learned from Civ 6. Ignoring the fact that it's mostly just a relational database, what core properties make it work? Off the top of my head:
Modders don't directly edit the games procedures - they instead declare their changes. It almost gets this for free because SQL is a declaritive language. This isn't possible in nwnscript because it is a prodecural language. However, you can work around this with indirection (such as Event callbacks).
Everyone gets to play in their own sandbox. You don't have to modify anything just to get your code called. NWN, in contrast, requires you to modify either a script, a module or a 2da in order to define an entry point into your scripts.
There are no magic numbers (such as 2da row ids). Every piece of data is given a string identifier. This makes conflicts exceptionally rare - especially when most people prefix their ids with $my_mod_name (namespacing in programming parlance).
Mods themselves are also composable. You have to go out of your way to make them not composable. You can make mods that modify mods that modify mods (in order to understand recursion you must first understand recursion )
There's definitely more great properties of Civ 6's approach that could be learned from, but I've been writing this for a while now and my faculties have left me.
I think that sounds pretty similar to the Elder Scrolls. Which while definitely a good idea, I'd say it is largely incompatible with NWN because there is no "sandbox".
In the stream this week this topic was brought up and went a little strange as Trent said it was a philosophical discussion. So he described that his desire is that people can come to a consensus of how the features work and then the same rules apply to every server module. In my mind there is no room for discussion. People already work around the hardcodedness and alter almost every aspect of the game rules, and the only question is do you make it easier for them or not?
I think it should be easier and modules should be able to provide a cleaner and smoother experience to players by having properly supported features without any kind of hackery or vestigial leftovers of the old rules.
Furthermore almost all servers run their own rules. Many ppl have strange ideas about what makes a good or fair game, a lot of changes on servers are made in a way that suggests that the server builders probably don't have a good grasp on the design concepts of D&D 3.5. They will often favor one particular play style and award bonuses in ways that by the book are unevenly distributed. This is something that a player of a persistent world has to live with because the owner pays the hosting bill. However players always have the choice of playing on another server that does cater to their tastes. There in lies the strength of NWN, not in providing the same wonky broken rules everywhere, but giving everyone their own opportunity to "fix" it as they see fit.
So I don't really see it as a philosophical discussion. It's more of a question like does the company spend time and development pain trying to unhardcode things which makes everyone's life easier, or do modders continue to spend collectively even more time and pain making work arounds.
So I have been modding for quite a lot of years. Working with Goudea/Enki on Starwars content for NWN and being administrator of the French Starwars module for many years. So here is my feedback: - Hardcoded spellbooks are the main hindrance right now. Not being able to create custom spellcasting classes or systems is quite terrible. Being able to choose the number of spells per day, the number of circles (we should be able to go over 9th circle), etc should be granted ASAP - Permanent Properties should be handled by script without having to use a "Skin" item. Just like the RDD immunity to fire, there should be a permanent property parameter in scripting, different from supernatural. - Speed limit. Yes the nwn speed limit prevents from many fun things. - Hardcoded VFX should be usable from the 2DA. For example, only the RDD dragon breath VFX can be found in 2da. Not electric dragon breath or acid or anything like that. - "ADDING" spells or feats to Player Characters by scripting while keeping them "legals". Relying on items to simulate gameplay acquired powers is far from being right. Cheats allow adding feats or spells to PCs. Scripting should allow the same thing without needing a level up. This would allow simulating alternative rulesets or just a bit of ingame customization.
I will have other ideas but I think you got my point.
Definitely unhardcoding feats, metamagic and spellbooks would make customised rulesets a lot easier. It would also help in making endgame wizards less terrible.
So I have been modding for quite a lot of years. Working with Goudea/Enki on Starwars content for NWN and being administrator of the French Starwars module for many years. So here is my feedback: - Hardcoded spellbooks are the main hindrance right now. Not being able to create custom spellcasting classes or systems is quite terrible. Being able to choose the number of spells per day, the number of circles (we should be able to go over 9th circle), etc should be granted ASAP - Permanent Properties should be handled by script without having to use a "Skin" item. Just like the RDD immunity to fire, there should be a permanent property parameter in scripting, different from supernatural. - Speed limit. Yes the nwn speed limit prevents from many fun things. - Hardcoded VFX should be usable from the 2DA. For example, only the RDD dragon breath VFX can be found in 2da. Not electric dragon breath or acid or anything like that. - "ADDING" spells or feats to Player Characters by scripting while keeping them "legals". Relying on items to simulate gameplay acquired powers is far from being right. Cheats allow adding feats or spells to PCs. Scripting should allow the same thing without needing a level up. This would allow simulating alternative rulesets or just a bit of ingame customization.
I will have other ideas but I think you got my point.
Thanks for reading !
Could you please expand on hardcoded VFX and Permanent Properties?
So I have been modding for quite a lot of years. Working with Goudea/Enki on Starwars content for NWN and being administrator of the French Starwars module for many years. So here is my feedback: - Hardcoded spellbooks are the main hindrance right now. Not being able to create custom spellcasting classes or systems is quite terrible. Being able to choose the number of spells per day, the number of circles (we should be able to go over 9th circle), etc should be granted ASAP
There's a seperate set of requests for these so I consider custom class spellbooks out of scope of this topic. The main fundamental problem in doing this that can't be used/worked around is the GUI. The PRC bends over backwards trying to do this with dynamic conversations to select spells and fill out your spell slots. It's really awful, especially when you have metamagic feats.
On the subject of >9th level spells I've also made a request about this in order to make epic-level casting relevant.
- Permanent Properties should be handled by script without having to use a "Skin" item. Just like the RDD immunity to fire, there should be a permanent property parameter in scripting, different from supernatural.
Is this really a problem? I mean, sure it's non-obvious and verbose to script - which sucks - but does it actually have any practical drawbacks? To deal with the arcane-ness of it someone could just make a script library with some sensible function names.
- Speed limit. Yes the nwn speed limit prevents from many fun things.
Oh god yes! I'd not even considered this. I've just had a flashback to trying to play NWN 2 SOZ without a Monk in my party... watching my party slowly shamble across the world map was like watching paint dry.
- "ADDING" spells or feats to Player Characters by scripting while keeping them "legals". Relying on items to simulate gameplay acquired powers is far from being right. Cheats allow adding feats or spells to PCs. Scripting should allow the same thing without needing a level up. This would allow simulating alternative rulesets or just a bit of ingame customization.
Hmm... I'm not sure how you'd actually go about making the character legality check actually work with this. It strikes me that the thing to do is to turn if off then script your own checker.
I definitely agree about being able to give people feats via script though; it's not without precedence in PnP. There's a magical location that has a challenge that when completed grants you a bonus feat (can't remember the name - o-something hole, I think). It'd also enable the epithets from NWN 2 which would be neat.
As for spells, that really should come part and parcel with opening up spellbooks. People are going to want to do interesting spell learning mechanics like Spirit Shaman (Spontaneous Druid Casting but you pick your spells each day) and Archivist (learns Cleric spells at level up but can scribe any Divine spell into their prayerbook).
The PRC bends over backwards trying to do this with dynamic conversations to select spells and fill out your spell slots. It's really awful, especially when you have metamagic feats.
I actually like the metamagics for PRC spellbooks of spontaneous casters as a toggle
@Behindflayer It's not bad for spontaneous casters. Prepared casters are another matter entirely. Try playing an Wizard/Archivist/Mystic Theurge at high levels - you'll very quickly see what I mean.
@Plok I know what you mean. Just wanted to mention that the toggle metamagics for spontaneous casters are pretty awesome (and I remember a time when that wasn't actually implemented). Proper spellbooks would be great but if that function could be kept intact, it would be neat.
Comments
Still, it's a bit dissapointing about the performance overhead of this. I mean, I knew there would be performance overhead (there's ALWAYS overhead with any kind of indirection like callbacks), but that still seems a bit much. A few small functions getting called about once per second (6 second round/6 attacks) per creature fighting (in the attack case) really doesn't seem like it should create *that* much performance overhead.
I don't really know much about NWNX, but could that itself be the cause of the performance problem? If it's a seperate process that uses RPC to communicate with NWN that would make for horrendous performance. Every time NWNX touches NWN it would have to completely trash all the CPU cache, regrab everything it needs from RAM (REALLY slow), do a bunch of other stuff to enforce process security, do its work then hand control back to NWN (which then has to go through all the steps again). While this is happening the CPU has "stalled" and can't do any work.
And on the subject of NWNScript, is it really that bad for performance? I'd been mentally thinking of it as being a nieve bytecode interpreter; they're on the order of 2-10 times slower than native code on an instruction-for-instruction basis (depends on the instruction). The big downside of them, performance-wise, is that they're usually cache-unfriendly - where native code would have a list of data entries the naive interpreter has a list of pointers to data. The pointers screw with prediction on the CPU (modern CPUs guess a lot to try and avoid stalls, if your memory access isn't predictable you get stalls).
NWScript on its own is quite acceptable performance-wise. Mostly because you simply can't do anything complex with it. All the builtin functions are implemented in the native code, nwscript bytecode is just the glue. The VM escape / function call is also fairly fast, maybe 100ns overhead per call.
However, setting up the VM to run the script and teardown, is on the order of 100us. A creature in combat could easily be generating up to 5 events per second, which is 500us per creature. At 100 creatures, that's 50ms pure overhead, even if you do nothing.
I don't have any concrete data about these numbers, but the orders of magnitude should be right. If anything, the estimates are pretty conservative. I wouldn't be surprised if stuff takes 2x or even 5x longer when you have a large server.
NWNX is no slower or faster than core NWN game engine. It is an extension to it. Whatever BD does to the core game, the community can do the exact same thing with NWNX. We can, and have, unhardcoded many of the mechanics various ways, including many discussed on these forums. When it is done through nwscript callbacks, there is a noticeable performance hit. When it is done with native code (i.e. re-hardcoding to another behavior), there is no hit, but the ease of use is lost. Merging NWNX will not increase performance.
On the other hand, if you are saying that NWNX should just be part of the core game, I must disagree. Many of the functionalities it implements - those that are useful in singleplayer - could and should be ported. But having NWNX allows you to customize your server in ways that is simply not possible with nwscript, even if they go the A) route and reimplement and open source the entire game engine.
Anyway, since this is the place to list wishes regarding customization, I'll throw mine in the mix:
BD should expose the client('s server) API like they do for the server API, so NWNX can be run on the client / in single player.
This is problematic, as it is easy to embed viruses and whatnot into NWNX plugins that people would download, and there's no easy way to check this. But requiring all the code to be open source and properly signed would go a long way.
This way, I could make a e.g. D&D 5e 'hak' that you'd download, and it would apply to all the modules you play. Or I could make it so all the mechanics are exposed to nwscript, so SP modules can script their own, but then they have to care about perf issues.
For reference, and I could be completely wrong, this is a high-level pseudocode model of how I currently think NWN works:
InitialiseNWN(); while (true){ //1 iteration = 1 frame doUserInput(); doGameLogicTick(); doPhysicsTick(); doScriptingTick(); doNetworkingTick(); //other stuff I haven't thought of drawGraphics(); drawOverlayUI(); }I am not a game programmer. This opinion is based on conversations I've had with a games programmer I used to hang around with. Apparently game frames usually do everything in phases like this to make the best use of branch prediction, memory prefetching and CPU caches (all features that have been in x86 since at least the original Pentium was released IIRC).
According to him it's all about making memory accesses as sequential and predictable as possible so the CPU can guess which chunk of memory you're going to access next. In doing so it avoids stalling while data is grabbed from RAM - 1 stall is on the order of hundreds of CPU cycles where the CPU doesn't do anything.
As for the overhead of running scripts, here's a quick rundown of what happens when RunScript is called - it is called for everything that needs a tick in doScriptingTick(), as well as the one-off scripts elsewhere:
- Almost 10 instances of allocating/copying/freeing a 16 byte string.
- Searching the list of already loaded resources for the compiled script
- Sanity check the .ncs file
- Allocate the VM stack, set up all the data
- Actually interpret the instructions one by one
There's definitely several thousand instructions worth of staging here, not counting the new/delete calls and the list iteration. There's also a ton of room for improvement however. Quite possibly enough to make the perf issue I mentioned disappear altogether.If not found, loading it from disk. This is slow, but also relatively rare, so ignore this case
By the way @Plok , if you're interested in how NWN engine works exactly, feel free to send me a PM (or better yet catch me on discord) and I can give you some heavy reading material.
First change would be to adjust Combat Expertise. +5 AC is very unbalancing. On a MP server it results in DMs and builders having to balance all encounters with a dedicated expertise tank in mind, which leads to encounters being too lethal for any party that doesn't have one. Making it +3/-3 and removing Improved CE would be better.
I'd like to see Combat Expertise and Power Attack changed so that players can choose how much of a penalty they take. Such as +1/-1, +2/-2, +3/-3, +4/-4, and +5/-5. Ditto for the improved versions. Script hooks could be added so modders can limit the maximum bonus.
I think it would be acceptable for Combat Expertise and Power Attack to get disabled while casting a spell though.
The whole thing I was going for (and still am) was composability - people not always stepping on each others toes as they mod the game. No having to specially modify haks to work with other haks, no having diverging extensions of haks... just install a hak, install another hak and (excepting extreme circumstances) it all just work. As an added benefit, haks could be smaller, more modular and be easier to reuse. Imagine being able to cherry pick which bits of the the PRC compedium you want/don't want - I want Psionics and Epic Spellcasting but not Tome of Battle. That would be stupid easy if scripting was composable - split the different systems into seperate haks and don't install the ones you don't want.
The whole point of the approach was that there are LOTS of callback calls. A hak that adds a prestige class could be registering a dozen callbacks - most would only fire occasionally but things like OnHit effects could fire up to 15 times a second per object they're registered to (IIRC 15 is the hard upper limit on number of attacks the engine supports). Any callback that works with an AOE could potentially be fired dozens of times in quick succession.
I honestly think it should be go big or go home. A half-hearted approach just causes more problems. Taking your suggestion of bundling everything up into one or two functions - you lose the composability but also make it more likely people step on each others toes - there's now a few big functions that everyone has to edit.
Sadly, in spite of the elegance of the approach, I think it needs reworking. Beamdog haven't really expressed much interest in making the game more plug and play from a mod standpoint (besides adding Steam Workshop support) - we can't really expect them to spend a huge gob of time working on making an approach like this performant when it would only really benefit new mods.
If an approach like my suggestion is added a year after launch when the buzz has died down and everyone's already done a load of cool new mods it's just going to seem like a ton of extra work for little benefit. While composability is a very, very nice property to have in any system it's sooo much more work to modify a system to be composable than just make it so in the first place.
After the hypes died down people won't update their projects to use an approach like this. They're not programming geeks like me who will rewrite/refactor things just to make it cleaner, more elegant and more ameniable to change and extension. Working code is good enough. Even professional programmers struggle with this; we use euphemistic phrases like "Technical Debt" (we wrote a prototype and then shipped it) and "Legacy Code" (this thing is doing dozens of things it was never designed to do, really badly) when we're on this subject.
In my experience you only really get to truly appreciate the merits when you've had to try and cram new features into years old projects that have been made by many hands - often many hands at once and not all of them competent.
On top of all this it doesn't play nice with how NWN actually works. When you're putting food on the table you've gotta work with what's there, not what you want to be there. We need a more realistic, less idealistic approach to dealing with this.
TLDR; I don't think it's gonna work because people will (rightly) think "if it ain't broke, don't fix it". It's a lot of work for something that ain't gonna take off.
Fundamentally it will need some manner of indirection. Indirection is programmer jargon for "replace this bit of code with some code that calls other code" - a middle-man basically. You usually want to make the indirection code accept some value to change the code it calls so you can customise program behaviour without resorting to a bunch of if/elseif spaghetti. There's tons of ways of handling this: pointers, objects, closures, lambdas, a string for a function name... of these nwnscript currently only sort-of supports one; a string containing a function name (or .nss filename in this case).
To simplify the above; you could imagine indirection as replacing things with placeholders and giving scripters the ability to fill those placeholders in (and having the standard nwn behaviour happen when the placeholder isn't filled in).
The approach I suggested in previous posts (and grudgingly abandonded) - event callbacks - is all just indirection, but turned up to 11. No matter what the approach is it is going to be bumping into the same performance bottlenecks of my previous approach. The trick will be to design it in such a way as to avoid/mitigate them as much as possible. I can't come up with something that fits that bill without being intimately familiar with how NWN's engine works.
I think that perhaps getting into the nitty technical details of HOW to do this might not be the best approach; ultimately we're not Beamdog/Bioware developers with access to the source code so any approach we suggest is just speculation. Perhaps it would be better to list what we WANT from this; provide Beamdog with specifications rather than approaches.
Personally, the things I want from the end approach are:
- Composability (as mentioned many times before) - the ability to stack changes to how the game works without touching other peoples' scripts.
- Elegance - the result should be as simple and general as possible. Ideally should have a nice, easily memorable interface to work with rather than the standard "modify global state and then spend untold dozens of lines of code fixing up the unwanted consequences" of nwnscript.
- Flexible enough to do things like add rider effects such as knockdown a target, give extra damage if a target is a given alignment or buff your character on use.
- Performant - not much point in doing this if you can't actually use it. (My suggested approach falls down here)
Obviously, we can't always have what we want and having the ability to modify hard coded feats without these properties is favourable to not having the ability at all.Is this basically what happens in the Elder Scrolls games?
This is going to be a long post, so I've broken it up with some handy dandy html <hr> tags
Composability is basically "I can have more than one mod changing something at the same time". Take this example; imagine there's two haks. One adds a prestige class that grants Sneak Attack and the other creates a feat that gives you an extra 1d6 sneak attack damage.
The first hak, cautious of the idea that people can enter the Prestige class from Rogue, Blackguard or Assassin, decides against just giving you Sneak Attack 1d6 in cls_feat.2da (since this wouldn't work with Rogue base class). Instead, it modifies the levelup script to calculate the next feat in the sequence and gives you that instead. The second hak, also aware of this problem, also modifies the levelup script to do the same thing.
If you try and use these haks together, what happens? One or the other works, but not both. That sucks. It *really* sucks when your haks are more complicated than toy examples and you have files that are edited by one hak but not the other. You get all the scripts in one hak working but only some of the scripts in the other, which is worse than none running because things break in unfathomable ways.
There's no logical reason why these two haks should conflict - the only relation between these two haks is that they both add Sneak Attack damage. The difference in the logic of these two haks is literally a single if statement. The fact that they break when used together is purely an implementation detail in how modding in NWN works. Everyone has to step on each others' toes to do even the simplest things. Even ignoring scripting, just having two files that add a feat will step on each others toes because there can only be one feats.2da.
A recent game you may have played that has composable modding is Civilization 6. In Civ 6, I can just install as many mods as I want from the steam workshop, enable them in the "Additional Content" section of the main menu and everything just composes together and works. It's not perfect, any time your mod uses Lua scripting you can run into the same problems NWN has, but it works for 99% of what you'd want to do.
In Civ 6, almost all changes to the core game logic are stored in an Sqlite database (there's also Lua scripting but it mainly seems to be used for scenarios and map generation). Modding is mostly just writing SQL statements to modify the game (or SQL encoded in XML... don't ask). The core approach is to allow modders to declare rules based on conditions in this database; eg.:
I do not think Civ 6's approach is even remotely a good idea for NWN. It works well in the domain of turn based 4x strategy games but probably won't in the domain of Real Time with Pause RPGs. However, there's lessons that can be learned from Civ 6. Ignoring the fact that it's mostly just a relational database, what core properties make it work? Off the top of my head:
- Modders don't directly edit the games procedures - they instead declare their changes. It almost gets this for free because SQL is a declaritive language. This isn't possible in nwnscript because it is a prodecural language. However, you can work around this with indirection (such as Event callbacks).
- Everyone gets to play in their own sandbox. You don't have to modify anything just to get your code called. NWN, in contrast, requires you to modify either a script, a module or a 2da in order to define an entry point into your scripts.
- There are no magic numbers (such as 2da row ids). Every piece of data is given a string identifier. This makes conflicts exceptionally rare - especially when most people prefix their ids with $my_mod_name (namespacing in programming parlance).
- Mods themselves are also composable. You have to go out of your way to make them not composable. You can make mods that modify mods that modify mods (in order to understand recursion you must first understand recursion
)
There's definitely more great properties of Civ 6's approach that could be learned from, but I've been writing this for a while now and my faculties have left me.I think it should be easier and modules should be able to provide a cleaner and smoother experience to players by having properly supported features without any kind of hackery or vestigial leftovers of the old rules.
Furthermore almost all servers run their own rules. Many ppl have strange ideas about what makes a good or fair game, a lot of changes on servers are made in a way that suggests that the server builders probably don't have a good grasp on the design concepts of D&D 3.5. They will often favor one particular play style and award bonuses in ways that by the book are unevenly distributed. This is something that a player of a persistent world has to live with because the owner pays the hosting bill. However players always have the choice of playing on another server that does cater to their tastes. There in lies the strength of NWN, not in providing the same wonky broken rules everywhere, but giving everyone their own opportunity to "fix" it as they see fit.
So I don't really see it as a philosophical discussion. It's more of a question like does the company spend time and development pain trying to unhardcode things which makes everyone's life easier, or do modders continue to spend collectively even more time and pain making work arounds.
- Hardcoded spellbooks are the main hindrance right now. Not being able to create custom spellcasting classes or systems is quite terrible. Being able to choose the number of spells per day, the number of circles (we should be able to go over 9th circle), etc should be granted ASAP
- Permanent Properties should be handled by script without having to use a "Skin" item. Just like the RDD immunity to fire, there should be a permanent property parameter in scripting, different from supernatural.
- Speed limit. Yes the nwn speed limit prevents from many fun things.
- Hardcoded VFX should be usable from the 2DA. For example, only the RDD dragon breath VFX can be found in 2da. Not electric dragon breath or acid or anything like that.
- "ADDING" spells or feats to Player Characters by scripting while keeping them "legals". Relying on items to simulate gameplay acquired powers is far from being right. Cheats allow adding feats or spells to PCs. Scripting should allow the same thing without needing a level up. This would allow simulating alternative rulesets or just a bit of ingame customization.
I will have other ideas but I think you got my point.
Thanks for reading !
On the subject of >9th level spells I've also made a request about this in order to make epic-level casting relevant. Is this really a problem? I mean, sure it's non-obvious and verbose to script - which sucks - but does it actually have any practical drawbacks? To deal with the arcane-ness of it someone could just make a script library with some sensible function names. Oh god yes! I'd not even considered this. I've just had a flashback to trying to play NWN 2 SOZ without a Monk in my party... watching my party slowly shamble across the world map was like watching paint dry. Hmm... I'm not sure how you'd actually go about making the character legality check actually work with this. It strikes me that the thing to do is to turn if off then script your own checker.
I definitely agree about being able to give people feats via script though; it's not without precedence in PnP. There's a magical location that has a challenge that when completed grants you a bonus feat (can't remember the name - o-something hole, I think). It'd also enable the epithets from NWN 2 which would be neat.
As for spells, that really should come part and parcel with opening up spellbooks. People are going to want to do interesting spell learning mechanics like Spirit Shaman (Spontaneous Druid Casting but you pick your spells each day) and Archivist (learns Cleric spells at level up but can scribe any Divine spell into their prayerbook).
From here:
"Icebox List
Cards in the Icebox list are features we've investigated and want to tackle but they are not the highest priority yet."
Anyway, if you have questions about this feature request, feel free to ask them during a livestream(s).
Closing the thread for now. Thanks for the feedback, everyone! It is very valuable for the team.