What is the state of the internal SetCampaign style commands in NWN:EE? Are they still super slow?

dwinblood

I love the fact that NWNX exists and for persistent worlds it is great. However, there is a great potential benefit to using databases for stand alone modules as well. The biggest problem is the built in functions are incredibly slow. In fact, to try to squeeze as much speed out of it as possible I know the internal DB supports writing objects instead of just variables, so to speed things up we would write all of our variables onto an object and just periodically serialize that. This was not without issue. It can introduce a lurch into the module. It does work, but that lurch can be an issue. What would be nice is if NWN:EE actually had Sql Lite or something integrated that would allow us to make modules for the AVERAGE person and they could just use them without knowing anything about setting up NWNX or something like that. Now if NWNX were somehow OFFICIALLY integrated into the NWN:EE game that too could mitigate this need. The biggest factor here is adding functionality without expecting the average user to jump through hoops. Some of them already have issues with installing HAKs and other things. Their loss, but still a loss.

highv_priest

Since no one answered. The campaign database is slow the first time it's used(this is when it is physically creating the database on the harddrive). After the first time it's plenty fast, but can only reliably be used with integers as it will bloat up everytime a variable is deleted, because it doesn't actually delete.

thirdmouse

Have you used NBDE at all?

On the subject of NWNX being an official Beamdog NWN:EE thing, I don't think that will ever happen, and that it's a good thing. Beamdog is for adding things to base game that make sense for their scope, while NWNX is for extending the capabilities of said base. It's meant to be cracked open, tinkered with, and added to by the community. That they're supporting it so strongly is fantastic, but it'll always be for those things out of scope for NWN.

That doesn't mean none of the features of its plug-ins can be integrated if they decide it does make sense for core, though. Some of them already have (eg area instancing) And since we have NWNX people with source access, the odds of base NWN getting more cool stuff that would otherwise have been NWNX-only seem pretty good. What is or isn't considered appropriate for "core" in the value equation is more where the divide probably comes, hehe.

sippelmc

dwinblood said:

I love the fact that NWNX exists and for persistent worlds it is great. However, there is a great potential benefit to using databases for stand alone modules as well. The biggest problem is the built in functions are incredibly slow. In fact, to try to squeeze as much speed out of it as possible I know the internal DB supports writing objects instead of just variables, so to speed things up we would write all of our variables onto an object and just periodically serialize that. This was not without issue. It can introduce a lurch into the module. It does work, but that lurch can be an issue. What would be nice is if NWN:EE actually had Sql Lite or something integrated that would allow us to make modules for the AVERAGE person and they could just use them without knowing anything about setting up NWNX or something like that. Now if NWNX were somehow OFFICIALLY integrated into the NWN:EE game that too could mitigate this need. The biggest factor here is adding functionality without expecting the average user to jump through hoops. Some of them already have issues with installing HAKs and other things. Their loss, but still a loss.

The native internal DB is not bad actually if you understand its limitations and how it works. What you did, as I understand it, which is to make one object with all your variables attached to it, and to StoreCampaignObject, is probably a worst case scenario in practice.

You can get very good performance if you realize two things:
- you can have as many "CampaignDatabases" as you like (they are essentially just files), meaning dont put everything in file as you can have many fast, small files as you like
- and you realize that old data is never deleted on updates it is just marked as invalid and a new record is created with new value, which bloats out file size quickly if you do frequent updates.

For the first, I've found that if you break out your variables as much as possible to different files it is fine. Size is often under 16k per file, which is no negative performance on an SSD.

For the second, the key is that when you are ready for an update, you first delete the old database file and write the new values to a new file. That way your file is always "packed" by virtue that it is always fresh.

see below, which creates a refreshed, fully packed, single and new file for each PC henchman. If I were to simply keep the same db file and update the variable with new object to store, size and perf would go bad

string sHenchmanTagId = GetTag(oHenchman);
string sDBFile = GetPlayerCharTag(oPC) + "_" + GetTag(oHenchman);

//Destroy the old file first to prevent the char file from exploding in size
DestroyCampaignDatabase(sDBFile);

//Now overwrite with the old file
int iSaveResult;
iSaveResult = StoreCampaignObject(sDBFile, sHenchmanTagId, oHenchman);

I store large store inventories, multiple companions with inventory, etc., and it is fine provided that I delete the db old file first, then re-store data in a net new file.

Sherincall

1.69 also had a utility program that "packs" the database - i.e. removes the deleted entries. You don't have to destroy them from scripts, just regularly run the pack utility.

EE doesn't distribute these utilities (yet?) but the format didn't change so the 1.69 will work for 1.74 as well.

dunahan

Sherincall said:

1.69 also had a utility program that "packs" the database - i.e. removes the deleted entries. You don't have to destroy them from scripts, just regularly run the pack utility.

EE doesn't distribute these utilities (yet?) but the format didn't change so the 1.69 will work for 1.74 as well.

Didn't knew that ... Used the DB like sippelmc described, due lacking experiance of sql. Which program is it, you mentioned?

But it's running fine for me ;-)

highv_priest

dunahan said:

Sherincall said:

1.69 also had a utility program that "packs" the database - i.e. removes the deleted entries. You don't have to destroy them from scripts, just regularly run the pack utility.

EE doesn't distribute these utilities (yet?) but the format didn't change so the 1.69 will work for 1.74 as well.

Didn't knew that ... Used the DB like sippelmc described, due lacking experiance of sql. Which program is it, you mentioned?

But it's running fine for me ;-)

You go to the utils folder in your install. In there they added lots of useful utilities....

One final note though.... The packer does not work when a database gets too large in size so there is in fact a limit window associated with it. I noticed it when using databases with lots of repeat entries such as storing and deleting bank chests. I think the size limit is somewhere around a gig or more though so you have a lot of leeway

Glorwinger

I will be pushing a script update to a project called the 'persistence facade' that has an API that matches all of the SetCampaign, GetCampaign, StoreCampaign, and RetrieveCampaign functions that allows a simple switch between using the built in DB or NWNX. The code doesn't change but the included file does and the desired database is activated. You can find an existing version here https://neverwintervault.org/project/nwn1/script/persistence-facade ... That version is not updated for NWNX EE - I have those changes that I will be packing up and posting.

fot1

Is there any reason to only mark something as deleted on the database apart from performance? I know there are reasons to do that, but I don't see any specific on neverwinter nights.

If the deletion performance is not terrible, maybe we can get a `SetMutableCampaignObject` that deletes the older object entry for good when called?

Sherincall

It's "only" performance. However, actually deleting an entry from a large database the way you'd intuitively think about "delete" is quite slow. All databases avoid doing this in the actual call. They just have a more sophisticated method to pack it instead of an external utility.

For NWN, a secondary thread that occasionally does the packing when resource use is low would be more than enough. But, that's not really different from a cronjob that calls the packer utility, which you can do yourself.

dunahan

highv_priest said:

You go to the utils folder in your install. In there they added lots of useful utilities....

One final note though.... The packer does not work when a database gets too large in size so there is in fact a limit window associated with it. I noticed it when using databases with lots of repeat entries such as storing and deleting bank chests. I think the size limit is somewhere around a gig or more though so you have a lot of leeway

Is this util named DataPacker.exe? Used it today on a Database file with around 28-30 MB... But it seemed that nothing happened. Will give it tomorrow another chance.

Glorwinger

I posted an update to a script library called the 'Persistance Facade'. The original was written as a drop in replacement for all of the Campaign database setters and getters allowing modules to switch back and forth from BioDB to NWNX SQL. I have updated this for NWNX EE. The post includes the persistent chest code I used for testing. https://neverwintervault.org/project/nwn1/script/persistence-facade-ee

jbbullet

Glorwinger said:

I posted an update to a script library called the 'Persistance Facade'. The original was written as a drop in replacement for all of the Campaign database setters and getters allowing modules to switch back and forth from BioDB to NWNX SQL. I have updated this for NWNX EE. The post includes the persistent chest code I used for testing. https://neverwintervault.org/project/nwn1/script/persistence-facade-ee

Hello Glorwinger,
I can read that your Persistent Facade 2 could help creators to easily switch from different types of persistencies.
How could it help me to convert my full NWNX PW into a Bioware DB PW? So I can launch it in NWN:EE

Thank you for your answer,

dunahan

@jbbullet: I took a look at the version for 1.69, which you should use dir that project.

First, import only facade without any extension. If you use APS then you should import the part for it too. The second step would be the change of the functions named in your old scripts that use APS plus add the inclusion of the facade include.

In the main include of facade is the way which it uses the databases. There ist also the way discribed how to use it

jbbullet

@dunahan
Will try this. Thanks a lot for your answer!

dwinblood

thirdmouse said:

Have you used NBDE at all?

NBDE is something I am familiar with. It works exactly as I indicated in my original post. It writes the variables to an object and then it writes that object to the database. It does speed things up but it is still lurchy.

highv_priest said:

it's plenty fast,

Well that is obviously subjective. Apparently it IS NOT or I wouldn't have written this. Depending upon YOUR use case it may be. For my needs it is very slow and it does introduce lurch even when combining it with techniques such as object caching in NBDE.

sippelmc said:

The native internal DB is not bad actually if you understand its limitations and how it works. What you did, as I understand it, which is to make one object with all your variables attached to it, and to StoreCampaignObject, is probably a worst case scenario in practice.

You can get very good performance if you realize two things:
- you can have as many "CampaignDatabases" as you like (they are essentially just files), meaning dont put everything in file as you can have many fast, small files as you like
- and you realize that old data is never deleted on updates it is just marked as invalid and a new record is created with new value, which bloats out file size quickly if you do frequent updates.

Actually I used NBDE which was writing to objects. It also gives a significant performance boost over other methods when testing.

Yet I am doing some pretty complex things. I had a module that was complex enough that the actual SAVE mechanism in the engine didn't actually save all data. So reloading would not always resume where it was at.

I therefore had to make my own system to store and restore things that the normal SAVE could not.

Thanks for the response.

I didn't try the destroy before updating technique before though. So that is worth looking into.

Xilanthus

Glorwinger said:

I posted an update to a script library called the 'Persistance Facade'. The original was written as a drop in replacement for all of the Campaign database setters and getters allowing modules to switch back and forth from BioDB to NWNX SQL. I have updated this for NWNX EE. The post includes the persistent chest code I used for testing. https://neverwintervault.org/project/nwn1/script/persistence-facade-ee

@Glorwinger - Do the persistence-facade-ee scripts require NWNX/APS? If so, where can I get a copy of NWNX/APS for Windows that works with EE?

Glorwinger

The NWNX for EE is only complete for Linux currently (Windows in progress) - the code and links to binaries is here - https://github.com/nwnxee/unified. There are also a discord channel for the NWNX where questions and alternate packages - such as a docker container with all required dependencies installed is available.

https://discord.gg/CukSHZq