The RFC builtin database replacement card discussion

JuliusBorisov

Greatings, NWN:EE enthusiasts!

Please, provide more details and feedback for

https://trello.com/c/urQVWOsc/14-rfc-builtin-database-replacement

in this thread.

Sherincall

I'm curious what use-case is this targeting? Is this for PWs that don't want to use NWNX? Or is it for module series to transfer data between modules?

badstrref

Basic saving-methods found inside the toolset are cool and super easy to use, at the hand of anyone, but partially flawed so some people use the NBDE http://nwn.wikia.com/wiki/Nbde_inc wich adresses the problems without having to setup and learn Mysql

Sql is always the most robust / flexible /reliable anyway.

Cablefish

I'd like that a lot! It would help tremendously towards my freescape PW which will allow the player to build structures and crafting stations around the gameworld and have the server remember them through resets.

Sherincall

Right, no denying SQL is great. But we already have SQL support as part of NWNX. NWNX doesn't work in singleplayer, and is arguably a hassle to set up for quick multiplayer sessions - but both of those don't have much to gain from SQL anyway. The big gains are for the PWs, all of which use NWNX.

So, @Cablefish , if you have a reason for not using NWNX on your PW, I'd love to hear it.

NeverwinterWights

Sherincall said:

Right, no denying SQL is great. But we already have SQL support as part of NWNX. NWNX doesn't work in singleplayer, and is arguably a hassle to set up for quick multiplayer sessions - but both of those don't have much to gain from SQL anyway. The big gains are for the PWs, all of which use NWNX.

So, @Cablefish , if you have a reason for not using NWNX on your PW, I'd love to hear it.

A couple things to point out here.

The nwdb does have some issues that, as of 1.69, I believe still have't been "fixed". If I remeber correctly it doesn't handle the float data type well. Thus the need for Nats scripts to flush the database and what not. Otherwise you will end up with a bloated database that can cause some in-game issues like lag when using "GetCampaign" functions.

Not all PWs use NWNX or at most, many are ONLY using the watchdog aspect of it to reset the PW. The one I used to script on was one of those. Setting up watchdog alone is enough to turn some would be PWs away from seeing the light of day. Now add on too that setting up the database, especially for people without much scripting knowledge, they just don't want to touch it. Integration and ease of use will make people less "afraid" of it.

Adding an updated, reliable database with more flexibilty and more functions may help the scripting and module community as well. And since Beamdog is already integrating aspects of NWNX, why not integrate it's database or similar?

Taro94

Sherincall said:

Right, no denying SQL is great. But we already have SQL support as part of NWNX. NWNX doesn't work in singleplayer, and is arguably a hassle to set up for quick multiplayer sessions - but both of those don't have much to gain from SQL anyway. The big gains are for the PWs, all of which use NWNX.

So, @Cablefish , if you have a reason for not using NWNX on your PW, I'd love to hear it.

While it's true that almost, if not all, PWs are strictly multiplayer, and thus use NWNX, the name "Persistent world" does not indicate it being necessarily multiplayer-only.

I, myself, am developing a sandbox-type module that I want playable singleplayer and on LAN sessions, so I need to provide persistence that can't be based on NWNX. If a PW's definition is in its name, then by all means my module is a PW.

Then there are other uses of databases:
-it's the only way of storing actual game objects and not just their IDs
-non-PW modules can have some creative uses of databases - see Infinite Dungeons and its dungeon sharing feature

Sherincall

@Taro94 But wouldn't your usecase be well served by keeping the locals on the PC's skin (and occasionally flushing that skin to the native database?). Or, just saving the game - loading a savegame restored all variables.

-it's the only way of storing actual game objects and not just their IDs

Right, because it makes a copy. You can also do a CopyObject(), and store that as a local, it'd do the same thing.
Or do you actually need the relational aspect of SQL to perform complex queries/updates?


@NeverwinterWights In that case, wouldn't a better feature request be "Make NWNX easier to use"? The endgame of the docker approach is to be able to just take the "server package", add your module/haks to it, and run a "deploy" script.

Keep in mind that all the features we request take dev time away from other features. So, isn't it better to just make it easier to use ALL of NWNX. One great advantage of NWNX is that it isn't bottlenecked by just a few devs - there's dozens of people willing to add features to it, which would then be usable by all servers.

Taro94

I can't use the PC skin because it can't storę objects between different sessions and some persistent data (ie. Town economy, world events) are global and thus can't be stored on any specific PC's skin.

NeverwinterWights

Sherincall said:

@NeverwinterWights In that case, wouldn't a better feature request be "Make NWNX easier to use"? The endgame of the docker approach is to be able to just take the "server package", add your module/haks to it, and run a "deploy" script.

Keep in mind that all the features we request take dev time away from other features. So, isn't it better to just make it easier to use ALL of NWNX. One great advantage of NWNX is that it isn't bottlenecked by just a few devs - there's dozens of people willing to add features to it, which would then be usable by all servers.

For a game that Beamdog wants to emphasize the multiplayer aspect of, (which I'm pretty sure I remember hearing in one of the streams) and hopefully bring in a lot of new players and a bigger community, what it comes down to is, "Hey everybody here's this awesome game you can buy, build a world for you and your friends to play on, oh but you need to go over to this website and download this thing that isn't really part of our game and do some stuff to make the world persistent for you and your friends." Or "Hey everybody here's this awesome game you can buy, build a world for you and your friends to play on. Just click this official button in the official menu. Or heck you don't even need to click anything..it just works automatically" Which do you think is going to appeal more to new players (or even old players and builders for that matter)?
Me and my friend were just talking about this the other day in fact (two 40 year old dudes drinking coffee at Denny's, reminiscing about the good old NWN days). He really got into NWN2, started his own PW, but handed it off years ago to someone else. Myself and a group of other peeps from a PW decided to start our own NWN1 PW. Lasted for a couple years and then life happened. Anyway, one of the things we both hated having to do was messing around with NWNX to get the game to do what a PW needs to do.

Sherincall

Well, after Taro94 explained his "singleplayer PW" project, I totally see the use case for this feature. I still personally think it's too niche, and not worth spending the resources on (NBDE will work well-enough), but it is perfectly valid.

On the other hand, I don't quite see how "you need to go over to this website and download this thing that isn't really part of our game" is the big barrier to making a traditional PW. Building a PW is a lot of work, measured in hundreds or even thousands of man-hours. There will be dozens of things you're not familiar with that you'll need to do, which will involve searching online, going through other projects on the vault and so on. Is following a 30 minute tutorial on how to set up NWNX really a dealbreaker here?

Granted, the tutorial doesn't exist, but it probably will - and certainly could. What I'm saying is that making such a tutorial, and making it so the process is simple enough to be doable in 30 minutes, is a much better use of the limited resources.

NeverwinterWights

Sherincall said:

Building a PW is a lot of work, measured in hundreds or even thousands of man-hours. There will be dozens of things you're not familiar with that you'll need to do, which will involve searching online, going through other projects on the vault and so on.

I guess I'm just looking at it from a different perspective here. When my daughter wanted me to play with her in her PW (which consisted of a few areas and a LOT of animals at first) guess who had to set up NWNX. Then my son did the same thing when he got older. Set up NWNX for him too. Had to set it up for the PW I worked on and the test PW I worked.
It's not that it's a deal breaker. It's a PITA. Especially when a couple years goes by before you have to do it again and you forget how and have to go look it up and install it again.
So this is just going to be one of those agree to disagree type things. Personally I think of NWN as more of this PW building software that should have always just had NWNX type features built in. And if they can, they should.

voidofopinion

Sherincall said:

Keep in mind that all the features we request take dev time away from other features. So, isn't it better to just make it easier to use ALL of NWNX.

Not true.

All features we request at worst, take eyes away from other requested features of which will still be seen and added to Trello.

Then the features we request get put into Trello... Which uses the community to work through the churn.

Then finally someone in charge will make the call if a feature can and should be implemented or if time is better spent elsewhere.

It's really only if Beamdog decide to move forward with an idea that it begins to take any serious time away from other potential features.

I can petition to have NWN fully rebuilt into a 16bit JRPG. I could get 10,000 likes on Trello and that means absolutely nothing the moment someone in charge takes a quick look and decides it's outside the scope of the EE and resources could better be spent elsewhere.

At most I would waste my time and the time of people who wanted to participate in the discussion. And the 30 seconds it took for someone in charge to skim over the post and go "NOPE!"

If you are concerned with preserving precious development time and making sure eyes are on the right feature requests then don't continuously respond to ideas you don't like on the forum and don't upvote them on Trello.

Because every time you respond to something like this you are bumping down the features you actually do want to see make it into the game and drawing more eyes onto the ones you do not.

Otherwise, it's all good healthy discussion.

On that subject...

-----------------------------------------------

My $0.02:

It would be nice if the bioware database worked better. However, NWNX has been handling the database side of PW's for a decade or more just fine. If it ain't broke... Don't fix it.

That's all.

Best regards,

- Jamie

Sherincall

voidofopinion said:

Sherincall said:

Keep in mind that all the features we request take dev time away from other features. So, isn't it better to just make it easier to use ALL of NWNX.

Not true.

All features we request at worst, take eyes away from other requested features of which will still be seen and added to Trello.

Then the features we request get put into Trello... Which uses the community to work through the churn.

Then finally someone in charge will make the call if a feature can and should be implemented or if time is better spent elsewhere.

It's really only if Beamdog decide to move forward with an idea that it begins to take any serious time away from other potential features.

Right, but a post in this forum means it's gone past the "input" stage, and the "proposed" stage and has already made the shortlist for development as soon as resources free up / current "active" features are done.

https://trello.com/b/K0C0i8wF/neverwinter-nights-enhanced-edition

Phenomen

Sherincall said:

In that case, wouldn't a better feature request be "Make NWNX easier to use"?

NWNX for EE needs some script like ASP (Avlis Persistence System) so you can use SQL DB with just NWScript instead of digging into depths of MySQL or writing complicated wrappers.

voidofopinion

Sherincall said:

voidofopinion said:

Sherincall said:

Keep in mind that all the features we request take dev time away from other features. So, isn't it better to just make it easier to use ALL of NWNX.

Not true.

All features we request at worst, take eyes away from other requested features of which will still be seen and added to Trello.

Then the features we request get put into Trello... Which uses the community to work through the churn.

Then finally someone in charge will make the call if a feature can and should be implemented or if time is better spent elsewhere.

It's really only if Beamdog decide to move forward with an idea that it begins to take any serious time away from other potential features.

Right, but a post in this forum means it's gone past the "input" stage, and the "proposed" stage and has already made the shortlist for development as soon as resources free up / current "active" features are done.

https://trello.com/b/K0C0i8wF/neverwinter-nights-enhanced-edition

Which means its being discussed and not that it is a dead cert.

Which still means by taking on a crusade you are drawing focus away from ideas you want to be implemented rather than pushing ideas you like.

Every time you adamently argue why database functionality shouldnt be extended then more people will respond as to why it should. There would have been all of 3 responses to this idea if you said your piece and left it alone and instead there is an active debate with lots more reasons why. Which is great if you want to see healthy discourse but bad if you have an agenda.

You do you my friend.

However, social media has taught us the best way to squash things we don't like is to not talk about them. Heck, it's the principle that reddit is founded on.

In 2017 we just ignore it and it goes away. Then signal boost the things we do like. It has been proven to be very effective.

So if you actually want to see the database system replaced then the devils advocate path is a great way to do so and I commend you on your dedication. Otherwise you could probably find more success focusing on telling people why the things you want are great as opposed to why the things they want are stupid.

Because anything else leads to the active discussion and the free sharing of ideas, much like you and I are doing now.

Imagine if every post you have made here and every response to you in kind was in support of a feature you cared about while this thread sat at the bottom of the list with all of 3 responses. That would probably be more effective in actualizing the features you want. Right?

Just a thought,

- Jamie

--------------------------

Barry_1066

Sherincall said:

Right, no denying SQL is great. But we already have SQL support as part of NWNX. NWNX doesn't work in singleplayer, and is arguably a hassle to set up for quick multiplayer sessions - but both of those don't have much to gain from SQL anyway. The big gains are for the PWs, all of which use NWNX.

So, @Cablefish , if you have a reason for not using NWNX on your PW, I'd love to hear it.

Nope - " All of which use NWNX" no no no - my world has over 2000 area - so a big world and we use the NWN database very effectivly. We do NOT use NWNX so your statement is in error.

NWNX is Linux only at present - nothing need be said more there.

Our systems are arguably the most advanced in NWN - a dumbed down version released in CEP 2.4 but we have far more than that and NO NWNX and NO SQL.

Barry_1066

I would agree that database update in capability would be great - I have one overriding concern - my existing databases must be backward compataible or a conversion program that is flawless need be available.

My crafting, much monster creation, storage and spawning come for the database as well as items not on the pallet we store in a DM resource area - items by the - don't have an exact count but - by the many hundreads. If the NWN db was remade and that broke my database it would be a major blow and loss of work, some irretrievable.

so yes more capability and yes we use some scripted solutions to db management but it is still all NWN DB.

I use to work with Dbase years ago - first just writing code and then came a wizard where you clicked options and selected data and like magic it all setup and worked. Now there is an idea, as long as one can modify the code and is not forced to use the wizard but can write their own instead.

Malishara

I have worked extensively with the built-in database system in NWN. It works. I have NOT had issues with it, nor have I had any need of more - despite coding some very complex systems which use it. Do I feel that SQL would provide a lot more functionality? Absolutely. Do I want to see SQL included? Sure. Do I want it to replace the database system we currently have? NO. It would require, as Barry has pointed out, a conversion program that is FLAWLESS to avoid destroying a massive amount of work. And given the problems that crop up in this type of undertaking, I do not believe that a flawless conversion program will happen. Just look at the sound issues that have occurred as a result of moving to OpenAL... SQL? Sure. But make it an addition rather than replacing what we already have.

voidofopinion

Barry_1066 said:

Sherincall said:

Right, no denying SQL is great. But we already have SQL support as part of NWNX. NWNX doesn't work in singleplayer, and is arguably a hassle to set up for quick multiplayer sessions - but both of those don't have much to gain from SQL anyway. The big gains are for the PWs, all of which use NWNX.

So, @Cablefish , if you have a reason for not using NWNX on your PW, I'd love to hear it.

Nope - " All of which use NWNX" no no no - my world has over 2000 area - so a big world and we use the NWN database very effectivly. We do NOT use NWNX so your statement is in error.

NWNX is Linux only at present - nothing need be said more there.

Our systems are arguably the most advanced in NWN - a dumbed down version released in CEP 2.4 but we have far more than that and NO NWNX and NO SQL.

I actually want to hear everything possible about this.

Are you using Natural Bioware Database Extension or just raw Bioware database?

What kind of persistent systems are you using? Location/chest/time/bank/journal/quest?

Tell me your secrets!

Barry_1066

We have custom creatures we build in game - set models and size - equip, set level, faction and all - then save this new creature or person as a widget that is persistent and spawns it - but don't stop there - it saves all aspects by the way including inventory - and one can make a group of mixed creatures and save as a widget - that widget can be put in a container that can be accessed from the spawn system and called by variable - the variable can call from a number of ordered containers for the purpose of spawning custom in game made persistent monsters and NPCs.

yes we have banks, saddle bags, storage systems and more - we have a pixie post system that can send money, items, and creatures.

The postal system is persistent and goes player to player and DM to player or player to DM.

We can build with placeables in game and keep it persistent or change out a set for seasonal changes -

We use the Stage Manager System to set up seasonal questing systems for October fest, Yule, Spring Feat, Summer Fest and Harvest - just put the proper widget set in the setup chest in the DM area and it is done.

There is more - much more - and all with the NWN database.

voidofopinion

That sounds exceptional!

So how are you getting around the NWN perceived slowness? Is it just no longer an issue in 2017?

I ask because as amazing as NWNX is, using the built in database has potentially greater long time compatibility should the game go dark again, simply by the benefit of being self contained.

NWNX was the gold standard in 2007 but it is a decade later and I honestly didn't bother to even try NWNDB when I returned.

I hope you don't mind me asking. However, the answer does change the complexion of this discussion.

- Jamie

JuliusBorisov

From the Trello board: "Cards in the Needs More Discussion list are features that we have started to investigate but we need more details or feedback from the community to further action the request."

@voidofopinion and @Sherincall These "Need More Discussion" cards appeared in the Roadmap at once as this is the start of the card system, and they were moved to the Roadmap and not to the Input because of the mix of popularity (as the forum and other places showed) and the devs' opinion.

So no need to worry, and these threads are for constuctive feedback about the topic, not the talk about whether the card(s) should exist.

voidofopinion

Fantastic to have that pinned and underlined @JuliusBorisov.

So these discussions are just that. Discussions. And no one needs to feel threatened by the conversation simply existing.

Thank you.

Shadooow

I think we don't need database replacement, rather than fix some of the BioDB issues such as unability to really delete entries etc. (I am no expert for bioDB issues).

Barry_1066

I posted this elsewhere but it does apply here -- the NWN DB needs these:

GetSystemDate() Returns the current date in the format (mm/dd/yyyy)
GetSystemTime() Returns current time in the format (24:mm:ss)
GetTimestamp() Returns the number of seconds since midnight on January 1, 1970

Their importance cannot be underestimated.

Yes there are some other nice things could be added but these three calls are very important and should not be breakng the bank to add them.

mangamuscle

voidofopinion said:

I ask because as amazing as NWNX is, using the built in database has potentially greater long time compatibility should the game go dark again, simply by the benefit of being self contained.

Hear! Hear!

Also, this is NWN:EE (Enhanced Edition) for a reason, the intention should be enhancing it and I think the right bet is using non obsolete standards that designers in 2018 and forward can easily use.

farhorizon

I'd love to see the 3 time functions above as well, but they don't have to be tied to the database. For speed purposes, ideally they'd come from the OS not a database.

If someone specifically wants the timestamp at the start of a database transaction, likely they want actual transactionally compliant SQL and can use nwnx.

Barry_1066

farhorizon said:

I'd love to see the 3 time functions above as well, but they don't have to be tied to the database. For speed purposes, ideally they'd come from the OS not a database.

If someone specifically wants the timestamp at the start of a database transaction, likely they want actual transactionally compliant SQL and can use nwnx.

The three time calls should be taken from the system - these are useful for the database but not just the database and should not be specifically tied to just the database.

more on this here:
https://forums.beamdog.com/discussion/comment/932243#Comment_932243

flyinghtcher

Sherincall said:

The big gains are for the PWs, all of which use NWNX.

LMAO, how arrogant that sounds. In my experience a lot of the PW I played had zero database use at all, let alone NWNX

ELadner

I can't speak for all PWs, but the one I'm associated with does use NWNX for persistent storage (PostgreSQL).

For the internal database that NWN uses, I'd personally think it'd be good to spend some brainpower and update it to something more modern. Those cruddy dBase files in the servervault directory are a nightmare and bloat horribly over time. SQLite would be a logical upgrade path and a good choice (embedded friendly, multi-platform, decent performance, standards based-ish). .

As far as the NWNX stuff goes - that's a separate discussion. That functionality should exist there for the people that need it. Currently, the unified NWNX only supports MySQL, but others will come in time, I'm sure.