2DA and Palette interaction.

iXtremist

Hello,
Reviving old module - and I encounter some errors in 2DA files.

Lets take placeables.2da as example.

Some items in the base palette have wrong appearance type - i identified that previous builder from some reason replaced the "vanilla" / biovare reserved lines in 2DA with some custom content.

Since i want to have proper clean 2DAs to allow builders to work with ease, I need to recover the 2da to the "vanilla state" by correcting the original lines and moving the custom content under USER RESERVED lines.

My problem is that I dont want to create enormous work for the builders forcing them to manually edit every item in the palette, so I was thinking about scripting it out in python.

Problem is, that I cannot find anywhere how the pallete is referencing the line number in 2DA.

Also in ITP pallete file, i can see only the tree structure of the "folders", but not items itself.
Where are the palette items/placeables/etc.. variables stored?

With this questions answered I would be able to create python script which will correct the pallete, move the custom under correct range and overwrite the referenced number in the palette object - thus the builder would need to just click "update instance" to rebuild the correct models.

If you find some flaw in my thinking process, feel free to bash me to the correct direction. :-)

Thanks,
Ix

Melkior

As I understand it, unless the 2DA is included in a HAK file being loaded on module start, the file will be in the overrides folder. If you move the 2DA out of the overrides folder, it will cease to have any effect. If this causes any problems then you can move it back in again.
As for the line numbers, as I understand it the line number in the file is not used at all. The lines are actually numbered from the first line of data onward. Any 2DA references will count down from the first line and ignore any numbers at the start of the lines.
I'm not sure about the rest. Hopefully, someone else with more knowledge will be able to help you.

Proleric

References to placeables.2da are in two places - placeable templates (.utp files) and area instances (.git files).

In both cases, you can find the field and path names by examining a file with gffeditor (which ships with the game).

If you have too many changes to do manually with gffeditor, moneo letoscript is the tool of choice.

https://neverwintervault.org/project/nwn1/script/moneo-v40224

Edit - corrected the reference to .utp files

The .itp palette is a list of categories. The .utp file contains the category under which the template is listed. So you won't find the info you want in the .itp.

Proleric

In .git files, the reference to placeables.2da is held in Placeable List[*].Appearance.

In .utp files, the reference is held in the Appearance field.

iXtremist

Many thanks for suggestions. I build my list of appearance changes in excel (around 1000 changes).
However, iterating trough UTP moneo mysteriously crashes after processing some of them with:

[EXCEPTION] Access violation at address 30428034 in module 'Moneo.exe'. Read of address 00000020

Script just for test crashes:

%mod = 'C:\Users\ondre\Documents\Neverwinter Nights\modules\Allhambrie_UTP_TEST.mod' or die$!;

print "Processing palette files UTP...\n\n";

for (%mod[{'*.utp'}]) 
{
  #  print "File: ", %_, "    :" or die$!; 
    print /Tag, " ", /Appearance, " ", /TemplateResRef, "\n\n" or die$!;
}

I have tried to run it in compat mode, admin mode, added the curly braces as i found Proleric is suggesting in one post and still.. crashes. Any idea what can cause this crash? I am running WIN11.

iXtremist

Ah, latest version of moneo fixed that.

Any idea how to make a dictionary (like python or rexx have)? I have around 1000 entries to be checked against the file and replaced if matched.

iXtremist

Melkior wrote: »

As for the line numbers, as I understand it the line number in the file is not used at all. The lines are actually numbered from the first line of data onward. Any 2DA references will count down from the first line and ignore any numbers at the start of the lines.

Regarding lines - it actually care about the first word of the record (line). I did several tests and adding empty lines (thus incrementing the actual line numbers) have 0 effect. Also i checked in GFF edit the UTP file and its referencing the line number described in the first word on the line (record). Maybe thats specific for placeables.2da .

meaglyn

No. The line number in the file is not used. It may be checked to determine if the line is empty. Empty lines after the header are not valid 2da entries though. Maybe the game skips them silently which would have your accurately numbered file work the same way after you added empty lines. Try adding some populated dummy lines...

Proleric

Regarding line numbers, the official Bioware documentation of the 2da format says

Note that the numbering in the first column is for the convenience of the person reading or editing the 2da file. The game and tools automatically keep track of the index of each row, so if a row is numbered incorrectly, the game and tools will still use the correct number for the row index. Nevertheless, it is a good habit to make sure that rows are numbered correctly to avoid confusion.

So, if row 10 is labelled 11, the value stored in files referencing that row is 10. For practical purposes, it's essential to get the labels right without gaps, otherwise when you gffedit a reference you will be tempted to enter 11 (the number you can see) rather than 10 which is implicit in the row order but invisible.

Proleric

iXtremist wrote: »

Any idea how to make a dictionary (like python or rexx have)? I have around 1000 entries to be checked against the file and replaced if matched.

I don't think moneo supports any kind of dictionary or lookup table.

What I do is make the table in Excel, then generate the Letoscript in the final column. In this case that will be a long tedious list of 1000 commands like "if x = 987 then x = 1001" but obviously it's easy to paste those lines into a master script that handles the loop through the files. Clumsy, but performance is better than you might think.

iXtremist

Proleric wrote: »

Regarding line numbers, the official Bioware documentation of the 2da format says

Note that the numbering in the first column is for the convenience of the person reading or editing the 2da file. The game and tools automatically keep track of the index of each row, so if a row is numbered incorrectly, the game and tools will still use the correct number for the row index. Nevertheless, it is a good habit to make sure that rows are numbered correctly to avoid confusion.

So, if row 10 is labelled 11, the value stored in files referencing that row is 10. For practical purposes, it's essential to get the labels right without gaps, otherwise when you gffedit a reference you will be tempted to enter 11 (the number you can see) rather than 10 which is implicit in the row order but invisible.

Thats rather interesting, because when i did update and tests, the dWord Appearance in the UTP GFF matched always exactly the first word on line, not the line number itself.

Anyway as we spoke in the PM - my problem is that moneo is from some reason skipping numerous of UTP files packed in the module. I did even testing with no replacement, just list of all UTP files with the same, buggy result.

%mod = 'C:\Users\ondre\Documents\Neverwinter Nights\modules\Allhambrie_UTP_TEST.mod' or die$!;

print "iXtremist - moneo - processing palette files UTP...\n\n";

for ('*.utp' in %mod)
{
$Ref		= /TemplateResRef;
$tag 	 	= /Tag;

  Print "\n[$Ref]\t{$tag}"; 

}
close %mod;

If I take for example, in the module are files:
zep_rail001.utp
zep_rail002.utp
zep_rail003.utp
zep_rail004.utp

But in the output of the script, only hit for rail00 is this:

[zep_pumpwater] {ZEP_PUMPWATER}
[zep_rack001] {ZEP_RACK001}
[zep_rail002] {ZEP_RAIL002}
[zep_rckprjectles] {ZEP_RCKPRJECTLES}
[zep_redawning] {ZEP_REDAWNING}

iXtremist

Ok i found out interesting interaction.
I edited one of the "invisible for moneo" UTP files via GFF editor in the module folder. Open the toolset back and save the module. And now moneo can see this exact file.

So probably something is wrong with the file format for moneo (not for toolset, toolset have no problem with the palette).

Anybody have some idea for script / tool which would open all gff and resave them in batch?

Just as sanity check i used some other "invisible to moneo" blueprints in the module (place the placeable in test area) and save the module. They are still invisble to moneo. So there have to be something with the GFF format or data areas in these files. (module is very old, reviving PW from 2005).

Proleric

I just wonder whether the "access violation" fix might also process the missing files?

As mentioned in a PM, this is

When looping through files, mysterious access violation crashes can sometimes be avoided by using the curly braces syntax, e.g.

for (%mod[{'*.utp'}]) 

not

for (%mod['*.utp']) 

iXtremist

Proleric wrote: »

I just wonder whether the "access violation" fix might also process the missing files?

As mentioned in a PM, this is

When looping through files, mysterious access violation crashes can sometimes be avoided by using the curly braces syntax, e.g.

for (%mod[{'*.utp'}]) 

not

for (%mod['*.utp']) 

That was what I have tried in the early stages. With 0 luck. It does not fix access violations in my case, neither the file skipping.

Proleric

Sorry, I'd forgotten that.

I reached out on the Vault as someone there may know more, though letoscript knowledge is rare these days.

I could look at your script via PM - preferably one that simply cycles through the .utp files to report their names, which has the skipping and crashing issues.

Proleric

A few ideas on the Vault link above.

Niv's nwnlib tool is here - I know nothing about it, but niv is still active.

iXtremist

So, i found the fix.
Niv tools. NWN_GFF.
I unpack the broken UTP to JSON and pack it back to GFF. Viola, MONEO no longer ignore it and older versions stopped throwing access violations.

Proleric

Good news.

Can you post a link to that tool?

I can't seem to find it for some reason.

iXtremist

https://github.com/niv/neverwinter.nim/releases

To see how to use it, browse source codes, there are comments for every tool.

Proleric

Thanks. I recorded the solution here.

iXtremist

in case anybody on windows need to execute niv tools in mass against files in folder (in this case unpack UTP to json and repack back to original file).. i remembered good old days of MSDOS, just create BAT file from this (bear in mind to have NIV tools in homepath to be able to execute them from any folder) and execute the BAT on command prompt from the folder:

@echo off
setlocal enabledelayedexpansion

echo Iterate through all files with .utp extension
for %%F in (*.utp) do (
echo Processing file: %%F
nwn_gff -i "%%F" -o "%%F.json"
nwn_gff -i "%%F.json" -o "%%F"
)

echo All files processed.

Melkior

iXtremist wrote: »

Ok i found out interesting interaction.
I edited one of the "invisible for moneo" UTP files via GFF editor in the module folder. Open the toolset back and save the module. And now moneo can see this exact file.

So probably something is wrong with the file format for moneo (not for toolset, toolset have no problem with the palette).

Anybody have some idea for script / tool which would open all gff and resave them in batch?

Just as sanity check i used some other "invisible to moneo" blueprints in the module (place the placeable in test area) and save the module. They are still invisble to moneo. So there have to be something with the GFF format or data areas in these files. (module is very old, reviving PW from 2005).

I'm wondering if the text file was made in a different format? I've found that Linux-based editors tend to save text files with just a line-feed at the end of each line whereas Windows editors always save text files with both a carriage return and a line feed at the end of each line of text. Perhaps Moneo requires the CR-LF format and won't recognise a LF-only as being a valid end-of-line?

Or the file may have been saved using something other than ASCII.

A long time ago, I found a simple fix for the first problem was to load the file into Notepad, which does recognise LF as EOL, and then re-save it which causes Notepad to save it with the CR-LF EOL format.