Skip to content

2DA and Palette interaction.

iXtremistiXtremist Member Posts: 13
Hello,
Reviving old module - and I encounter some errors in 2DA files.

Lets take placeables.2da as example.

Some items in the base palette have wrong appearance type - i identified that previous builder from some reason replaced the "vanilla" / biovare reserved lines in 2DA with some custom content.

Since i want to have proper clean 2DAs to allow builders to work with ease, I need to recover the 2da to the "vanilla state" by correcting the original lines and moving the custom content under USER RESERVED lines.

My problem is that I dont want to create enormous work for the builders forcing them to manually edit every item in the palette, so I was thinking about scripting it out in python.

Problem is, that I cannot find anywhere how the pallete is referencing the line number in 2DA.

Also in ITP pallete file, i can see only the tree structure of the "folders", but not items itself.
Where are the palette items/placeables/etc.. variables stored?

With this questions answered I would be able to create python script which will correct the pallete, move the custom under correct range and overwrite the referenced number in the palette object - thus the builder would need to just click "update instance" to rebuild the correct models.

If you find some flaw in my thinking process, feel free to bash me to the correct direction. :-)

Thanks,
Ix

Comments

  • MelkiorMelkior Member Posts: 181
    As I understand it, unless the 2DA is included in a HAK file being loaded on module start, the file will be in the overrides folder. If you move the 2DA out of the overrides folder, it will cease to have any effect. If this causes any problems then you can move it back in again.
    As for the line numbers, as I understand it the line number in the file is not used at all. The lines are actually numbered from the first line of data onward. Any 2DA references will count down from the first line and ignore any numbers at the start of the lines.
    I'm not sure about the rest. Hopefully, someone else with more knowledge will be able to help you.
  • ProlericProleric Member Posts: 1,281
    edited March 29
    References to placeables.2da are in two places - placeable templates (.utp files) and area instances (.git files).

    In both cases, you can find the field and path names by examining a file with gffeditor (which ships with the game).

    If you have too many changes to do manually with gffeditor, moneo letoscript is the tool of choice.

    https://neverwintervault.org/project/nwn1/script/moneo-v40224

    Edit - corrected the reference to .utp files

    The .itp palette is a list of categories. The .utp file contains the category under which the template is listed. So you won't find the info you want in the .itp.
    Post edited by Proleric on
  • ProlericProleric Member Posts: 1,281
    In .git files, the reference to placeables.2da is held in Placeable List[*].Appearance.

    In .utp files, the reference is held in the Appearance field.
  • iXtremistiXtremist Member Posts: 13
    Many thanks for suggestions. I build my list of appearance changes in excel (around 1000 changes).
    However, iterating trough UTP moneo mysteriously crashes after processing some of them with:

    [EXCEPTION] Access violation at address 30428034 in module 'Moneo.exe'. Read of address 00000020

    Script just for test crashes:
    %mod = 'C:\Users\ondre\Documents\Neverwinter Nights\modules\Allhambrie_UTP_TEST.mod' or die$!;
    
    print "Processing palette files UTP...\n\n";
    
    for (%mod[{'*.utp'}]) 
    {
      #  print "File: ", %_, "    :" or die$!; 
        print /Tag, " ", /Appearance, " ", /TemplateResRef, "\n\n" or die$!;
    }
    

    I have tried to run it in compat mode, admin mode, added the curly braces as i found Proleric is suggesting in one post and still.. crashes. Any idea what can cause this crash? I am running WIN11.
  • iXtremistiXtremist Member Posts: 13
    Ah, latest version of moneo fixed that.

    Any idea how to make a dictionary (like python or rexx have)? I have around 1000 entries to be checked against the file and replaced if matched.
  • iXtremistiXtremist Member Posts: 13
    Melkior wrote: »
    As for the line numbers, as I understand it the line number in the file is not used at all. The lines are actually numbered from the first line of data onward. Any 2DA references will count down from the first line and ignore any numbers at the start of the lines.

    Regarding lines - it actually care about the first word of the record (line). I did several tests and adding empty lines (thus incrementing the actual line numbers) have 0 effect. Also i checked in GFF edit the UTP file and its referencing the line number described in the first word on the line (record). Maybe thats specific for placeables.2da .
  • meaglynmeaglyn Member Posts: 149
    edited April 2
    No. The line number in the file is not used. It may be checked to determine if the line is empty. Empty lines after the header are not valid 2da entries though. Maybe the game skips them silently which would have your accurately numbered file work the same way after you added empty lines. Try adding some populated dummy lines...

  • ProlericProleric Member Posts: 1,281
    Regarding line numbers, the official Bioware documentation of the 2da format says
    Note that the numbering in the first column is for the convenience of the person reading or editing the 2da file. The game and tools automatically keep track of the index of each row, so if a row is numbered incorrectly, the game and tools will still use the correct number for the row index. Nevertheless, it is a good habit to make sure that rows are numbered correctly to avoid confusion.
    So, if row 10 is labelled 11, the value stored in files referencing that row is 10. For practical purposes, it's essential to get the labels right without gaps, otherwise when you gffedit a reference you will be tempted to enter 11 (the number you can see) rather than 10 which is implicit in the row order but invisible.
  • ProlericProleric Member Posts: 1,281
    iXtremist wrote: »
    Any idea how to make a dictionary (like python or rexx have)? I have around 1000 entries to be checked against the file and replaced if matched.
    I don't think moneo supports any kind of dictionary or lookup table.

    What I do is make the table in Excel, then generate the Letoscript in the final column. In this case that will be a long tedious list of 1000 commands like "if x = 987 then x = 1001" but obviously it's easy to paste those lines into a master script that handles the loop through the files. Clumsy, but performance is better than you might think.
  • iXtremistiXtremist Member Posts: 13
    Proleric wrote: »
    Regarding line numbers, the official Bioware documentation of the 2da format says
    Note that the numbering in the first column is for the convenience of the person reading or editing the 2da file. The game and tools automatically keep track of the index of each row, so if a row is numbered incorrectly, the game and tools will still use the correct number for the row index. Nevertheless, it is a good habit to make sure that rows are numbered correctly to avoid confusion.
    So, if row 10 is labelled 11, the value stored in files referencing that row is 10. For practical purposes, it's essential to get the labels right without gaps, otherwise when you gffedit a reference you will be tempted to enter 11 (the number you can see) rather than 10 which is implicit in the row order but invisible.

    Thats rather interesting, because when i did update and tests, the dWord Appearance in the UTP GFF matched always exactly the first word on line, not the line number itself.

    Anyway as we spoke in the PM - my problem is that moneo is from some reason skipping numerous of UTP files packed in the module. I did even testing with no replacement, just list of all UTP files with the same, buggy result.
    %mod = 'C:\Users\ondre\Documents\Neverwinter Nights\modules\Allhambrie_UTP_TEST.mod' or die$!;
    
    print "iXtremist - moneo - processing palette files UTP...\n\n";
    
    for ('*.utp' in %mod)
    {
    $Ref		= /TemplateResRef;
    $tag 	 	= /Tag;
    
      Print "\n[$Ref]\t{$tag}"; 
    
    }
    close %mod;
    

    If I take for example, in the module are files:
    zep_rail001.utp
    zep_rail002.utp
    zep_rail003.utp
    zep_rail004.utp

    But in the output of the script, only hit for rail00 is this:

    [zep_pumpwater] {ZEP_PUMPWATER}
    [zep_rack001] {ZEP_RACK001}
    [zep_rail002] {ZEP_RAIL002}
    [zep_rckprjectles] {ZEP_RCKPRJECTLES}
    [zep_redawning] {ZEP_REDAWNING}


  • iXtremistiXtremist Member Posts: 13
    Ok i found out interesting interaction.
    I edited one of the "invisible for moneo" UTP files via GFF editor in the module folder. Open the toolset back and save the module. And now moneo can see this exact file.

    So probably something is wrong with the file format for moneo (not for toolset, toolset have no problem with the palette).

    Anybody have some idea for script / tool which would open all gff and resave them in batch?

    Just as sanity check i used some other "invisible to moneo" blueprints in the module (place the placeable in test area) and save the module. They are still invisble to moneo. So there have to be something with the GFF format or data areas in these files. (module is very old, reviving PW from 2005).
  • ProlericProleric Member Posts: 1,281
    I just wonder whether the "access violation" fix might also process the missing files?

    As mentioned in a PM, this is
    When looping through files, mysterious access violation crashes can sometimes be avoided by using the curly braces syntax, e.g.
    
    for (%mod[{'*.utp'}]) 
    
    not
    
    for (%mod['*.utp']) 
    
  • iXtremistiXtremist Member Posts: 13
    Proleric wrote: »
    I just wonder whether the "access violation" fix might also process the missing files?

    As mentioned in a PM, this is
    When looping through files, mysterious access violation crashes can sometimes be avoided by using the curly braces syntax, e.g.
    
    for (%mod[{'*.utp'}]) 
    
    not
    
    for (%mod['*.utp']) 
    

    That was what I have tried in the early stages. With 0 luck. It does not fix access violations in my case, neither the file skipping.
  • ProlericProleric Member Posts: 1,281
    Sorry, I'd forgotten that.

    I reached out on the Vault as someone there may know more, though letoscript knowledge is rare these days.

    I could look at your script via PM - preferably one that simply cycles through the .utp files to report their names, which has the skipping and crashing issues.
  • ProlericProleric Member Posts: 1,281
    A few ideas on the Vault link above.

    Niv's nwnlib tool is here - I know nothing about it, but niv is still active.
  • iXtremistiXtremist Member Posts: 13
    So, i found the fix.
    Niv tools. NWN_GFF.
    I unpack the broken UTP to JSON and pack it back to GFF. Viola, MONEO no longer ignore it and older versions stopped throwing access violations.
  • ProlericProleric Member Posts: 1,281
    Good news.

    Can you post a link to that tool?

    I can't seem to find it for some reason.
  • iXtremistiXtremist Member Posts: 13
    https://github.com/niv/neverwinter.nim/releases

    To see how to use it, browse source codes, there are comments for every tool.
  • ProlericProleric Member Posts: 1,281
    Thanks. I recorded the solution here.
  • iXtremistiXtremist Member Posts: 13
    in case anybody on windows need to execute niv tools in mass against files in folder (in this case unpack UTP to json and repack back to original file).. i remembered good old days of MSDOS, just create BAT file from this (bear in mind to have NIV tools in homepath to be able to execute them from any folder) and execute the BAT on command prompt from the folder:

    @echo off
    setlocal enabledelayedexpansion

    echo Iterate through all files with .utp extension
    for %%F in (*.utp) do (
    echo Processing file: %%F
    nwn_gff -i "%%F" -o "%%F.json"
    nwn_gff -i "%%F.json" -o "%%F"
    )

    echo All files processed.
  • MelkiorMelkior Member Posts: 181
    iXtremist wrote: »
    Ok i found out interesting interaction.
    I edited one of the "invisible for moneo" UTP files via GFF editor in the module folder. Open the toolset back and save the module. And now moneo can see this exact file.

    So probably something is wrong with the file format for moneo (not for toolset, toolset have no problem with the palette).

    Anybody have some idea for script / tool which would open all gff and resave them in batch?

    Just as sanity check i used some other "invisible to moneo" blueprints in the module (place the placeable in test area) and save the module. They are still invisble to moneo. So there have to be something with the GFF format or data areas in these files. (module is very old, reviving PW from 2005).

    I'm wondering if the text file was made in a different format? I've found that Linux-based editors tend to save text files with just a line-feed at the end of each line whereas Windows editors always save text files with both a carriage return and a line feed at the end of each line of text. Perhaps Moneo requires the CR-LF format and won't recognise a LF-only as being a valid end-of-line?

    Or the file may have been saved using something other than ASCII.

    A long time ago, I found a simple fix for the first problem was to load the file into Notepad, which does recognise LF as EOL, and then re-save it which causes Notepad to save it with the CR-LF EOL format.
Sign In or Register to comment.