I wish I had a tool to translate a module

Japualtah

It's easy to translate a tlk file, but strings scattered all around scripts, dialogs and areas are a pain.
Does such a tool exist?

MadHatter

Wouldn’t the better way to do this be to not include strings directly in your scripts and instead use GetStringByStrRef()?

Or are you translating someone else’s work where they didn’t do that?

Japualtah

Only using resref calls isn't the elegant solution it seems to be, because of how time consuming it is always having the (thrid party) tlk editor open and filling 16777xxx references everywhere or going through the area/dialog dropdown menus twice or more, we speak 3 languages in my country not counting English.

If we had a tool to extract all strings (mostly from dialog/areas though, scripts are cumbersome but manageable) into a single file, translation would be a breeze.

Now, my 15 years old PW is in French, I recently began using tlks for every script and going through the menus hassle but soon forgot about it: way too time consuming and inefficient.

dunahan

You should take a look at https://forums.beamdog.com/discussion/67044/nwn1-command-line-utilities#latest from @niv. With this tools you can extract the ggf-files from every module to json and edit them. Even recompile from json to for example .dlg is possible

niv

Funny you should mention that. We're working on a similar project. I had to write some tooling to support this usecase, and it's not been tested/proven yet, but it'll get there.

https://github.com/niv/neverwinter_utils.nim/blob/master/src/nwn_erf_tlkify.nim

tl;dr:
* Create your module as usual.
* Pipe it into the tool: ./tlkify mytlk.tlk mymod.mod mymodrewritten.mod
* The tool will find all new strings and rewrite them into STRREFs, adding them to the tlk.
* Thus, the generated TLK will be authoritative and the rewritten module depend on it.
* The rewritten module will be written to mymodrewritten. You don't have to continue your work on the rewritten one; as the process is repeatable.
* In fact, I suggest to only do a translation effort once you are done and keep the non-translated mod as the authoritative source.
* Repeat as needed as you go through the release cycles.

Caveats:
* You can rewrite any erf container - if you bundle templates in haks, for example - but I haven't tested that.
* It might miss some strings. (I did say it was untested, right?)
* It might add strings to the translation table that don't need translation ("Tree 48b", I'm looking at you).

Edit: Binaries are here: https://github.com/niv/neverwinter_utils.nim/releases

Would appreciate feedback! Happy to fix up any issues you find.

Japualtah

Geez! Is it XMas already?
Going to check this out as soon as my website porting to WordPress is done, this looks fantastic

dunahan

@niv I have seen that too, but hadn't time to test it. I should compile the newer version by my self, cause I have some problems with your binaries at win 10.
But now I know for what that tool is

Japualtah

I couldn't make it work, unfortunately.
Downloaded your binaries, put an empty .tlk file and module in the same folder as nwn_erf_tlkify, launched nwn_erf_tlkify with, as parameters, the empty tlk, the module and a choosen name for the output and I got an output once (a stripped down module) with an unchanged tlk.
It never worked after that, command line progamming isn't my thing I must say.

Oh well...

Japualtah

Nailed it.

> ./nwn_erf_tlkify.exe mytlk.tlk mymod.mod out.mod
I [2018-10-16T21:04:50] Base TLK name: mytlk
I [2018-10-16T21:04:50] Translations: starting from scratch
I [2018-10-16T21:04:50] Reading: mymod.mod
Error: unhandled exception: len(exolocstr.entries) == 1 exolocstring has multiple languages, but we can only do one language [AssertionError]

Would it be possible to add a parameter to choose the primary language to translate from and discard any other resource the analysis stumbles across?

niv

Good point, will do.

dunahan

Okay, tried with an only english mod. THe tool works like a charm. With >nwn_erf_tlkify mytlk.tlk mymod.mod mytranslated.mod and >nwn_tlk -i mytlk.tlk -o mytlk.tlk.json -p, I could work out the translations.
I got only one warning with nwn_erf_tlkifiy: Error: unhandled exception: getPosition(io) == entriesTableOffset + entriesTableSize [AssertionError]... But the converted tlk seems fine.

+ for an opt to choose the language to translate

niv

You don't have to supply a empty .tlk file. It will create one if it doesn't exist yet (and even put it into module.ifo for you).

The assert triggers AFTER writing the tlk (so it would have been written), indicating a possible error. Likely a encoding/charset issue. Some special character that transformed into a different byte width than expected.

Edit: Can you share the module that's failing? Would love to look at/possibly fix.

dunahan

Okay, I thought that, reading your source code, but wasn't sure about.
Yes it wrote the tlk file flawlessly, didn't spell checked it yet. I wanted to test it first with one english mod, so I can get used to it.
The module I shamelessly used for testing was this one, because mine use only german.
https://neverwintervault.org/project/nwn1/module/mortari-riseth-module

Japualtah

PS C:\Users\Thierry\Desktop\NivTools> ./nwn_erf_tlkify.exe --language "English" althea.tlk althea.mod altheaNiv.mod
I [2019-02-16T13:36:07] Base TLK name: althea
I [2019-02-16T13:36:07] Building translations from given tlk: 629
I [2019-02-16T13:36:07] Reading: althea.mod
Error: unhandled exception: /Users/niv/code/neverwinter.nim/nwn_erf_tlkify.nim(95, 17) `hasKey(exolocstr.entries, int(selectedLanguage)) and
1 < len(exolocstr.entries)` exolocstring has other language strings but not the selected language [AssertionError]

I guess this means I have text resources with French strings while the English resource is empty.
Could it be possible to just put the unexpected string into the expected but empty string please?
My shell skills have been rusty for over 20 years otherwise I'd do it myself, but I really need your help here, I'm now in the process of translating the mod.

Thanks for your continuous efforts mate, much appreciated ;-)

niv

Can you send me the test module?

Japualtah

Thanks mate ;-)

Here is the module in .zip format

niv

I've pushed a experimental commit that supports squashing multiple languages. I've not made binaries yet, so if you need this right now, you'll need to compile it yourself (with nim 0.19.4).

neverwinter.nim → master • ./nwn_erf_tlkify --languages "en,2" althea.tlk althea.mod out.mod
I [2019-02-19T16:51:21] Base TLK name: althea
I [2019-02-19T16:51:21] Selected languages: @[English, German]
I [2019-02-19T16:51:21] Base game dialog TLK at: /Applications/Games/Beamdog Library/00840/data/dialog.tlk
I [2019-02-19T16:51:21] Translations: starting from scratch as language English
I [2019-02-19T16:51:21] Reading: althea.mod
I [2019-02-19T16:51:21] Skipping string: 1_cleric
I [2019-02-19T16:51:21] Skipping string: 2_ranger
I [2019-02-19T16:51:21] Skipping string: 3_war
I [2019-02-19T16:51:21] Skipping string: 4_wiz
I [2019-02-19T16:51:21] Skipping string: 5_pala
I [2019-02-19T16:51:21] Skipping string: 6_monk
I [2019-02-19T16:51:21] Skipping string: ours
I [2019-02-19T16:51:21] Skipping string: alchishop
I [2019-02-19T16:51:21] Skipping string: Althea
I [2019-02-19T16:51:21] Skipping string: Amendil
I [2019-02-19T16:51:21] Skipping string: Gong
I [2019-02-19T16:51:21] Skipping string: Anka
I [2019-02-19T16:51:21] Skipping string: antimelee
I [2019-02-19T16:51:21] Skipping string: archer025
I [2019-02-19T16:51:21] Skipping string: art_spawn_minerais
I [2019-02-19T16:51:21] Skipping string: Aruvil
I [2019-02-19T16:51:21] Skipping string: Chaise
I [2019-02-19T16:51:21] Skipping string: Jukalotto
I [2019-02-19T16:51:21] Skipping string: Balmart
I [2019-02-19T16:51:21] Skipping string: Bazari
I [2019-02-19T16:51:21] Skipping string: Faux
I [2019-02-19T16:51:21] Skipping string: Abishai
I [2019-02-19T16:51:21] Skipping string: Casio
I [2019-02-19T16:51:21] Skipping string: biblio
I [2019-02-19T16:51:21] Skipping string: diampass
I [2019-02-19T16:51:21] Skipping string: Borag
I [2019-02-19T16:51:21] Skipping string: Briquet
I [2019-02-19T16:51:21] Skipping string: bububu
I [2019-02-19T16:51:21] Skipping string: Cache
I [2019-02-19T16:51:21] Skipping string: caster25
I [2019-02-19T16:51:21] Skipping string: Campfire
I [2019-02-19T16:51:21] Skipping string: cleric025
I [2019-02-19T16:51:21] Skipping string: Coco
I [2019-02-19T16:51:21] Skipping string: Cocon
I [2019-02-19T16:51:21] Skipping string: Coluccia
I [2019-02-19T16:51:21] Skipping string: Windmill
I [2019-02-19T16:51:21] Skipping string: Critters
I [2019-02-19T16:51:21] Skipping string: br_stat_boum
I [2019-02-19T16:51:21] Skipping string: errant
I [2019-02-19T16:51:21] Skipping string: Monolithe
I [2019-02-19T16:51:21] Skipping string: d20plus4feu
I [2019-02-19T16:51:21] Skipping string: d20plus4froid
I [2019-02-19T16:51:21] Skipping string: stun
I [2019-02-19T16:51:21] Skipping string: Dalamar
I [2019-02-19T16:51:22] Skipping string: Dergul
I [2019-02-19T16:51:22] Skipping string: Diamant
I [2019-02-19T16:51:22] Skipping string: dispell
I [2019-02-19T16:51:22] Skipping string: Oubliette
I [2019-02-19T16:51:22] Skipping string: Oreille
I [2019-02-19T16:51:22] Skipping string: Eorkwold
I [2019-02-19T16:51:22] Skipping string: Vikan
I [2019-02-19T16:51:22] Skipping string: Pourfendeuse
I [2019-02-19T16:51:22] Skipping string: Saphiria
I [2019-02-19T16:51:22] Skipping string: Dragonique
I [2019-02-19T16:51:22] Skipping string: Psychedelia
I [2019-02-19T16:51:22] Skipping string: Runiolite
I [2019-02-19T16:51:22] Skipping string: Eclaireuse
I [2019-02-19T16:51:22] Skipping string: mercenaire
I [2019-02-19T16:51:22] Skipping string: Majordome
I [2019-02-19T16:51:22] Skipping string: drain
I [2019-02-19T16:51:22] Skipping string: drainlevel
I [2019-02-19T16:51:22] Skipping string: Fly
I [2019-02-19T16:51:22] Skipping string: Frog
I [2019-02-19T16:51:22] Skipping string: Emerald
I [2019-02-19T16:51:22] Skipping string: Emeraude
I [2019-02-19T16:51:22] Skipping string: Enclume
I [2019-02-19T16:51:22] Skipping string: Equilibre
I [2019-02-19T16:51:22] Skipping string: Brazier
I [2019-02-19T16:51:22] Skipping string: chiens
I [2019-02-19T16:51:22] Skipping string: Ercmal
I [2019-02-19T16:51:22] Skipping string: Etripeuse
I [2019-02-19T16:51:22] Skipping string: Euzebius
I [2019-02-19T16:51:22] Skipping string: Fenlick
I [2019-02-19T16:51:22] Skipping string: Forge
I [2019-02-19T16:51:22] Skipping string: forgefollet
I [2019-02-19T16:51:22] Skipping string: Fernal
I [2019-02-19T16:51:22] Skipping string: Fourche
I [2019-02-19T16:51:22] Skipping string: Fraogan
I [2019-02-19T16:51:22] Skipping string: furiatorus
I [2019-02-19T16:51:22] Skipping string: guerrier025
I [2019-02-19T16:51:22] Skipping string: Gysele
I [2019-02-19T16:51:22] Skipping string: H9G9
I [2019-02-19T16:51:22] Skipping string: I8I9
I [2019-02-19T16:51:22] Skipping string: Hennissement
I [2019-02-19T16:51:22] Skipping string: Fireplace
I [2019-02-19T16:51:22] Skipping string: immunbludg
I [2019-02-19T16:51:22] Skipping string: immuncold
I [2019-02-19T16:51:22] Skipping string: immunfoudre
I [2019-02-19T16:51:22] Skipping string: immunfire
I [2019-02-19T16:51:22] Skipping string: immunsonic
I [2019-02-19T16:51:22] Skipping string: immunweaps
I [2019-02-19T16:51:22] Skipping string: instakillcold
I [2019-02-19T16:51:22] Skipping string: Juster
I [2019-02-19T16:51:22] Skipping string: tyiks
I [2019-02-19T16:51:22] Skipping string: kobolds
I [2019-02-19T16:51:22] Skipping string: bats
I [2019-02-19T16:51:22] Skipping string: Toad
I [2019-02-19T16:51:22] Skipping string: Firepit
I [2019-02-19T16:51:22] Skipping string: m_saphhirepass
I [2019-02-19T16:51:22] Skipping string: Well
I [2019-02-19T16:51:22] Skipping string: Furnace
I [2019-02-19T16:51:22] Skipping string: ombrelumiere
I [2019-02-19T16:51:22] Skipping string: MagTroll
I [2019-02-19T16:51:22] Skipping string: Porte
I [2019-02-19T16:51:23] Skipping string: malaar_hide
I [2019-02-19T16:51:23] Skipping string: Malvira
I [2019-02-19T16:51:23] Skipping string: Bookshelf
I [2019-02-19T16:51:23] Skipping string: Mannequin
I [2019-02-19T16:51:23] Skipping string: m_vnp1
I [2019-02-19T16:51:23] Skipping string: gnolls
I [2019-02-19T16:51:23] Skipping string: Martouf
I [2019-02-19T16:51:23] Skipping string: meubleshop
I [2019-02-19T16:51:23] Skipping string: Derguls
I [2019-02-19T16:51:23] Skipping string: Mirroir
I [2019-02-19T16:51:23] Skipping string: Lassiter
I [2019-02-19T16:51:23] Skipping string: ModuleReloader
I [2019-02-19T16:51:23] Skipping string: m_1_pretre
I [2019-02-19T16:51:23] Skipping string: m_base_skin
I [2019-02-19T16:51:23] Skipping string: m_cw001
I [2019-02-19T16:51:23] Skipping string: m_cw002
I [2019-02-19T16:51:23] Skipping string: m_cw003
I [2019-02-19T16:51:23] Skipping string: m_cw004
I [2019-02-19T16:51:23] Skipping string: N11O11
I [2019-02-19T16:51:23] Skipping string: r_mag_troll
I [2019-02-19T16:51:23] Skipping string: Nakar
I [2019-02-19T16:51:23] Skipping string: Database
I [2019-02-19T16:51:23] Skipping string: Ogorni
I [2019-02-19T16:51:23] Skipping string: Opale
I [2019-02-19T16:51:23] Skipping string: Ormidel
I [2019-02-19T16:51:23] Skipping string: P12O12
I [2019-02-19T16:51:23] Skipping string: batayeter
I [2019-02-19T16:51:23] Skipping string: WP_gol
I [2019-02-19T16:51:23] Skipping string: Fountain
I [2019-02-19T16:51:23] Skipping string: Scorpions
I [2019-02-19T16:51:23] Skipping string: Chiselings
I [2019-02-19T16:51:23] Skipping string: Pynitharalotel
I [2019-02-19T16:51:23] Skipping string: queenspider
I [2019-02-19T16:51:23] Skipping string: receptacle
I [2019-02-19T16:51:23] Skipping string: regen20
I [2019-02-19T16:51:23] Skipping string: Robe
I [2019-02-19T16:51:23] Skipping string: Rubis
I [2019-02-19T16:51:23] Skipping string: Sanbrui
I [2019-02-19T16:51:23] Skipping string: Sangflamme
I [2019-02-19T16:51:23] Skipping string: Saphire
I [2019-02-19T16:51:23] Skipping string: Sceptrius
I [2019-02-19T16:51:23] Skipping string: Scorpion
I [2019-02-19T16:51:23] Skipping string: Serveuse
I [2019-02-19T16:51:23] Skipping string: Situlapri
I [2019-02-19T16:51:23] Skipping string: slow
I [2019-02-19T16:51:23] Skipping string: Centre_Egouts
I [2019-02-19T16:51:23] Skipping string: Forges
I [2019-02-19T16:51:24] Skipping string: Stabilisateur
I [2019-02-19T16:51:24] Skipping string: Tavernier
I [2019-02-19T16:51:24] Skipping string: Tonton
I [2019-02-19T16:51:24] Skipping string: Torcol
I [2019-02-19T16:51:24] Skipping string: TS
I [2019-02-19T16:51:24] Skipping string: Banc
I [2019-02-19T16:51:24] Skipping string: Verdanite
I [2019-02-19T16:51:24] Skipping string: Violite
I [2019-02-19T16:51:24] Skipping string: Willytell
I [2019-02-19T16:51:24] Skipping string: malaar
I [2019-02-19T16:51:24] Skipping string: zupashop
I [2019-02-19T16:51:24] Skipping string: _nbde
I [2019-02-19T16:51:24] Saving tlk, new translations: 2971

Japualtah

I'll be waiting for your binaries, I have no way to compile that atm, no hurry
I don't quite see where the problem is though, the first item in your list is a creature blueprint (1_cleric) and besides the name (which only exists in English) I can't see any other translation mismatch possibility.
Whatever, it's not important, I'm very happy with your support and will wait for the binaries, all is fine ;-)

niv

@Japualtah https://github.com/niv/neverwinter.nim/releases/tag/1.2.3

Japualtah

@niv We're getting there!

./nwn_erf_tlkify.exe --languages 0,1 --root 'C:\Program Files (x86)\Steam\steamapps\common\Neverwinter Nights' althea.tlk althea.mod a2.mod
I [2019-02-20T10:39:50] Base TLK name: althea
I [2019-02-20T10:39:50] Selected languages: @[English, French]
I [2019-02-20T10:39:50] Base game dialog TLK at: C:\Program Files (x86)\Steam\steamapps\common\Neverwinter Nights/data/dialog.tlk
I [2019-02-20T10:39:50] Building translations from given tlk: 83
I [2019-02-20T10:39:50] Reading: althea.mod
Error: unhandled exception: /Users/niv/code/neverwinter.nim/nwn_erf_tlkify.nim(119, 15) `isSome(foundLanguage)` none of your selected languages satisfy string: [4294967295]{2: "Syrthe, Office du recrutement, 1er Útage"} [AssertionError]

I tried toying with --nwn-encoding CHARSET trying UTF-8 and ANSI (unknown) but it seems the word "étage" is the problem and gets transformed into "Útage", I can't go any further.

My original files are still available if needed.

Please note all my scripts in the new module a2.mod contain "MOD V1.0" in place of any code, which I guess is because the process didn't go to the end, since the areas can not be opened either and have their name changed with their tag.

Thanks again for you support, I can finally see the light at the end of the tunnel ;-)

niv

[4294967295]{2: "Syrthe, Office du recrutement, 1er Útage"}

2 Is German. You're only adopting languages 0 and 1. See languages.nim here: https://github.com/niv/neverwinter.nim/blob/master/neverwinter/languages.nim

Japualtah

type
Language* {.pure.} = enum
English = 0
French = 1
German = 2

reason why I typed ./nwn_erf_tlkify.exe --languages 0,1

*confused*

There's no German fork in the aforementioned resource, both English and French contain the same string "Syrthe, Office du recrutement, 1er étage".

*wanders around in utter confusion*

niv

Well, your .mod contains texts stored under language ID 2, which is the german codepath. Even if the text itself is in french.

Japualtah

As you can tell from the picture, I'm happy!
You're the man Niv, consider yourself hugged!

Now a last question if I may: all my text resources now appear on a yellow background which means they come from my .tlk file.
Is it possible when, for example, creating a new zone, to have the zone Name directly pointing towards the .tlk like your tool did?

Cheers!

niv

The toolset doesn't support that, but you can just re-run the utility as you go along and it will adopt any new untranslated entries into the existing tlk.