A single TLK file

Wisp

@TrentOster recently tweeted the following (Source):

Scott was fighting with multiple TLK files earlier in the day. This will open ease of modification a fair bit and reduce content collisions

For reasons I will detail below, I ask you to please not do this.

1. It does not actually open ease of modification or reduce content collisions. From a modding perspective, a single TLK presents no problems whatsoever and there is no advantage to having more than one. I will go into more details on the technical reasons why further down.
2. On the contrary, multiple TLK files will actually make modding harder:
   1. There is the obvious problem of every existing modding tool needing to be updated. Until they have been updated, the tools will either not function at all, or produce incorrect results, stymieing modding on the EE platform. Some still-popular tools (e.g., Shadow Keeper) are currently unmaintained and have not been for years, presenting further problems.
   2. ITM, SPL, CRE and other relevant file formats do not contain any information on what source should be used to dereference string references. Nor could they be extended to include such information without breaking compatibility with essentially all existing mods. You could construct various overarching systems intended to solve this problem. For example, you could associate each resource with a TLK file. But it would be one more layer of complexity, which actually reduces ease of modification.
   3. It would make it more difficult to add text to the game, since you would need to specify which TLK the text should be added to. Depending on the circumstances, it may mean an existing mod needs to be updated to account for this, which can be a lot of work, depending on exactly what needs to be done. Some mods have the additional obstacle of no longer having a dedicated maintainer. While they can generally still be updated by the community, response times typically increase significantly.
3. I am only guessing here, but I assume it is a big draw for you because the alternative would mean client-side patching of game resources. But you need to do client-side patching anyway, unless you want the installation procedure for DLC to be (i) uninstall all mods (potentially invalidating saved games, depending on the exact circumstances), (ii) download (and install) DLC and (iii) reinstall mods. DLC content would need to be accessible from some existing point in the game world, which means altering at least one area, script or other existing game resource. To do so safely while mods are installed, you would have to patch the game resource, lest your overwriting of the resource destroys modifications made to it by said mods. This is also why multiple TLK files hold no use to the modding community, becuase you still do not get around the requirement to patch resources client-side.

In summary
If you want to be nice to the modding community (or indeed, anyone modding the game, new or old), a single dialog.tlk (and dialogf.tlk), located in the root of the game's directory hierachy, is undoubtedly the way to go.

Elys

@wisp

About 2 & 3: It depends what methods Scott is implementing.

If each TLK has its own independant String IDs, then indeed you can argue about these points.

However if the String IDs are global to all the TLK files (Like String IDs in Dragon Age), there won't be the issues you are pointing out. As it's the engine at runtime that simply "merge" all the TLK into a unique one.
Then of course that creates a new practical requirement in this case: There will be the need of a String ID ranges public repository so different mods does use the same String ID.

There is also another aspect to consider:

It's better when possible to have each mod using its own resources folder (like what Dragon Age is using). It's clean and easy to manage. The downside is it makes all existing community tools and current mod format obsolete.

So the question is: should the IE engine data system not improve toward a better system with also potential new feature, in order to maintain maximum backward compatibility with existing tools and mods, or should it better evolve relying on the community to convert the existing mods and update the tools ?

Wisp

About 2 & 3: It depends what methods Scott is implementing.

If each TLK has its own independant String IDs, then indeed you can argue about these points.

However if the String IDs are global to all the TLK files (Like String IDs in Dragon Age), there won't be the issues you are pointing out. As it's the engine at runtime that simply "merge" all the TLK into a unique one.
Then of course that creates a new practical requirement in this case: There will be the need of a String ID ranges public repository so different mods does use the same String ID.

If string references are global and the contents of all TLKs are merged in memory, it makes no sense at all to organise them into separate files. It will just make it even more likely that there will be overlap and eclipsing. Your idea of assigning string-reference ranges is also qualifies as one of those additional layers of complexity I wrote of. It is also not backwards compatible with existing mods, where strings are stored as symbolic references within the mod's namespace and translated into a global string reference at install time. You also have the scalability concern of what happens when you exhaust all strings in your range.

It is also a solution to something that is not a problem. Adding/changing text in the current system is EASY.

There is also another aspect to consider:

It's better when possible to have each mod using its own resources folder (like what Dragon Age is using). It's clean and easy to manage. The downside is it makes all existing community tools and current mod format obsolete.

Your proposal would not work, not unless you also respecified all file formats, rewrote all resource handling and/or were willing to accept only having one to a handful mods installed concurrently. Without patching, you very quickly run into the problem of two mods needing to alter the same file. Without advanced patching (better than diff/patch), you have a system vastly less able than the current, with proportionally worse mod compatibility as a result.

So the question is: should the IE engine data system not improve toward a better system with also potential new feature, in order to maintain maximum backward compatibility with existing tools and mods, or should it better evolve relying on the community to convert the existing mods and update the tools ?

If you give the finger to all existing modders, I can virtually guarantee you they will, as one man (or woman), return the gesture. Besides, unless I am mistaken, Trent has already stated that he wishes to see the EEs backwards compatible with existing mods.

Elys

If you give the finger to all existing modders, I can virtually guarantee you they will, as one man (or woman), return the gesture. Besides, unless I am mistaken, Trent has already stated that he wishes to see the EEs backwards compatible with existing mods.

That depends how each particular existing modders answers the question

As for patching it could still be done, you could still have all the files modified by Weidu in an override folder, the same way it works as now. You just have to make sure that the "override folder used by Weidu mods" is the last one loaded.
And obviously you need Weidu-Installation to not only browse Vanilla datas, but other DLC/MOD/Override folder when seeking for files. That's what would need to be updated by the community.

Wisp

As for patching it could still be done, you could still have all the files modified by Weidu in an override folder, the same way it works as now. You just have to make sure that the "override folder used by Weidu mods" is the last one loaded.
And obviously you need Weidu-Installation to not only browse Vanilla datas, but other DLC/MOD/Override folder when seeking for files. That's what would need to be updated by the community.

You can mod this way now (the player just has to back up their override folder before they start installing mods; which would be the sum total of the problems solved by this system). It sucks. It sucks something unimaginably, for the player and the modder.

cmorgan

Another modder perspective here... please listen to Wisp.

I have written and rewritten this post, and basically in the end the Wall of Text(tm) I keep creating is not helping advance the discussion, so I will just toss three basic points I think have relevance:

1. "evolving" an engine *out* of its current strengths seems conterproductive. I think multiple .tlk fails on this idea.
2. I like the idea of BG:EE, and want it to succeed. I can't speak for other modders, but I am the coordinator of BG1NPC, and I would be the one who would have to convert the arguably most popular BG expansion (and certainly the largest expansion of BG content in the SoA/BG2 engine currently available). I even converted the project to BGT, working with a small team of fellow modders, recoding the entire project from the ground up, so you have evidence that I am progressive - some would even say aggressive - in my drive to make sure BG1NPC content gets shared everywhere BG content can exist. And I would seriously have to be convinced it was worth the immense work and time necessary to try to build independent versions that would be compatible under a hierarchical system of .tlk files.
3. I have watched other modders struggle with NWN, NWN2, DA:O, Mount & Blade, Morrowind, and several other games to do one simple thing... modify the existing campaign in a way that allows other mods to coexist peaceably and share content easily. The tools are impressive, you can do all sorts of great stuff... but you just can't patch your story/idea/addition easily into existing game content without ending up adding the other content and distributing it too. If those tools did the job, then the attempts to build BG content in NWN and DA:O would have already had me jumping on their bandwagon.

( RL example, tested and in the field - http://forums.gibberlings3.net/index.php?showforum=166 Romance Pack for the NWN2 OC-MotB. )

Elys, I see your point - but I most definitely do not agree in any way shape or form that this is simpler for players or modders.

Galactygon

I completely agree with Wisp, cmorgan, and Cuv. The only advantage I see here with multiple .tlk and override files is switching from one mod installation to another. But even that can be solved with mutliple backups.

Cross-compatibility between mods is clearly not an issue for the modding communities, in general.

I feel the time and effort should be spent on bugfixing and improving on other parts of the game.

-Galactygon

Wisp

@Galactygon
From what I understand, the override thing is not that you could completely swap out one override for another. It seems to simply be a method for softcoding which directories the game should look in, in addition to "override", "portraits", "scripts" etc.

Elys

@cmorgan: How will the non-free DLC be handled if only using the current TLK system. I doubt DLCs gonna be installed Weidu-like-wise. They might be encrypted to prevent piracy. So my guess is that separate tlk files makes sense from a publisher standpoint because they can be encrypted inside the DLC. And same will probably go for the rest of the files protected inside the DLC. And if they use such addon method for non-free DLC, they maybe gonna use the same for free DLC too.

Encrypted DLC , is more my primary concerns. Because if they add and modify content to the game, you simply cannot (at least legally) read and "patch" them to live along the other community mods.

Additional Override folders are optional and it will be up to the community to use them or not. People can just continue to work out on the main Override folder as usual.
But for those that want to use more override folders for specific purpose (For example for what I do with BGTutu and HyperOverride), the possibilty will exist. So that's not an issue for existing mods, but just an additional optional tool.

It can be the same for TLK, all the community mods could still work on the unique dialog.tlk as usual. So it's not an issue as well. If then some people want to use standalone TLK for whatever personal need, that's up to them. Just a technical possibility, not a must-use.

So I don't see these new features as a problem because all the mod authors should still be able to create mod they way they currently do. Excepted for DLC compatibility of course for the reasons I just cited before.

But that's just all hypothetical talk since we don't have any technical info about what's really going on, and what form the DLC will use

William_Imm

Another modder speaking (well, modder and audience surrogate):

I would say that you should avoid changes that require modding tools to require major changes. Spend your time on bugfixing primary, and then tackle the more doable feature requests.

One thing that would really benefit modders and players alike is implementing an comprehensive and flexible modding system similar to Civilization V, and to a lesser extent, Steam Workshop. This should be complimentary to the WeiDU way and not replace it, but it should allow for selecting from wide variates of mods that tweak, fix, and enhance the game. Yes, I said this several times before, but it really needs to be included.

I completely agree with @Wisp, @cmorgan, @Cuv, and @Galactygon. Don't leave the modders in the dust. Try also to win back some of the modders who already given up on this game - an example would be Solaufein from the Chosen of Mystera forums.

AndreaColombo

I'm no modder but I'd like to chime in and support @Wisp, @Cuv, @cmorgan, @Galactygon and @William_Imm. BG:EE should be as modder-friendly as possible (indeed, it should be a chance to increase the modder-friendliness of the originals).

Cuv

On a side note: I have made contact with several old modders (on an unrelated subject) and directed them here, hopefully they will show up and give some input: @FredRichardson, @Xyx, @Quitch, @the_bigg

Glad to see all these concerns coming out. It's possible we are overreacting though... they may have patching techniques we have only dreamed of that will cause no conflicts. I hope I am right... if not, this thread is saying it all.

Elys

PS: I'm just triggering discussions. Don't get mad at me! xD

And overreacting is usually a symptom of lack of information, with concerns and imagination running wild .. and since we have none ٩(-̮̮̃-̃)۶

cmorgan

Elys, to be clear, I think no one is getting mad at you - the discussion is good. It is an expansion of the comments for your HyperOverride tool, which provides players a good opportunity to create several different mod loadouts placed in different places. Using a tool to have multiple pathing to multiple master overrides, only one of which is active at any given time, is a cool idea - my current way of doing that is to mod the install, back up the modded override and portrait directories, and manually swap them in (labelling saves carefully). But we are talking multiple .tlk as part of the structure of the game, not creating a stable install and then a separate stable install and then another...

On the subject of paid DLC distribution... I don't know. I would think that the most likely scenario is that new content would allow the same access as old content, but I could be wrong - all the original stuff was copyrighted, too. The hope/wish is that (for instance) I could simply add banters between my NPC and the DLC NPC the same way we can now - have weidu look for the existence of the .dlg file, and if it exists, add the content to the appropriate files.

I don't know, though, if that will be the way it happens. I do know that the current i.e. modding community is far more conservative than most modding communities. We turn a blind eye towards the copyright infringement of writing new things for existing content using the characters created by the original team, much like extended fanfics do, but we have an entire culture of "be nice to other modders hard work". I doubt very much that any i.e. modder would be interested in tearing out and distributing what amounts to another person's mod - whether that "fellow modder" person is a paid professional (who, by the way, isn't raking in the big money - heck, no-one on a project like this is raking in big cash, though hopefully it will eventually make the company filthy rich so they can keep single pl;ayer CRPGs alive) or an unpaid traditional amateaur modder. In fact, we are *less* likely to, and likely to make a pretty fierce self-policing community, because we potentially have a vested interest in this project succeeding. If it succeeds and we can use our mods, then an entirely new audience of folks will see our content.

AndreaColombo

From @TrentOster's twitter account:

The system still works fine with weidu style mods, for us, it works better to have multiple tlk spaces for our new content

Could some of the devs elaborate further on the matter in this thread? I believe an open dialog (.tlk - w00t, play on words!) with modders on the matter could greatly benefit both the ongoing development of BG:EE and the modding community.

Tanthalas

Isn't it possible that: "BGEE will have multiple TLK files" is simply a misinterpretation of Trent's words?

I'm no modder so I can only throw out some speculations:

- How does BGT handle the BG1 and BG2 TLK files, they're two separate entities right? Does BGT merge them into a single file? Isn't it possible that that was what Trent was referring to when he said "Scott was fighting with multiple TLK files earlier in the day. This will open ease of modification a fair bit and reduce content collisions", aka what Scott is doing is making a single TLK file?

- Of course, I have no idea what "multiple tlk spaces for our new content" means. /shrug

William_Imm

I think he was responding to one of my Twitters. No say on the Civ-V style mod management, however.

Wisp

@Tanthalas
BGT adds the BG1 strings to the BG2 TLK.

Given the recent tweet about "multiple tlk spaces" it seems very likely that they are in fact referring to multiple parallel TLK files. A TLK is basically a big array, where strings are indexed from 0 and up. With "multiple tlk spaces" they very likely mean they want to have multiple files, so they can have multiple strings with the index 0, one in each TLK file.

It is this context switch I am concerned with. Up to now, there has been zero ambiguity regarding which string is string 0. But with multiple TLKs, it will become ambiguous (string 0 in which file?).

Like @Elys said, maybe they do not intend for their new content to be moddable, in which case this will probably not matter much to anyone but them. (And I certainly hope I am overreacting, Running into problems on account of some change in the EEs would suck.)

ScottBrooks

Sorry, it was Canada day yesterday, so nobody was in the office.

Many people think string IDS are 32bit ints. This is not actually the case.

const DWORD     CTlkTable::STRING_ID_MASK =         0x00FFFFFF;

From the string fetch code.
    //Remove the file id from the strId
    strId &= STRING_ID_MASK;

The top 8 bits of a stringID refer to an internal file id, the other 24 bits refer to the string id as everyone expects.
The BG2 tob engine supports multiple tlk files already, there is just no code on the back end to load arbitrary extra tlk files.

BGEE will have seperate tlk files with the top bits set. The current layout will be as follows.

gui.tlk will refer to all the strings that the GUI expects.
N(2 through 255).tlk will be loaded into the tlk table for the corresponding N bit set.

ex: 2.tlk will be loaded to match any strings in the range 0x02FFFFFF

All of the items, that reference 2.tlk will have string refs with the matching 0x02FFFFFF

To summarize.
If you use weidu to modify gui.tlk, all of the 0x00FFFFFF strings will work exactly as they have in the past.
If you drop a file 14.tlk into the language directory, and have an item that has the 0x0EFFFFFF range of strings set, it will lookup from 14.tlk. If that item references 0x00FFFFFF it will lookup from gui.tlk.

This is not set in stone, but the main reason we are looking to implement this is to resolve the issue we currently have where we have a mixture of BG2 and BG1 strings.

Elys

@ScottBrooks: Thanks for the info . But what about 01000000 .. 01FFFFFF that you did not mention? Is that some reserved range ?

ScottBrooks

01 is used internally already for dialogF.tlk in some of the multilingual versions of the game.

Elys

Oh ok

William_Imm

@ScottBrooks

Any word on implementing @AndreaColombo and my ideas for a Civilization V style mod manager? Also, try to revert back to using single tlk files for the same reasons @Wisp and the others gave, which are still very valid.

TrentOster

First off @Wisp If I want to give someone the finger it will be very up front, obvious and clear. Accusing me or the dev team of giving modders the finger is rude and uncalled for. This is not the internet, this is a private message board and we're all going to behave well in here, agreed?

Second, we are attempting to leverage and existing construct in the code (the first 8 bits as a fileID mask) to make it easier for everyone involved. If the mod tools were not developed with this in mind, we will attempt to look deeper into the issue and come to a better solution.

Thirdly, We are looking into the multiple mod manager concept. This was always one of our goals and is likely going to be pushed post-release.

-Trent

Wisp

First off @Wisp If I want to give someone the finger it will be very up front, obvious and clear. Accusing me or the dev team of giving modders the finger is rude and uncalled for. This is not the internet, this is a private message board and we're all going to behave well in here, agreed?

@TrentOster, I hasten to apologise. I was only speaking of the hypothetical scenario in which Overhaul would break with existing modding practices in favour of a "better system," one that would require existing mods and tools to be rewritten to be compatible (and saying such a move would not go over well). It was not my intention to liken it to anything Overhaul has actually done, or would be prepared to do.

William_Imm

@TrentOster

Thanks for implemting my main suggestion. If you haven't played Civilization V - go do it now, and pay special attention to the modding system. And thanks for clearing up the TLK situation. I won't bug you about it any more on Twitter. But I will ask you to Astral Finish Uwe Boll.

TrentOster

We're going to look deeper into the issues. I understand the concerns around making it hard to "plug" content into the existing campaign. We'll come up with a good solution.

@Wisp - All good.

-Trent

Cuv

Thanks @TrentOster :-) I have complete faith in you guys!

AndreaColombo

Thanks @ScottBrooks and @TrentOster for partaking this thread and shedding some light on the multiple TLK matter. I sure hope devs and modders can work together to make BG:EE the best and most modder-friendly version of Baldur's Gate ever (which I'm quite stoked about).

Boaster

Modding the original TLK file will still be possible, would it not?

Boaster

@TrentOster

We're going to look deeper into the issues. I understand the concerns around making it hard to "plug" content into the existing campaign. We'll come up with a good solution.

@Wisp - All good.

-Trent

What about the suggestion of referencing external text files for descriptions?

http://forum.baldursgate.com/discussion/129/request-text-descriptions-dialog-tlk-modding