Single File Mod Format (benefits iPad, Android, and everyone else)

Nifft

Hi. As an experienced programmer, I'd like to suggest a format for BGEE and BG2EE mods which will help mods, help hot-fixes, and generally be better for everyone.


Here's the format:
- ZIP file named with the Override file structure for mods ("modname.mod"); and
- ZIP file named with hot-fixes for both Override mods and the core game ("modname.fix" or "bg1core.fix").
- Mod ZIP files will include a file named "modname.txt", which will have a list of language prefix to UTF-8 strings describing the mod file*.
- "whatever.fix" will be loaded as a hotfix for "whatever.mod", allowing modders to provide hotfixes for their mods.
- "bg1core.fix" will be loaded as a hotfix for the vanilla game.

Format Specifics:

Filenames:
.../override/modname_1.2.3.mod
.../override/modname_1.2.3.fix

Contents (same for .MOD and .FIX files):
./metadata.txt - a key-value file for metadata

name="My Mod Stack Name"
description="This stack comprises my best attempt to combine KitFix, DevineRemix, the BG1NPC beta (2013-03-03) and my own ease-of-use hacks. Blah blah blah..."
... whatever other metadata needs to be stored ...

./lang/ - like the current "lang" folder
./lang/de_DE/
./lang/de_DE/dialog.tlk
./lang/en_US/
./lang/en_US/dialog.tlk
etc., etc.
./override/ - can you guess?
./override/kit.ids
./override/kitlist.2da
etc., etc.

Migration Path:

Stage 1: Add support for .MOD / .FIX files.
- Keep support for existing Override folder files.

Stage 2: Deprecate "old-style" mods once new-style .MOD and .FIX format is working.
- If old-style use of the Override folder is detected, pop up a warning.
- Post instructions on your website about how to zip up an old-style Override folder.

Stage 3: Exclusively support new format.
- Any files other than .MOD or .FIX in the Override folder cause a warning, but are otherwise ignored.

I'd expect migration to take a few months.

Here's how the game uses these formats:
- When you start a new BGEE game, the game looks at all available mods (".mod" files in your Override folder) and you can choose one based on its internal name and modfile name. When you choose a mod, your save games will include the mod's internal name and MD5 checksums.
- The "new game" screen displays both the internal and external names for a mod. So, if you install a mod with a file name like "bg1scs46.mod", inside the game you'll be asked if you want to start a game with "BG1 Sword Coast Strategies 4.6 :: bg1scs45.mod" or just the vanilla game.


Internally, the BGEE game can do the following:
- On start-up, look at the checksums of the file structure and file contents of all the "*.mod" files in the Override folder. Store these checksums (not mod names).
- When saving a game, save the MD5 checksums of the mod file structure and mod file contents. (All zeros for a vanilla game, of course.)
- When loading a new or saved game, load the mod of the same checksum, and any hotfix file ("zzz.fix", where 'zzz" = mod name, or 'bg1core' for a vanilla game).

- - -

This allows the following benefits:
- People can create and share mod-packs for any arbitrary set of mods.
- These mod-packs can be loaded as a singe file on any platform.
- Mod-packs are identical across platforms.
- Saved games can load the correct mod even when shared between platforms.
- Hot fixes for mods can be provided by modders (and even by you, Beamdog and Overhaul, as necessary).
- Multiple mods can be installed for a single binary (which is important on platforms which prohibit duplication of the BG binary, like iPads).
- Multiple versions of the same mod can be used on a single system (important for modders who want to test their mod before shipping changes).

Let me know if you see any issues with this proposed system.

Thanks, --N

---

*) File format for language-prefix to UTF-8 strings:
en_US:Sword Coast Strategems
es_SZ: Espada Costa Estratagemas
fr_FR: Épée Côte Stratagèmes
... et cetera.


EDIT: added some specifics.

CamDawg

Anything which relies on updates from modders who are long gone is not going to work.

Even for those still active, breaking BG2 Tweaks into 85 mods (or SCS into its 100+) is not going to help, especially when those components will no longer affect other modded content. The .mod file already exists as .tp2.

The real problem is that we simply lack the ability to execute the scripts on some platforms. The rest of your proposed improvements could probably be accomplished by adding the local weidu.log into save games.

Nifft

Thanks for taking a look at this proposal, @CamDawg.

CamDawg said:

Anything which relies on updates from modders who are long gone is not going to work.

Sure. I'm not seeing how they'd be required to do any updates. This is something for WeiDU to support as an output format, not something modders need to care about... except insofar as modders want a better environment for testing.

CamDawg said:

Even for those still active, breaking BG2 Tweaks into 85 mods (or SCS into its 100+) is not going to help, especially when those components will no longer affect other modded content. The .mod file already exists as .tp2.

I'm aiming for two things: portable stacks of mod installations (so you can run a set of .tp2 files on ONE platform and then install the results on ALL your devices), and parallel stacks of mod installations (so you can kick the tires on a new mod without screwing up your other games in progress).

Nothing about splitting up SCS. Rather the opposite.

CamDawg said:

The real problem is that we simply lack the ability to execute the scripts on some platforms. The rest of your proposed improvements could probably be accomplished by adding the local weidu.log into save games.

New platforms also have new integrity constraints. You may not have the ability to modify "dialog.tlk" on the iPad, and you certainly don't have the ability to install multiple copies of the game on the iPad. Package integrity may also be a concern on Macs which go through the App Store.

Just porting WeiDU to iPad may not be possible.

What I'm proposing circumvents the integrity issues, and I think the result will be a lot more useful to people who play the game on multiple platforms, especially people who want to use the same saved games on multiple platforms.

As a side benefit, it'd be easy to post your finished .MOD file to share with your friends for guaranteed identical mods in multi-player -- and for the game to be able to check if you're actually all playing with the same mods.

Thanks, -- N

DavidW

Why doesn't this lead to combinatorial explosion? There are something like 2^130 possible installs of SCS alone. I can't possibly produce one hotfix for each combination. (And that's before allowing for other mods - not that it matters by this stage!)

Nifft

There's no combinatorial explosion because this format is the output of WeiDU, not a replacement for WeiDU.

CamDawg

Well, I think my concern here is that mods have really gone far beyond simple override dumps; the install scripts actually do possess a great deal of complexity and will result in vastly different outcomes based on the presence of other mods. Finding a way to process those existing scripts remains the best way forward. Even within something like BG2 Tweaks, there are checks and install branches based on other components. Your override output will depend heavily on what's gone before, which is why I don't think the single ubiquitous file will work. We're pretty close to this anyway--mods are usually a copy of WeiDU and a folder; older mods will sometimes still have their tp2 outside of the folder.

Nifft said:

CamDawg said:

I'm aiming for two things: portable stacks of mod installations (so you can run a set of .tp2 files on ONE platform and then install the results on ALL your devices), and parallel stacks of mod installations (so you can kick the tires on a new mod without screwing up your other games in progress).

Sure, but what I think you're ultimately going to need is simply a friendly, portable WeiDU front end that can process an external weidu.log as an install script. My skepticism stems, in part, from the fact that this has been a goal of the modding community for over a decade, and yet, here we are.

I'd actually rather see such a project as independent of EE, since the existing IE games would also benefit from it.

Nifft said:

CamDawg said:

The real problem is that we simply lack the ability to execute the scripts on some platforms. The rest of your proposed improvements could probably be accomplished by adding the local weidu.log into save games.

New platforms also have new integrity constraints. You may not have the ability to modify "dialog.tlk" on the iPad, and you certainly don't have the ability to install multiple copies of the game on the iPad. Package integrity may also be a concern on Macs which go through the App Store.

Well, if we can't modify dialog.tlk we're screwed, new format or no.

And just so that I don't get Overhaul in trouble or make anyone think I'm talking about future plans, I'm speaking for me personally as a modder.

edit: borked the quote tags

Nifft

CamDawg said:

Well, I think my concern here is that mods have really gone far beyond simple override dumps; the install scripts actually do possess a great deal of complexity and will result in vastly different outcomes based on the presence of other mods. Finding a way to process those existing scripts remains the best way forward. Even within something like BG2 Tweaks, there are checks and install branches based on other components. Your override output will depend heavily on what's gone before, which is why I don't think the single ubiquitous file will work.

My proposed file format is a replacement for the override directory. In my schema, you could say:

"Any given .MOD file's contents depend heavily on what's gone before."

CamDawg said:

Sure, but what I think you're ultimately going to need is simply a friendly, portable WeiDU front end that can process an external weidu.log as an install script. My skepticism stems, in part, from the fact that this has been a goal of the modding community for over a decade, and yet, here we are.

I'd actually rather see such a project as independent of EE, since the existing IE games would also benefit from it.

The reason I'm posting this in BGEE Feature Requests is that this format would require support from the BG game engine. Basically I want to get rid of the override folder as it currently exists, and replace it with something better. That's not something I can do alone, it's got to come from Beamdog / Overhaul.

They'll need to compile in libzip.a to support the file structure, and they'll need to change the game to do all the checksum checking I described in the first post. This format allows one level of indirection, but otherwise ought to function just like the current override folder once you pick a .MOD to load.

Older games don't have an active development team to petition with Feature Requests, so I'm not hopeful that this format will be of any use to them.

CamDawg said:

Well, if we can't modify dialog.tlk we're screwed, new format or no.

The BGEE devs want to support modding, from everything I've read, so I think it's in their interest to help UN-screw us.

This format is a way to do that.

If that means the game needs to look in the .MOD file and handle a replacement dialog.tlk correctly, I think they'll be amenable to that... especially if the alternative is to petition Apple and Google to change their app package integrity rules!

Dee

Wouldn't that change break all of the existing mods that are out there?

Nifft

Aosaw said:

Wouldn't that change break all of the existing mods that are out there?

Why would it? The override folder still exists, and for the one platform where mods currently work, you can keep using it.

This new mechanism is a better replacement for the override folder -- works on more platforms, etc. -- but I don't see why you'd be required to throw away the old mechanism.

Dee

That's why I was asking; you said you wanted to "get rid of the override folder".

Personally, I think there's a better way to handle it without causing compatibility issues and still make it user-friendly.

Nifft

Aosaw said:

That's why I was asking; you said you wanted to "get rid of the override folder".

The eventual goal certainly is to deprecate the override folder, by providing a better mechanism that everyone would prefer to use.

Breaking the override folder is NOT the first step to this process! ;-)

Aosaw said:

Personally, I think there's a better way to handle it without causing compatibility issues and still make it user-friendly.

Do tell!
What is the better way?

Thanks, -- N

Dee

Well, like @CamDawg, I'm speaking personally as a modder and a player, not in any official capacity. What I would rather see, though, is a structuring of the override folder to allow for nested directories, and then a "Mods" interface in the game itself to allow specific folders to be checked off and reordered to determine prioritization.

Even that has potential for issues, but it would be more player-friendly; and because it would still be folders, it would be a simpler solution for modders (you'd just have to set the output directory to override\modname\) than compressing the files.

DavidW

Nifft said:

There's no combinatorial explosion because this format is the output of WeiDU, not a replacement for WeiDU.

But you're proposing use of hotfixes. Theoretically there might have to be 2^130 hotfixes.

DavidW

Just to expand on Cam's point (and with apologies for double-posting):

SCS, as an illustration, goes through every creature in the game, looks at its level and the weapons it's equipped with, and assigns it a script, a set of spell choices, and a set of weapon proficiencies (and, if appropriate, kit powers and HLAs) based on that level and those weapons. So any mod installed before SCS that changes the level, or weapon choices, of any creature in the game, will change the output from SCS.

No amount of separating out SCS's files from other files (e.g. by Aosaw's proposal) is compatible with this, because there aren't any "SCS files": there are just "SCS instructions", whose effects on the files are dependent on the instructions given by previous mods. (As another illustration, note that SCS itself is a couple of megabytes exclusive of WEIDU, but it generates several hundred megabytes of output.)

This is the beauty (but also the complexity) of IE modding. Because we distribute instructions, not files, it's vastly easier to manage compatibility between multiple complex mods. Both SCS and BG2 Tweaks modify huge numbers of overlapping files, but they coexist perfectly happily in a way that would be virtually impossible for, say, KOTOR.

Of course, any combination of mods generates some fixed override file (and possibly dialog.tlk). It's already the case (modulo issues of BIFFing) that you can just zip up the dialog and override files and put them in a new install. But that doesn't work as a general method of distribution.

Nifft

Aosaw said:

Well, like @CamDawg, I'm speaking personally as a modder and a player, not in any official capacity. What I would rather see, though, is a structuring of the override folder to allow for nested directories, and then a "Mods" interface in the game itself to allow specific folders to be checked off and reordered to determine prioritization.

Even that has potential for issues, but it would be more player-friendly; and because it would still be folders, it would be a simpler solution for modders (you'd just have to set the output directory to override\modname\) than compressing the files.

It sounds like you're looking for something orthogonal to what I'm proposing. What I'm talking about is all back-end stuff, and could be totally compatible with your front-end vision. ZIP files explicitly allow structure, and my proposal takes advantage of that for the multi-language 'dialog.tlk' files (if nothing else).

My proposal is explicitly designed to be a SMALL change on the part of the BGEE coders, which allows a very high degree of safety, ease of management, and ease of experimentation for the end-user. I want this because I'm an end user, and the game is currently much more painful to mod than it has to be.

If Beamdog / Overhaul embrace my vision for the back-end, you may be able to leverage the management and file-structure features to implement your vision. We are coming at a similar problem with different solutions, but I think our solutions are compatible.

DavidW said:

But you're proposing use of hotfixes. Theoretically there might have to be 2^130 hotfixes.

... all stuffed within a single file, which is clearly labeled as a "hotfix" file. Sure.

Since the mod-stack is contained in a single file, it will be possible to write a tool to generate the "hotfix" file based on the mods you have installed and the modder's website(s), if we modders can agree on a common way to post hotfixes.

What we have now are hotfixes mixed in your /override/ folder willy-nilly, unlabeled, and therefore when you upgrade to a new version of a mod, your only option is to throw everything out and start over from scratch. With 2^130 possible hotfixes, and no possible automation, and no easy way to check if any given hotfix is even installed, it sounds like what we have right now is very hard to manage, and what I'm proposing is much easier to manage.

Are you seeing an upside or a downside that I'm missing?

Thanks, -- N

Nifft

DavidW said:

Just to expand on Cam's point (and with apologies for double-posting):

SCS, as an illustration, goes through every creature in the game, looks at its level and the weapons it's equipped with, and assigns it a script, a set of spell choices, and a set of weapon proficiencies (and, if appropriate, kit powers and HLAs) based on that level and those weapons. So any mod installed before SCS that changes the level, or weapon choices, of any creature in the game, will change the output from SCS.

No amount of separating out SCS's files from other files (e.g. by Aosaw's proposal) is compatible with this, because there aren't any "SCS files": there are just "SCS instructions", whose effects on the files are dependent on the instructions given by previous mods. (As another illustration, note that SCS itself is a couple of megabytes exclusive of WEIDU, but it generates several hundred megabytes of output.)

This is the beauty (but also the complexity) of IE modding. Because we distribute instructions, not files, it's vastly easier to manage compatibility between multiple complex mods. Both SCS and BG2 Tweaks modify huge numbers of overlapping files, but they coexist perfectly happily in a way that would be virtually impossible for, say, KOTOR.

Of course, any combination of mods generates some fixed override file (and possibly dialog.tlk). It's already the case (modulo issues of BIFFing) that you can just zip up the dialog and override files and put them in a new install. But that doesn't work as a general method of distribution.

Interesting! I'm definitely curious about the machinery behind such complex mods. And we are not at odd here!

I'm not proposing a replacement for WeiDU.

I'm proposing a different back-end from the (difficult to manage) single /override/ directory + hacked single 'dialog.tlk' file.

You certainly COULD share the back-end if you really wanted to, and it's one way I envision getting an identical mod-stack on two different devices (i.e. run WeiDU on windows to generate a complex mod-stack, then copy mod-stack to iPad), but it's NOT intended to be a general method of distribution.

A very specialized method of distribution, perhaps, like if you wanted to play multi-player with a deep mod-stack, and you wanted to make sure everyone had exactly the right mods... but that's unusual, I think.

Thanks, -- N

DavidW

I don't see how it's possible to have a fundamentally different back end. Mods are instructions to change various files, possibly including files already changed by previous mods. There's just no way of dividing the override into files modified by mod A, files modified by mod B, etc.

As for hotfixes, we already do have a community-agreed standard for issuing them: update your mod. When "hotfixes" are distributed to complicated mods, they're nearly always things to drop into the mod folder before installing, not things to drop into the override folder after installing. And it has to work that way for complicated mods - having 2^130 different hotfixes isn't just annoyingly complicated, it's physically impossible.

Anton

Still hope that Overhaul & Modders will be able to find some solution that would allow us to run modded version of BGEE on Androids & iPads

SirK8

I've been giving some thought to @Nifft's idea and I think it has some merit, if I'm understanding what he wants to do correctly. Basically a "mod stack" as he puts it would be a compilation of multiple mods or even a single mod. So let's say I want to create my mod mix or "stack", I could call it SirK8_super_mod_mix. Then I run the WeiDU installer for the mods I want, specifying the mod stack want the output written to instead of the normal override directory. Also the edited dialog.tlk would be included but the original would not be overwritten, the custom one would just be loaded any time that mod was loaded.

So for example I install some NPC mod, then SCS and now instead of the override directory containing the output and the main dialog.tlk being overwritten, all of the files would be inside of SirK8_super_mod_mix.mod, including a custom dialog.tlk. Now when I start a new game I can choose to run Vanilla BG:EE OR start a game using SirK8_super_mod_mix. Now every time I load that game it checks for the right mod and dialog.tlk and loads them.

Now let's say I want to try out some fancy new mod with kits, but I don't want to screw with my current game, I want to include the NPC mod but I don't want SCS. So I compile a new mod stack, using WeiDU again to install those two mods into NPC_Kit.mod. Now I can start a new game loading that mod combo and my other game is unaffected.

Next, let's say SCS comes out with a major update where I would have to restart a game due to changes being incompatible with current saves. I want to update it but I don't want to give up my previous SCS game. So I can run the update against my SirK8_super_mod_mix.mod mod stack using WeiDU, but specify a new mod stack for output, meaning the original files in that mod stack would not be edited, but instead the edited files would be written to SCS_updated.mod. Now I can enjoy both my current game and check out the new SCS features all with just one BG:EE install.

Not sure how hot fixes works into this o.O

Disclaimer: I am not a modder or computer programmer, I'm just trying to wrap my head around this idea that seems to have some merit/potential.

Nifft

SirK8 said:

I've been giving some thought to @Nifft's idea and I think it has some merit, if I'm understanding what he wants to do correctly. Basically a "mod stack" as he puts it would be a compilation of multiple mods or even a single mod. So let's say I want to create my mod mix or "stack", I could call it SirK8_super_mod_mix. Then I run the WeiDU installer for the mods I want, specifying the mod stack want the output written to instead of the normal override directory. Also the edited dialog.tlk would be included but the original would not be overwritten, the custom one would just be loaded any time that mod was loaded.

So for example I install some NPC mod, then SCS and now instead of the override directory containing the output and the main dialog.tlk being overwritten, all of the files would be inside of SirK8_super_mod_mix.mod, including a custom dialog.tlk. Now when I start a new game I can choose to run Vanilla BG:EE OR start a game using SirK8_super_mod_mix. Now every time I load that game it checks for the right mod and dialog.tlk and loads them.

Now let's say I want to try out some fancy new mod with kits, but I don't want to screw with my current game, I want to include the NPC mod but I don't want SCS. So I compile a new mod stack, using WeiDU again to install those two mods into NPC_Kit.mod. Now I can start a new game loading that mod combo and my other game is unaffected.

Next, let's say SCS comes out with a major update where I would have to restart a game due to changes being incompatible with current saves. I want to update it but I don't want to give up my previous SCS game. So I can run the update against my SirK8_super_mod_mix.mod mod stack using WeiDU, but specify a new mod stack for output, meaning the original files in that mod stack would not be edited, but instead the edited files would be written to SCS_updated.mod. Now I can enjoy both my current game and check out the new SCS features all with just one BG:EE install.

Not sure how hot fixes works into this o.O

Disclaimer: I am not a modder or computer programmer, I'm just trying to wrap my head around this idea that seems to have some merit/potential.

Yes! You have it perfectly correct.

This also solves some issues like "what if I install some mods in English and suddenly my gf wants to start a game in German", which the core game would support, but which the current state of modding screws up entirely.

DavidW said:

I don't see how it's possible to have a fundamentally different back end. Mods are instructions to change various files, possibly including files already changed by previous mods. There's just no way of dividing the override into files modified by mod A, files modified by mod B, etc.

That's not what I'm proposing at all. Maybe it's not clear because @Aosaw said something, and you're reacting to his post?

DavidW said:

As for hotfixes, we already do have a community-agreed standard for issuing them: update your mod.

The reality is that people are using hotfixes, right now, for a variety of issues: Neera romance, archery APR, and in recent memory for the exceptionally painful Beregost crash bug in BGT / Tutu.

I want my proposed backend to have no less functionality than what we have now, so support for hotfixes is something I'm including. You can dislike them if you want, but I don't really see why. People are using them right now to fix broken things, and that's Lawful Good in my code of conduct handbook.

Nifft

Aosaw said:

Well, like @CamDawg, I'm speaking personally as a modder and a player, not in any official capacity. What I would rather see, though, is a structuring of the override folder to allow for nested directories, and then a "Mods" interface in the game itself to allow specific folders to be checked off and reordered to determine prioritization.

Even that has potential for issues, but it would be more player-friendly; and because it would still be folders, it would be a simpler solution for modders (you'd just have to set the output directory to override\modname\) than compressing the files.

@Aosaw - I've been thinking about this.

IMHO it's to Beamdog's benefit to have a clean separation of game and modding tools, both from a support standpoint, and from a legal liability standpoint.

Basically, if you put the mod runner internal to the BGEE game executable, you're opening yourself up to a huge number of potential bugs, and Beamdog support is going to have to hear about those bugs -- even though the modders will be the ones who caused, and who can fix, the bugs.

If I were you, I'd stay out of that. The user who is running WeiDU knows who to blame when his mod install pops up an error.

That's before your legal guys butt in and start asking about liability when a modder's script causes someone's video card to explode while the script is being run internally by Baldur.exe...

Dee

I think you may be misinformed about how liability works with regard to software development.

If you modify the software, any issues resulting from that modification are your own responsibility. The behavior I described above would simply allow simpler mods (the kind that are just extracted into your override folder) to be sorted into folders and ticked as "active" or "inactive" within the game. It doesn't install them or change how they work; it only determines whether those folders are used as game resources.

Nifft

Aosaw said:

The behavior I described above would simply allow simpler mods (the kind that are just extracted into your override folder) to be sorted into folders and ticked as "active" or "inactive" within the game.

According to @DavidW, "Mods are instructions to change various files, possibly including files already changed by previous mods." ... so it sounds like the two of you disagree about what modding requires.

@Aosaw, your choices are:
1 - Prohibit SCS (and the like), but you get simple "just extracted" mods; or
2 - Allow SCS (and the like), but the game engine must now run mod scripts, and what I said before applies.

IMHO neither of those are desirable for Beamdog, but hey, I'm just some outsider who wants better mod support. You Beamdogs do what you think is best.

- - -

Either way, I feel we're wandering off topic.

As an iPad user, I'd like to be able to install mods. This format I've detailed would suffice for all platforms, would make mod-stacks portable across devices just like saved games are -- and they should be, since you need identical mod-stacks to open a saved game on different devices -- and it should be relatively easy to implement on Beamdog's side, since the internal file structure is the same, with just one added level of indirection.

If y'all choose to do something else, and it works, that's GREAT! This is here as an example of a system which would work, which would support all current mods, and which would be easy for you to implement.

Thanks, -- N