Override Folder
CameronTofer
Administrator, Developer Posts: 279
So, we have the Override folder in BG1, BG2, IWD, etc. I'm glad it's there. But would it be possible to create and utilize Override sub folders? This way we might keep additions players want to add more organized?
A simple suggestion would be folders called "Patch" with a subsequent number following: "Patch001"
http://www.reddit.com/r/FriendsOfBaldursGate/comments/u52l4/request_override_folder/
A simple suggestion would be folders called "Patch" with a subsequent number following: "Patch001"
http://www.reddit.com/r/FriendsOfBaldursGate/comments/u52l4/request_override_folder/
Post edited by CameronTofer on
6
Comments
MrManlyMan1 said:
This has just been fixed. You can now specify a list of override folders in the .ini or at the command line.
Modders rejoice!
Actually other games manage to navigate subfolders with no trouble somehow, letting you organize and name all your folders as you like since it only looks for the heading "override" in the proper directory and presumably ignores all but the files and their type. DAO definitely does, but I'm not sure what process they're employing. Not sure either what learning process is going to be needed in the Patch folder. Does this mean we'd put our mod "Mega Cheat Game Breaker" into a folder called "Patch001"? Then we'd have to remember that MCGB is associated with 001, but on a different install it might then be in Patch003 given a new install order that puts mods "TrentONPC" and "Add Machete Proficiency" first... Tell me it's not like that...
[Overrides]
myoverride=c:\somefolder
supermod=c:\supermod\files
The take precedent in the order in the file. As well you can specify them at the command line:
for example:
BGMain.exe --set OVERRIDES mymod c:\mymodfiles\
Instead of pointing to a folder outside of the Override folder, it should look more like so:
[Overrides]
PLUSmod=mods\PLUS\
Profmod=mods\Proficiency\
Simply point to the folders inside the Override folder. Making the custom overrides sub-directories outside of the override folder may just make it confusing for some.
No need to point to extraneous folders such as: C:\users\Boaster\Documents and Settings\Application Data\BG1EE\mods\PLUS
Keep it super simple.
And I agree, order taking precedence is a good feature!
For example, this would be my directory architecture leading up to the Override folder. What I would do, and perhaps it's just me, is that if I were to make a mod for BG1 or BG2 (which I have, though not extensive as some) would be to organize any such mods I install.
Inside the "override" folder I would create a folder called "mods" and anything that's in the root directory of the Override folder I, personally, would classify this as "patch" data or "from the makers."
Then, inside the mods folder (D:\games\Black Isle\BG1EE\override\mods\) I would then create a folder for each mod I would install to keep things separate. In my humble opinion, it is organized and simple. And to be able to just simply point to the folder, as mentioned previously...: ...would simply work best (in my opinion).
Simplicity and organization. Anyone else have thoughts on this? Or would people just defer to how I stated things?
Moreover, with several folders you can have the same file in several places and the same file can have several descriptions in tlk... If you want to make the life of the modder a hell, it's a good idea... Otherwise it's a very bad idea.
This looks and sounds chaotic. On the surface, you can't tell what mod's changes are taking precedence. Visually, it looks less organized.
BUT - the WeiDU.log contains the details, and weidu itself can tell you exactly what has touched what file in what order.
WeiDU does not touch the original installation; if you want a vanilla game, delete your override folder and pop the vanilla one back in place. Voila.
AND here comes the smashing great huge blow that makes most "modern" modding tools and systems subpar at the very best...
...every mod can add to the same file, manipulate it, change it, all at once, on the same install, at the same time.
With an override hierarchy, you have juggling balls. Whichever one is highest takes precedence, and everything else vanishes from existence. Removing specific things manually *looks* easy from an end-user standpoint, but it is very scary for things more intricate than graphics manipulation. Look at the hades it is to create multiple mod collaboration on any number of modern games.
With a layered approach like WeiDu uses, the game keeps track of the changes, allowing content to be added like the rings of an onion. Actually, it is more like a ball of string - the original content is a few unwinds away, and every mod added can check for dependencies all the way back down the string to the original source file, unwind to whatever point it needs to, work in new content, and reroll the ball of string. If I wanted length 1.5cm to 2cm to be blue, I check it, and if it is already blue, I move on - if not, I change it blue.
As far as priority orders, having more places to stuff things into is not in itself bad. The problem is that by creating the visual idea that all mods are to be "drag'n'drop" affairs means that it is likely that the modding tool itself will not allow cross-mod integration at a level currently allowed.
So if folder ONE and folder TWO have the same item, and folder ONE is higher up on the list in the config file, then folder ONE's data get's used first.
I'd say, just try to avoid duplicate files anyway, like you would presently do.
I think I wasn't clear enough on my point; having a set of user-defineable subfolders for each mod makes sense only if the mods are operating completely independently. If the intent is to do the equivalent of a code merge, then having sub-folder overrides is not a good idea. If the user can remove a patch or a mod independently, without referent to the dependencies (or if the structure looks like they can) then unless it is a simple texture mod, modders have significant challenges to overcome. The tool creator has another - now they have to parse the .ini for any added subfolders and look within those folders for content to modify. And then put the result in the master override.
Modding on Mount & Blade and many other modern modding tools requires that people build modules out of other people's code and then make them work together that way; you have to distribute other people's "submods" in order to get the merged mod to work. The same applies to Morrowind and Oblivion, anfd a host of other games. Currently, our way of adding content to BG/BG2 is massively more powerful. Everyone can create their own modification independently, working from the source file, and the installation tool helps determine and act on dependencies, creating a newly recoded game. I don't see customizable and added override/sub-override helping modders work together - it looks like a move towards independently constructed plugins placed in hierarchical order. That is a step backwards, not forwards. I don't think adding 4 NPCs at once, for example, all working with the same dialog.tlk, is going to be helped in this kind of setup.
(Someone has to clear up weidu compatibility and modding tool usage soon, please, because if we are in a toolset that does not allow mods to look for each other's content and work together, most of my points are moot. I don't want to waste people's time with referent to what we do now if the groundrules are to significantly change.)
Also, modders like to mod other modders work, heh... so, there you have it
The difference is solely in what instructs the engine on precedent/installation-order; the 'hard' file-system structure or a 'soft' runtime-alterable configuration. In memory at runtime, they wind up exactly the same: it's only prior to loading the game that there is any difference. So how one solution can be more flawed than the other has me at a loss.
This isn't my area of expertise (I'm a programmer, but I've never dealt with mods), but nobody has made a convincing argument to me why my idea would fail. I was keeping in mind compatibility clashes from the start when I presented the problem, in fact, it was intended to avert them.
ie: Mod 1 alters the Aerie dlg
Mod 2 alter the Jahiera dlg, but also includes an interjection by Aerie.
Mod 2 won't be able to effectively find the Aerie.dlg to insert the interjection because it wont know which folder to look in without guidance... so the Aerie interjection will not be included in Mod 2. If the Aerie interjection had actions that led to other interjections or plot detail or other actions, none of that would be passed on effectively breaking Mod 2.
As a programmer, if you have a solution to this problem by indexing the folder structure somehow and passing that info on to WeiDU, then everything should be fine. Or, update WeiDU to include that search through the folders. Either way, but mods old and new to re-release is not going to be a realistic option. Just saying.
Two things immediately occur to mind:
- This idea is meant to replace weidu--it 'does it all automatically' but via a run-time 'soft installation' procedure that does the effective equivilent of what weidu does, but in RAM rather than to the file-system, for much swifter manipulation and sorting of the mods. Internally these file-targets have indirection, so large amounts of data isn't being shifted around, in RAM or otherwise.
- A single 'mod-compatibility' description file can instruct the engine on where mods that predate the feature should seek a given file. Shouldn't be that hard to construct it for someone who knows the mod scene well.
A quick look tells me aeriep.dlg and aeriej.dlg are in the override folder? I'm not sure what you mean by 'the mod finding'... if the game ships with a file, the game will know where to find it due to it being written into a table that also ships with the game. It's the engine that needs to find it, not the mod.My past experience with mods is that you eventually get a certain number of them and once you've got it all stabilized you're afraid to touch anything for fear of breaking it, if only because doing the uninstalling and reinstalling via weidu is pretty slow (or it was, when I used it?). This idea is meant to overcome that, and not just at the user-end; modders can test and experiment with compatibility without restraint.
Mod 1 and Mod 2 would be wherever you felt like putting them (let's say <installation-directory>/mod1 and <installation-directory>/mod2). (Part of the intention behind my idea is to do away with installation in the conventional sense--that you can just drag & drop the mod where you like.) The engine would be instructed at runtime by the user via an interface (or outside of it by hard-editing the config file, same difference) about the existence of mod1 and mod2, and on the precedence.
Anytime the game engine attempts to load a file, it will fall through the mods by precedence 1st seeking it in any of those directories (caching the result). The 'soft-installation' isn't even installation at all, really, it's solely about manipulating directory targets for a file-structure independent solution. However, it does use the file-structure within a targetted directory to inform it of where the file would equivilently be if directly overwriting the override folder--maybe that wasn't implicit in what I wrote before...
Another possible confusion may be due to my assumption that mods only affect files by overwriting them, which I have been assuming all of this time... so a question is in order: does weidu support the patching in of file differences as well? I may take time out to educate myself by downloading a mod that requires weidu and see how it operates, if it is in aid of this discussion. I'll see what you respond with first, though.
@caruga Yes, WeiDU can patch anything in the game provided the offsets are known. It isnt just for dlg, but that has been it's primary use until IA and Widescreen mod came around. I was using it to patch items, spells, areas/wed files, IDS, 2da, exe, etc. It can do it all. I was very reluctant to use it early on when all mods were drag/drop based or IEEP based... but I saw the light finally.
I am still not the one to answer these questions, but I will try until someone can get the_bigg over here to see this. He may understand everything you are talking about. @cmorgan can also add to what I have said (or correct me!) My main area of expertise in the IE are areas, items and creatures (and scripts to make them work). I just use the tools I have available.
I don't know. If I read you right, the basic idea is to create a new tool that creates an interactive .exe in RAM at runtime which seeks out and adds content from multiple mods; in i.e. modding terms, a hugely interactive version of TobEX that can also look at all the potential mods found in a customizeable location and take them into account/patch them into the game. Do I have this right?
I think you are looking for something similar to a souped-up Wrye mash
http://www.uesp.net/wiki/Tes3Mod:Wrye_Mash
with mods operating like Morrowind/Oblivion Plugins.
the bigg and @Ascension64 and @Wisp and @Taimon would be the people I would recommend asking about something like what you are describing, if I do have it right. Nythrun and devSin, too, but I don't know if they are fully retired and completely unavailable.
I can answer some of what you are asking:
[*] [quote]A quick look tells me aeriep.dlg and aeriej.dlg are in the override folder?[/quote]
Only if they have been modified. The originals are in the compressed game files, and located via the chitin.key file. So it means loading, parsing, and messing about with that file. WeiDU does it, so it is definitely doable. Some of these files were fixed and patched by the BioWare team moving repaired versions into override, which is why they are there and not everything, and also why we can mod - and brings us to
[*] [quote] assumption that mods only affect files by overwriting them[/quote] In essense, for most of mod operations, I think this is correct, because very little actually changes the original files - the changes are made to copies of the data extracted from the compressed data files and placed in override. WeidDU can make a backup of the original file in the individual mod's subdirectory, and when the mod is uninstalled, roll everything back one layer at a time until the mod's backup is the one restored to override, then roll forward again - it works linearly on a mod stack to make sure everything can be reversed. But you may need better modders than I to confirm. I do know that the .exe hacks operate on the original game files, and are patching the .exe directly.
[*] [quote]Part of the intention behind my idea is to do away with installation in the conventional sense--that you can just drag & drop the mod where you like[/quote]
It is a cool idea - but I just have to say, giving that kind of freedom comes at one heck of a price.
How does a modder troubleshoot something that can be so open? How do we figure out if the problem is because a user has permissions problems on a folder? How do we see what the runtime has as an actual installed order, and what things are interacting incorrectly? Current tools fail or throw warnings step by step. That is how we troubleshoot, especially troubleshooting inter-mod things. Most of all, we build chains of choice:
1 = IF mod A component #102 is installed THEN [do X GOTO 2] ELSE [do Y GOTO 3] ELSE FAIL
2 = IF Original Content D SAYS Z THEN [do X GOTO 3] ELSE [install new content] ELSE FAIL
etc.
in order to take other folk's work into account. And let's face it - players want more mods working together, rather than less. So inter-mod fixing is both the most player-demanded function and the hardest thing to achieve.
The biggest deal here is referencing the file that holds all the text strings. WeiDU installations map .dlg strings that already exist and add new strings that don't. It sounds like a nightmare to script for an interactive runtime, exponentially getting more complicated as more mods are thrown into the mix. The "One step at a time" approach, building the stairs on what the installer sees each time you add a mod is very conservative - which is a great thing for troubleshooting.
On the positive side of your idea, there are current .bat file install orders that can be placed into an install; the weidu docs have some examples, I think, and the Big World Project has some tools that create install order lists and scanning for existing mods. You could take a look at their work and see what they say about creating this sort of tool, and what they ran into as troubles/advantages.
I can sort out a few of the challenges the tool would need to overcome:
[*] What happens to the game when a user removes a subdirectory and then starts a savegame without the information available at runtime? Currently, scrambled strings are the primary bug report, were a user uninstalls a mod and then uses a savegame that expects the mod is in place. So now we are talking about dynamically reordering and editing the savegames, as well, I think, or some way of marking saves as inaccessible if they don't match up - or recreating the dialog.tlk each and every runtime, and then altering the savegames to match. This would slow down the number of folks requesting technical support to try to resolve PEBCAK. But the time and care involved in that... wow. Lots of stuff to deal with. Way above my paygrade to even figure out if that kind of interactivity is reliable enough to put in the field.
[*] With customizeable subdirectories, what sort of troubleshooting output could be made available regarding the specifics of the install and the files being modified at runtime?
And now I should scoot and let other people talk; I have blabbed about too much these past few days. My biggest concerns are already on the table. Again, I am not shooting the general concept down - I am trying to show some of the challenges that modders would want addressed if something like this were implemented.