Skip to content

Override Folder

CameronToferCameronTofer Administrator, Developer Posts: 279
edited June 2012 in Fixed
So, we have the Override folder in BG1, BG2, IWD, etc. I'm glad it's there. But would it be possible to create and utilize Override sub folders? This way we might keep additions players want to add more organized?
A simple suggestion would be folders called "Patch" with a subsequent number following: "Patch001"

http://www.reddit.com/r/FriendsOfBaldursGate/comments/u52l4/request_override_folder/
Post edited by CameronTofer on
AndreaColomboBhryaenChnapyAssPopperg314ShapiroKeatsDarkMage
«1

Comments

  • BhryaenBhryaen Member Posts: 2,874
    edited June 2012
    @CameronTofer They'll never go for it. It's a great idea, but just a pipe dream. *dramatic sulk*

    Actually other games manage to navigate subfolders with no trouble somehow, letting you organize and name all your folders as you like since it only looks for the heading "override" in the proper directory and presumably ignores all but the files and their type. DAO definitely does, but I'm not sure what process they're employing. Not sure either what learning process is going to be needed in the Patch folder. Does this mean we'd put our mod "Mega Cheat Game Breaker" into a folder called "Patch001"? Then we'd have to remember that MCGB is associated with 001, but on a different install it might then be in Patch003 given a new install order that puts mods "TrentONPC" and "Add Machete Proficiency" first... Tell me it's not like that...
  • CameronToferCameronTofer Administrator, Developer Posts: 279
    We've implemented as a list of folders in your .ini, so for example:

    [Overrides]
    myoverride=c:\somefolder
    supermod=c:\supermod\files


    The take precedent in the order in the file. As well you can specify them at the command line:

    for example:
    BGMain.exe --set OVERRIDES mymod c:\mymodfiles\
    caruga
  • BoasterBoaster Member Posts: 622
    edited June 2012
    Cameron, I believe the best way to implement override subfolders would simply to change how you guys are planning to implement the feature request.

    Instead of pointing to a folder outside of the Override folder, it should look more like so:

    [Overrides]
    PLUSmod=mods\PLUS\
    Profmod=mods\Proficiency\

    Simply point to the folders inside the Override folder. Making the custom overrides sub-directories outside of the override folder may just make it confusing for some.

    No need to point to extraneous folders such as: C:\users\Boaster\Documents and Settings\Application Data\BG1EE\mods\PLUS

    Keep it super simple.

    And I agree, order taking precedence is a good feature!
  • CameronToferCameronTofer Administrator, Developer Posts: 279
    In that case I may move the override root into some other folder that's write-able since users may not have permission to modify the application folder. Really the installed game shouldn't be touched.
  • DeeDee Member Posts: 10,447
    I think that would be best overall - it does bug me that certain mods can't be uninstalled without completely reinstalling my game. A cleaner "BGMod" directory would solve that problem.
  • BoasterBoaster Member Posts: 622
    edited June 2012
    In that case I may move the override root into some other folder that's write-able since users may not have permission to modify the application folder. Really the installed game shouldn't be touched.
    Just curious, but why would you be worried about someone not having permissions to modify a directory? Surely, if someone is quite capable of modding a game (either the maker or the end-user of the mod) they would be able to have write-access to the folder.

    For example, this would be my directory architecture leading up to the Override folder.
    "D:\games\Black Isle\BG1EE\override\"
    What I would do, and perhaps it's just me, is that if I were to make a mod for BG1 or BG2 (which I have, though not extensive as some) would be to organize any such mods I install.

    Inside the "override" folder I would create a folder called "mods" and anything that's in the root directory of the Override folder I, personally, would classify this as "patch" data or "from the makers."

    Then, inside the mods folder (D:\games\Black Isle\BG1EE\override\mods\) I would then create a folder for each mod I would install to keep things separate.
    D:\games\Black Isle\BG1EE\override\mods\PLUS\
    D:\games\Black Isle\BG1EE\override\mods\Proficiency\
    D:\games\Black Isle\BG1EE\override\mods\HardcoreMonsters\
    D:\games\Black Isle\BG1EE\override\mods\Samurai_Kit\
    In my humble opinion, it is organized and simple. And to be able to just simply point to the folder, as mentioned previously...:
    [Overrides]
    PLUSmod=mods\PLUS\
    Profmod=mods\Proficiency\
    ...would simply work best (in my opinion).

    Simplicity and organization. Anyone else have thoughts on this? Or would people just defer to how I stated things?
  • BhryaenBhryaen Member Posts: 2,874
    Why isn't it possible to make it like in DAO where there's like no structure at all, just an override folder where the game looks and every option available to organize it with subfolders the way you like? I think NWN2 has this also, though I haven't played it. Is it a matter of Infinity engine limitations?
  • DeeDee Member Posts: 10,447
    @Boaster: I think it's more about not wanting mods to feel like hacking; they should be a seamless integration with the original game, so that players can use them without feeling like they're "breaking" something.
  • BoasterBoaster Member Posts: 622
    A mod is, in a sense, a hack. So I don't think that would be it.
  • KerozevokKerozevok Member Posts: 695
    Currently Near infinity, DLTCEP & Weidu can't check subfolders in override.

    Moreover, with several folders you can have the same file in several places and the same file can have several descriptions in tlk... If you want to make the life of the modder a hell, it's a good idea... Otherwise it's a very bad idea.
  • BoasterBoaster Member Posts: 622
    Cameron was saying how the top listed sub-folder would have priority.
  • smeagolheartsmeagolheart Member Posts: 7,963
    Anything else implemented?
  • carugacaruga Member Posts: 375
    edited June 2012
    Here's how I think modding should work, here (post #15 since the jump table doesn't work). Might be a difficult-to-implement solution, though.
    Post edited by caruga on
  • LuneverLunever Member Posts: 307
    Well, originally there was just the quick and dirty general "override" folder. That's certainly better than changing the actualy game install, but still - if 2 differenty patches/mods wanted to override the same file - without one of them already including the changes of the other - one would undo the other. Much cleaner was the approach of some modpacks, that didn't simply overrided files, but actually created a CHANGED version of the previous file in file the override folder. Still, having each modification in a separate sub-folder, will make it a lot easier to undo changes of a game. MOD1 creates a changed FILE0001 and works fine, MOD2 creates a changed FILE0001 too but has a bug in its uninstaller? NP, if the changed files are stored in different override-subfolders you can undo the change simply manuallly.
  • cmorgancmorgan Member Posts: 707
    WeiDU has a huge advantage over the modding tool override "organizers". It leverages having one override by leaving the original game files untouched, lets a modder make a change, backs up the old one, and puts the new file in the override.

    This looks and sounds chaotic. On the surface, you can't tell what mod's changes are taking precedence. Visually, it looks less organized.

    BUT - the WeiDU.log contains the details, and weidu itself can tell you exactly what has touched what file in what order.

    WeiDU does not touch the original installation; if you want a vanilla game, delete your override folder and pop the vanilla one back in place. Voila.

    AND here comes the smashing great huge blow that makes most "modern" modding tools and systems subpar at the very best...

    ...every mod can add to the same file, manipulate it, change it, all at once, on the same install, at the same time.

    With an override hierarchy, you have juggling balls. Whichever one is highest takes precedence, and everything else vanishes from existence. Removing specific things manually *looks* easy from an end-user standpoint, but it is very scary for things more intricate than graphics manipulation. Look at the hades it is to create multiple mod collaboration on any number of modern games.

    With a layered approach like WeiDu uses, the game keeps track of the changes, allowing content to be added like the rings of an onion. Actually, it is more like a ball of string - the original content is a few unwinds away, and every mod added can check for dependencies all the way back down the string to the original source file, unwind to whatever point it needs to, work in new content, and reroll the ball of string. If I wanted length 1.5cm to 2cm to be blue, I check it, and if it is already blue, I move on - if not, I change it blue.

    As far as priority orders, having more places to stuff things into is not in itself bad. The problem is that by creating the visual idea that all mods are to be "drag'n'drop" affairs means that it is likely that the modding tool itself will not allow cross-mod integration at a level currently allowed.
    smeagolheartAndreaColomboLuneverGalactygon
  • BoasterBoaster Member Posts: 622
    Priority of files will be given to the highest rank folder, according to what Cameron was saying.

    So if folder ONE and folder TWO have the same item, and folder ONE is higher up on the list in the config file, then folder ONE's data get's used first.

    I'd say, just try to avoid duplicate files anyway, like you would presently do.
  • cmorgancmorgan Member Posts: 707
    Yes, the priority order of folders makes sense. The current game structure uses a heirarchy as well, with /portraits/ acting as a sub-override.

    I think I wasn't clear enough on my point; having a set of user-defineable subfolders for each mod makes sense only if the mods are operating completely independently. If the intent is to do the equivalent of a code merge, then having sub-folder overrides is not a good idea. If the user can remove a patch or a mod independently, without referent to the dependencies (or if the structure looks like they can) then unless it is a simple texture mod, modders have significant challenges to overcome. The tool creator has another - now they have to parse the .ini for any added subfolders and look within those folders for content to modify. And then put the result in the master override.

    Modding on Mount & Blade and many other modern modding tools requires that people build modules out of other people's code and then make them work together that way; you have to distribute other people's "submods" in order to get the merged mod to work. The same applies to Morrowind and Oblivion, anfd a host of other games. Currently, our way of adding content to BG/BG2 is massively more powerful. Everyone can create their own modification independently, working from the source file, and the installation tool helps determine and act on dependencies, creating a newly recoded game. I don't see customizable and added override/sub-override helping modders work together - it looks like a move towards independently constructed plugins placed in hierarchical order. That is a step backwards, not forwards. I don't think adding 4 NPCs at once, for example, all working with the same dialog.tlk, is going to be helped in this kind of setup.

    (Someone has to clear up weidu compatibility and modding tool usage soon, please, because if we are in a toolset that does not allow mods to look for each other's content and work together, most of my points are moot. I don't want to waste people's time with referent to what we do now if the groundrules are to significantly change.)

    CuvAndreaColomboDiscoCat
  • CuvCuv Member, Developer Posts: 2,535
    I don't see customizable and added override/sub-override helping modders work together - it looks like a move towards independently constructed plugins placed in hierarchical order. That is a step backwards, not forwards.
    I agree, this needs to be cleared up exactly what is planned. The folder structure could potentially break most mods where more than one is installed at a time.

    AndreaColombo
  • AndreaColomboAndreaColombo Member Posts: 5,524
    I invoke the mighty @CameronTofer to interact with modders and work out a common solution :-)
  • carugacaruga Member Posts: 375
    Looks like the link to my idea was broken all of this time. Fixed. I've only had 1 response so far and it was negative but I'd be interested in hearing what's wrong with a 'file-structure agnostic' solution.
  • CuvCuv Member, Developer Posts: 2,535
    Nothing at all for any stand-alone mods... the problems arise when more than one mod changes files, this is most common with NPC mods, but happens elsewhere too.

    Also, modders like to mod other modders work, heh... so, there you have it
  • carugacaruga Member Posts: 375
    edited June 2012
    Nothing at all for any stand-alone mods... the problems arise when more than one mod changes files, this is most common with NPC mods, but happens elsewhere too.
    That's the same argument put to me on the beamdog forum... but I'm not seeing it.

    The difference is solely in what instructs the engine on precedent/installation-order; the 'hard' file-system structure or a 'soft' runtime-alterable configuration. In memory at runtime, they wind up exactly the same: it's only prior to loading the game that there is any difference. So how one solution can be more flawed than the other has me at a loss.

    This isn't my area of expertise (I'm a programmer, but I've never dealt with mods), but nobody has made a convincing argument to me why my idea would fail. I was keeping in mind compatibility clashes from the start when I presented the problem, in fact, it was intended to avert them.
  • CuvCuv Member, Developer Posts: 2,535

    The difference is solely in what instructs the engine on precedent/installation-order; the 'hard' file-system structure or a 'soft' runtime-alterable configuration.
    Oh! I see the confusion. What will cause the problems are how the mods look for the files via the WeiDU exe and TP2. If the folder structure changes, then all mods will have to be updated to point to the correct paths. And that isnt going to happen for many mods, the modders have moved on and there has been no reason to revisit installation or compatibility since WeiDU came on the scene... its all done automatically. It's how WeiDU looks for other mods that may share the same files.

    ie: Mod 1 alters the Aerie dlg
    Mod 2 alter the Jahiera dlg, but also includes an interjection by Aerie.

    Mod 2 won't be able to effectively find the Aerie.dlg to insert the interjection because it wont know which folder to look in without guidance... so the Aerie interjection will not be included in Mod 2. If the Aerie interjection had actions that led to other interjections or plot detail or other actions, none of that would be passed on effectively breaking Mod 2.

    As a programmer, if you have a solution to this problem by indexing the folder structure somehow and passing that info on to WeiDU, then everything should be fine. Or, update WeiDU to include that search through the folders. Either way, but mods old and new to re-release is not going to be a realistic option. Just saying.

  • AndreaColomboAndreaColombo Member Posts: 5,524
    @Cuv - isn't the_bigg taking care of WeiDU updates? He's always struck me as an experienced programmer, so if a solution like the one you're suggesting (i.e. updating WeiDU to include search through folders) is actually feasible, then the_bigg most likely knows how to implement it.
  • carugacaruga Member Posts: 375
    edited June 2012

    The difference is solely in what instructs the engine on precedent/installation-order; the 'hard' file-system structure or a 'soft' runtime-alterable configuration.
    Oh! I see the confusion. What will cause the problems are how the mods look for the files via the WeiDU exe and TP2. If the folder structure changes, then all mods will have to be updated to point to the correct paths. And that isnt going to happen for many mods, the modders have moved on and there has been no reason to revisit installation or compatibility since WeiDU came on the scene... its all done automatically. It's how WeiDU looks for other mods that may share the same files.
    We must be cross-communicating still and not having the same things in mind. My back of my mind tells me that I may be being an ignorant idiot, here, and that I need to educate myself on how the mods work and what weidu actually does... but let's keep communicating until we see who is at fault. :) I haven't used mods or weidu for 4 years or more so it may have evolved since then...

    Two things immediately occur to mind:
    1. This idea is meant to replace weidu--it 'does it all automatically' but via a run-time 'soft installation' procedure that does the effective equivilent of what weidu does, but in RAM rather than to the file-system, for much swifter manipulation and sorting of the mods. Internally these file-targets have indirection, so large amounts of data isn't being shifted around, in RAM or otherwise.
      My past experience with mods is that you eventually get a certain number of them and once you've got it all stabilized you're afraid to touch anything for fear of breaking it, if only because doing the uninstalling and reinstalling via weidu is pretty slow (or it was, when I used it?). This idea is meant to overcome that, and not just at the user-end; modders can test and experiment with compatibility without restraint.
    2. A single 'mod-compatibility' description file can instruct the engine on where mods that predate the feature should seek a given file. Shouldn't be that hard to construct it for someone who knows the mod scene well.

    ie: Mod 1 alters the Aerie dlg
    Mod 2 alter the Jahiera dlg, but also includes an interjection by Aerie.

    Mod 2 won't be able to effectively find the Aerie.dlg to insert the interjection because it wont know which folder to look in without guidance... so the Aerie interjection will not be included in Mod 2.
    A quick look tells me aeriep.dlg and aeriej.dlg are in the override folder? I'm not sure what you mean by 'the mod finding'... if the game ships with a file, the game will know where to find it due to it being written into a table that also ships with the game. It's the engine that needs to find it, not the mod.

    Mod 1 and Mod 2 would be wherever you felt like putting them (let's say <installation-directory>/mod1 and <installation-directory>/mod2). (Part of the intention behind my idea is to do away with installation in the conventional sense--that you can just drag & drop the mod where you like.) The engine would be instructed at runtime by the user via an interface (or outside of it by hard-editing the config file, same difference) about the existence of mod1 and mod2, and on the precedence.

    Anytime the game engine attempts to load a file, it will fall through the mods by precedence 1st seeking it in any of those directories (caching the result). The 'soft-installation' isn't even installation at all, really, it's solely about manipulating directory targets for a file-structure independent solution. However, it does use the file-structure within a targetted directory to inform it of where the file would equivilently be if directly overwriting the override folder--maybe that wasn't implicit in what I wrote before...

    Another possible confusion may be due to my assumption that mods only affect files by overwriting them, which I have been assuming all of this time... so a question is in order: does weidu support the patching in of file differences as well?
    If the Aerie interjection had actions that led to other interjections or plot detail or other actions, none of that would be passed on effectively breaking Mod 2.

    As a programmer, if you have a solution to this problem by indexing the folder structure somehow and passing that info on to WeiDU, then everything should be fine. Or, update WeiDU to include that search through the folders.
    I may take time out to educate myself by downloading a mod that requires weidu and see how it operates, if it is in aid of this discussion. I'll see what you respond with first, though.
  • CuvCuv Member, Developer Posts: 2,535
    @Cuv - isn't the_bigg taking care of WeiDU updates? He's always struck me as an experienced programmer, so if a solution like the one you're suggesting (i.e. updating WeiDU to include search through folders) is actually feasible, then the_bigg most likely knows how to implement it.
    Yes, He would be the one to know ALL the ins and outs of WeiDU. He would also be the one to know what needs to be done here with regards to keeping mods useable and compatible. He is still very active, but he doesnt like to be bothered unless its imperative. I would say that is fall under that category. We worked together on projects, he is very knowledgeable!!!!!

    @caruga Yes, WeiDU can patch anything in the game provided the offsets are known. It isnt just for dlg, but that has been it's primary use until IA and Widescreen mod came around. I was using it to patch items, spells, areas/wed files, IDS, 2da, exe, etc. It can do it all. I was very reluctant to use it early on when all mods were drag/drop based or IEEP based... but I saw the light finally.

    I am still not the one to answer these questions, but I will try until someone can get the_bigg over here to see this. He may understand everything you are talking about. @cmorgan can also add to what I have said (or correct me!) My main area of expertise in the IE are areas, items and creatures (and scripts to make them work). I just use the tools I have available.

  • carugacaruga Member Posts: 375

    @caruga Yes, WeiDU can patch anything in the game provided the offsets are known. It isnt just for dlg, but that has been it's primary use until IA and Widescreen mod came around. I was using it to patch items, spells, areas/wed files, IDS, 2da, exe, etc. It can do it all. I was very reluctant to use it early on when all mods were drag/drop based or IEEP based... but I saw the light finally.

    Oh good. Well I had that written into my feature description as well that I wrote on the beamdog forum. The only disadvantage I can see to the feature is that it would take work to implement, and would make weidu redundant (it also cannot be replicated by weidu itself, it needs to be written into the engine--unless they make the EE engine extensible with DLL's and such). But I can see it saving modders and mod-users a fair amount of time.
  • cmorgancmorgan Member Posts: 707
    edited June 2012
    Well... hmmm. @Cuv, you are the experienced one - I just tag along with the big boyz and girlz.

    I don't know. If I read you right, the basic idea is to create a new tool that creates an interactive .exe in RAM at runtime which seeks out and adds content from multiple mods; in i.e. modding terms, a hugely interactive version of TobEX that can also look at all the potential mods found in a customizeable location and take them into account/patch them into the game. Do I have this right?

    I think you are looking for something similar to a souped-up Wrye mash
    http://www.uesp.net/wiki/Tes3Mod:Wrye_Mash
    with mods operating like Morrowind/Oblivion Plugins.

    the bigg and @Ascension64 and @Wisp and @Taimon would be the people I would recommend asking about something like what you are describing, if I do have it right. Nythrun and devSin, too, but I don't know if they are fully retired and completely unavailable.

    I can answer some of what you are asking:

    [*] [quote]A quick look tells me aeriep.dlg and aeriej.dlg are in the override folder?[/quote]
    Only if they have been modified. The originals are in the compressed game files, and located via the chitin.key file. So it means loading, parsing, and messing about with that file. WeiDU does it, so it is definitely doable. Some of these files were fixed and patched by the BioWare team moving repaired versions into override, which is why they are there and not everything, and also why we can mod - and brings us to

    [*] [quote] assumption that mods only affect files by overwriting them[/quote] In essense, for most of mod operations, I think this is correct, because very little actually changes the original files - the changes are made to copies of the data extracted from the compressed data files and placed in override. WeidDU can make a backup of the original file in the individual mod's subdirectory, and when the mod is uninstalled, roll everything back one layer at a time until the mod's backup is the one restored to override, then roll forward again - it works linearly on a mod stack to make sure everything can be reversed. But you may need better modders than I to confirm. I do know that the .exe hacks operate on the original game files, and are patching the .exe directly.

    [*] [quote]Part of the intention behind my idea is to do away with installation in the conventional sense--that you can just drag & drop the mod where you like[/quote]
    It is a cool idea - but I just have to say, giving that kind of freedom comes at one heck of a price.

    How does a modder troubleshoot something that can be so open? How do we figure out if the problem is because a user has permissions problems on a folder? How do we see what the runtime has as an actual installed order, and what things are interacting incorrectly? Current tools fail or throw warnings step by step. That is how we troubleshoot, especially troubleshooting inter-mod things. Most of all, we build chains of choice:

    1 = IF mod A component #102 is installed THEN [do X GOTO 2] ELSE [do Y GOTO 3] ELSE FAIL
    2 = IF Original Content D SAYS Z THEN [do X GOTO 3] ELSE [install new content] ELSE FAIL
    etc.

    in order to take other folk's work into account. And let's face it - players want more mods working together, rather than less. So inter-mod fixing is both the most player-demanded function and the hardest thing to achieve.

    The biggest deal here is referencing the file that holds all the text strings. WeiDU installations map .dlg strings that already exist and add new strings that don't. It sounds like a nightmare to script for an interactive runtime, exponentially getting more complicated as more mods are thrown into the mix. The "One step at a time" approach, building the stairs on what the installer sees each time you add a mod is very conservative - which is a great thing for troubleshooting.

    On the positive side of your idea, there are current .bat file install orders that can be placed into an install; the weidu docs have some examples, I think, and the Big World Project has some tools that create install order lists and scanning for existing mods. You could take a look at their work and see what they say about creating this sort of tool, and what they ran into as troubles/advantages.

    I can sort out a few of the challenges the tool would need to overcome:
    [*] What happens to the game when a user removes a subdirectory and then starts a savegame without the information available at runtime? Currently, scrambled strings are the primary bug report, were a user uninstalls a mod and then uses a savegame that expects the mod is in place. So now we are talking about dynamically reordering and editing the savegames, as well, I think, or some way of marking saves as inaccessible if they don't match up - or recreating the dialog.tlk each and every runtime, and then altering the savegames to match. This would slow down the number of folks requesting technical support to try to resolve PEBCAK. But the time and care involved in that... wow. Lots of stuff to deal with. Way above my paygrade to even figure out if that kind of interactivity is reliable enough to put in the field.
    [*] With customizeable subdirectories, what sort of troubleshooting output could be made available regarding the specifics of the install and the files being modified at runtime?


    And now I should scoot and let other people talk; I have blabbed about too much these past few days. My biggest concerns are already on the table. Again, I am not shooting the general concept down - I am trying to show some of the challenges that modders would want addressed if something like this were implemented.
    CuvAndreaColombo
  • AndreaColomboAndreaColombo Member Posts: 5,524
    edited June 2012
    That's it, I'm asking Wisp to join these boards. His name has been called out invain too many times. Hopefully, he'll soon be able to join this discussion (don't worry, we're working on getting the_bigg here too) ;-)
    looter
Sign In or Register to comment.