modmerge: merge your Steam/GOG zip based DLC into something Weidu/NearInfinity/DLTCEP can use.

ScottBrooks

Hey,

Updated to 1.1 to look for sod-dlc.zip where GOG puts it.

People have certainly made their voices heard about the zip based dlc that we use on Steam and Gog. I had a few hours to spare Saturday and Sunday night, so I made modmerge for those who may be interested.

You can download binaries for your platform at http://github.com/ScottBrooks/modmerge/releases or view the source at http://github.com/ScottBrooks/modmerge

This comes with no warranty, and if you need to restore your steam copy to normal, you can use the 'Properties->Local Files->Verify Integrity of Game Cache' feature to restore things to how they were before.

If you download this and place it beside your game data folder(aka, modmerge.exe should be sitting beside chitin.key), and run it, it will do the following.

1. Backs up your chitin.key to chitin.key.bak, just in case
   
2. Unzips your dlc/sod-dlc.zip, copying over top of the existing lang/, movies/, music/ folders. The data folder inside of the zip will be renamed to no overwrite files in the existing data directory.
   
3. Updates your chitin.key with so any resources from inside of the zip file will point to their appropriate bif files.
   
4. Renames your dlc/sod-dlc.zip to dlc/sod-dlc.disabled so the engine no longer loads the zip based dlc.
   

This should give you a chitin.key that references all the BGEE assets in data/*.bif, along with referencing all the SOD assets in the sod-dlc/*.bif files.

I've tested this on two machines here(PC/Mac), on steam. Let me know if you have any success.

@elminster 's edit: Mac users should make sure the "modmerge-osx" file they download is renamed as "modmerge" before trying the above steps.

Thels

Nice!

There's some reports though, that the GoG version is in the directory with the chitin.key file, rather than in the DLC subdirectory. I don't have GoG myself, so I can't verify, but if any GoG users check this thread, please check where the zip file is located and confirm or deny this!

Horg

I tested this on a fresh GoG install. The sod-dlc.zip file is in the root directory, so i moved it to a dlc folder, and it worked.
I ran BG1EE with NPC Project, tested some conversations for 10 minutes and i didn't find any problems. I didn't test Dragonspear though.

Thels

Nice to know it runs smoothly on GoG as well. Mods seem to install properly on the Steam version too.

Hopefully @ScottBrooks can make it GoG compatible with a little tweak, without requiring users to move their zip file?

AstroBryGuy

@ScottBrooks - You deserve the Bag of Everlasting s!

argent77

It worked flawlessly on my test installation (Windows 7, Steam). To support GOG and potential future expansions or user-made DLCs I'd like to suggest expanding this tool to scan supported DLC paths for compatible zip archives and merge all of them automatically.

Thels

I've tested this (Windows 10, Steam) with a couple of mods, such as BG1 NPC, BG1 UB and Tweaks Anthology, and it seemed to work fine, both during installation, and having stuff pop up in game.

Only the glowy shields from the Tweaks pack didn't install for me, but since that's a new component of that pack, it may be a bug in the component itself.

Achievements still work.

shawne

Oh, well done!

Swifty_Magee

Modmerge seems to install all my mods except soundsets using the Awesome Soundset mod (https://forums.beamdog.com/discussion/38319/mods-awesome-soundsets-mods-master-thread-custom-soundsets-for-bgee-bg2ee-and-iwdee/p1).

When I try to install soundsets using the mod I get the following error:

Cannot append column-wise because there are 48 lines in CHARSND.2da but I was only given 28 things to append

ERROR: Failure("cannot append column-wise to CHARSND.2da")

Other gameplay mods installed fine and seem to work correctly in-game.

Thels

@Swifty_Magee: I've tried to run that mod on the Beamdog version, and it caused the same problems. Ergo, it's unrelated to the Steam/GoG packaging.

The mod is simply not 2.0 compatible, and needs to be updated.

The glowy shield thingy that I mentioned in my previous post also doesn't load on the Beamdog version.

Swifty_Magee

@Thels , actually, I remember installing my Awesome Soundset mods fine before and after installing Siege of Dragonspear. The voice clips and subtitles ran correctly in BGEE 2.0. There were subtitle issues after installing Siege of Dragonspear.

Before I used Modmerge, the only problem was that the subtitles for the soundsets were showing incorrect text. After using Modmerge, I can't install the soundsets at all.

Maybe the Awesome Soundset mod has issues with the Beamdog version of the Infinity Engine games, but this is the first I'm hearing of it. Point is, the 2.0 patch didn't seem to break the mod; Siege of Dragonspear messed up the subtitles, and it looks like Modmerge broke it completely.

Thels

If the mod doesn't install properly on the Beamdog version, then that's an issue with the mod, and it needs to be made compatible.

This tool has one job, making the Steam/GoG installation similar to the Beamdog installation for mod compatibility. So far, it seems to do an excellent job at that. It's not designed to make your mod work, if your mod doesn't work with the Beamdog version in the first place.

Swifty_Magee

@Thels , I just wanted to be sure that the problem is with the non-Beamdog versions of the games. I use the Steam version, which has never had problems running the Awesome Soundset mod. And as I said before, you are the first person I've heard of that has had issues running the mod with the Beamdog version. This incompatibility issue is all new to me; I never would have known about it if I didn't try modmerge, hence why I thought it was the problem.

I've posted in the Awesome Soundset thread about the issue and informed the mod maker, Smeagolheart, about the problem. Hopefully he looks into it and can find a solution.

Thels

2.0 did change some things here and there, and the SoD installation did change some things on top of that. It could very well be that SoD expanded on the charsnd.2da file, and now your mod has no idea how to handle that.

If you were able to install it post-SoD before, that was because you weren't installing it on Sod, but on the original files.

Swifty_Magee

@Thels, you're right; like I said, the Awesome soundsets can install pre and post 2.0 patch, at least on the Steam/GOG versions. The subtitles get screwy after installing SoD, but I realize that was because SoD data is compiled in a zip in the Steam/GOG versions.

I don't know why there are multiple versions of the games to begin with, honestly. Why have a Beamdog version and a Steam/GOG version? Wouldn't it make more sense to just make one version and distribute that one version to different outlets? Then we wouldn't have problems like these to begin with.

Anyway, apologies for any offenses, and I've reported this issue in the correct forum thread. Hopefully Smeagolheart can fix this and have the Awesome Soundset compatible with all versions of the Infinity Engine games.

Thels

I didn't take any offense, don't worry about it. I merely sought to fix your misunderstanding of the situation.

The zip file is an odd issue, and it's questionable why it's done like that. Considering the zip file seems to be identical for Steam and GoG, it's likely that that's how Beamdog presented the expansion set to both platforms. There's probably certain requirements for how DLCs are supposed to work on these platforms.

Ulb

Thank you for doing this on your free time, you are awesome!

Still disappointed in Beamdog though.
Unless I have misread your post you're not here as an official Beamdog representative but just as a private person. That means there is still no official answer.

I didn't expect an official statement during the weekend but I certainly expected one today. (Actually I would have expected BD to inform the community about this issue before release.)

Also, Beamdog should have let you do this on company time instead of just counting on someone to sacrifice personal time to fix issues caused by BD.

agris

@ScottBrooks worked beautifully for my Steam version. I uninstalled my non-text string-adding weidu mods, ran this, and re-installed them. No text errors in my saved game!

Thels

While I agree it would be nice if these zip problems weren't there in the first place, and also agree that this tool is something that is an absolutely mandatory inclusion for a modding-friendly development, and thus should probably have been created during work hours, rather than private hours, I personally think this is enough.

As long as people run this little tool before installing mods, they can install any mod that the beamdog users can, and honestly, those people that install mods will have to look around regardless. The larger mods will probably point towards this, and perhaps even Weidu could be designed to alarm the user if it finds the zip file.

(Though obviously, the GoG zip location should be added to the tool.)

Swifty_Magee

I do appreciate that Modmerge exists, as without it I wouldn't be able to play with any mods right now. I also think ScottBrooks should have been paid an honest wage for creating it, and it a forum thread should have been created at 2.0 launch day detailing why it's needed.

I just wish there wasn't a need for it in the first place. If everything was compiled uniformly to begin with, and there weren't slightly different versions of one game being sold at different places, there wouldn't be a need for the program.

It's just disappointing that all of this happened, I guess is what I'm trying to say. Needing a mod to install a mod just feels counterproductive and headache-inducing. Hopefully future patches and a WeiDU update resolves everything.

ALIEN

@ScottBrooks Can you change the destination folde of the .biff files to be \data instead of sod-dlc?

AWizardDidIt

THANK YOU FOR THIS.

that is all

ScottBrooks

I'm compensated appropriately for my job, lets try to keep the discussion about the tool.

I'll look into adding support for searching for the file where gog has placed it.

@ALIEN: Is there any real need to put the bif files in data? Some of them share names with the existing data/ biffs, so the files would need to be renamed either way. I would prefer to not adjust any of the files in the data folder.

woowoovoodoo

@ScottBrooks, sir, you're a savior.

ALIEN

@ScottBrooks Yes, the reason is: you tool allow mods to be installed and at the same time create "yet another release of thee game which has different directory structure" - that is not desire situation. The source of the problem was "different release" and adding yet another might cause problems in the future. We have seen similar problems with two Polish versions of the BG1/BG2 - using later, you cannot install mods which are modifying exe file due to different size.

Yet, indeed as you write, some files have the same name and different size compared to BG1EE-SoD release from Beamdog. I will check if you can skip it and if not, we need another solution or a quick statement from Beamdog that they will fix this soon.

megamike15

thanks for this.

now i'm curious about something. in the event of a patch would we have to reset the folder back to it's original state so it can take effect? or can we leave it as is?

Thels

It's very likely that you need to restore file integrity before updating to the next patch.

StefanO

Does anyone know who's to blame for the steam/gog zip based DLC? And why they did it at all?

Grammarsalad

Please do add gog support

Thels

@Grammarsalad: If you're on GoG, then you can use the tool as well, you just need to perform an extra step.

Go to the folder containing the chitin.key file.
Create a new subfolder called "dlc" (without the quotes).
Move the sod-dlc.zip file into this dlc subfolder (do not copy it).
Download this tool and save it in the folder containing the chitin.key file.
Run the tool.

Hopefully, ScottBrooks will update the mod so that you no longer need to move the zip file into a dlc folder, but until then, this should do the trick as well.

Ulb

StefanO said:

Does anyone know who's to blame for the steam/gog zip based DLC? And why they did it at all?

No because Beamdog apparently doesn't think this warrants an official statement.
Now that Scott has provided us with this tool (again, great work!) I guess Beamdog considers this issue "done".