Releasing the source code of the IE games

Alonso

As a completely new modder I am starting to understand how difficult it is to mod the Infinity Engine. In a recent discussion I suggested that it might be a good idea for Beamdog to release the source code of the Infinity Engine. There were several views on the idea, but my perception was that overall it would be quite beneficial for everybody: The company, the players and the modders. I found this post by @ALIEN very insightful (edited for clarity):

ALIEN wrote: »

Beamdog could use the same approach Id Software and John Carmack used for game series like Doom, Quake and Enemy Territory (before ZeniMax bought the studio): Releasing the source code as GPL, without changing the legal state of the game assets. That way anyone can distribute the game code but they can't distribute game files. Also because of the GPL, other companies wouldn't be allowed to release modifications without including the source code. This approach has allowed other gaming communities to expand the capabilities of their game engines, which has actually boosted the sales of the original games because the legal approach protects them against piracy.

By making the EE editions Beamdog gave the BG series a second life. Releasing the source code would make these games immortal.

I create this thread to discuss the idea further, and maybe propose it to Beamdog if it gets support from the community.

Edit: I imagine some people might not understand some of the concepts in this post, like GPL or code distribution. If so, just go ahead and ask, I'll try my best to give a detailed explanation of anything you need to know.

AndreaColombo

I’d love for the source code to be released but it’s owned by EA—the kind of people who would rather kill an IP than to give it to its fans.

Aquadrizzt

While I would love to be able to play with the source code, I think it's important that we accept that the chances we ever get access to it are slim to none (especially now that I know that it's owned by EA... yikes).

That being said, there are many projects that could stand in for having access to the source code. GemRB, while unrefined and kinda abandoned, would be a worthwhile project for those chomping at the bit of messing with the sourcecode. I'm not much of a Ruby guy myself (HAIL PYTHON), but we have an open source engine emulator, why not start with that instead of some pipe dream?

I also think that the one thing truly holding back IE modding (e.g. why there are so few custom campaigns) is the lack of a good area editor. I realize that the original games had hand drawn area art, so area art generated otherwise will look a little crunchy, but something is better than nothing right? Load it full of free assets plus textures extracted from the games and cleaned up, and I'm sure people could come up with some interesting stuff. I know there is that very rudimentary tileset-ish editor, but that's far from usable for any kind of sizable project.

Finally, while WeiDU is fun, perhaps a tool written in a slightly more... modern (?) ... readable (?) ... language would be a worthwhile investment? I started playing around with writing my own Python-based version of WeiDU (PyDU on my Github).

tl;dr Source code is likely unattainable, but there are many ways (some already well underway) that we could expand modding capabilities for the games in their current state.

Alonso

Very insightful stuff, @Aquadrizzt.

GemRB seems to be abandoned indeed, so it might be a dead end. While I fully appreciate the merit of WeiDU, I feel frustrated by its very chaotic and bizarre nature, an organized and systematic modding language would be great. What's the current state of PyDU?

Aquadrizzt

(I'm not trying to derail the thread, so admins feel free to move this to its own thread if you think its posing a problem.)

PyDU isn't "ready" by any means, but it is able to handle simple bitcode editing of existing files. The code below, for example, creates a copy of Accalia (named Accalia2.cre) with a metal colour of 155 rather than 30.

global keydata 
keydata = read_key('chitin.key')
Accalia = Resource('Accalia.cre')
Accalia.write_byte(0x2c,155)
Accalia.copy_as('Accalia2.cre')

The next thing I was working on for it before GRE prep and grad school applications consumed my life was having it innately know what the various offsets meant. Instead of saying "change byte 0x273 to 7", you could just say "creature.class = 'Fighter/Mage'", and PyDU would handle the rest. Overarching goal was something that did everything WeiDU did, but with an emphasis on readability and more modern syntax.

OlvynChuru

Aquadrizzt wrote: »

The next thing I was working on for it before GRE prep and grad school applications consumed my life was having it innately know what the various offsets meant. Instead of saying "change byte 0x273 to 7", you could just say "creature.class = 'Fighter/Mage'", and PyDU would handle the rest. Overarching goal was something that did everything WeiDU did, but with an emphasis on readability and more modern syntax.

It's already possible to do something along those lines in WeiDU, if one makes a lot of constants naming various values and offsets. It's not quite as high-level as what you were working on, but it makes the code a bit easier to read and write:

COPY_EXISTING ~drasus.cre~ ~override~
	WRITE_LONG killxp 3000
	WRITE_SHORT currenthp 92
	WRITE_SHORT maxhp 92
	WRITE_BYTE strength 18
	WRITE_BYTE exceptionalstrength 100
	WRITE_BYTE dexterity 18
	WRITE_BYTE thac0 6
	WRITE_BYTE levelfirstclass 15
	WRITE_ASCII classscript ~medrasus~ #8
	SET_BG2_PROFICIENCY PROFICIENCYFLAILMORNINGSTAR 5
	SET_BG2_PROFICIENCY PROFICIENCYSWORDANDSHIELD 2
	LPF ADD_CRE_EFFECT INT_VAR opcode=protectionfromopcode target=1 timing=9 parameter2=silence END
	LPF ADD_CRE_EFFECT INT_VAR opcode=modifymovementrate2 target=1 timing=9 parameter1=255 parameter2=1 END
	LPF ADD_CRE_EFFECT INT_VAR opcode=criticalhitbonus target=1 timing=9 parameter1=20 END
	LPF ADD_CRE_EFFECT INT_VAR opcode=haste2 target=1 timing=9 parameter2=improvedhaste END

Aquadrizzt

Wow I didn't know that that worked in WeiDU. How new is this feature (or did i just miss it entirely)? EDIT: Oh okay so you could do this in WeiDU as well, but no one has yet; that makes sense.

Bitcode editing is mostly just a proof of concept that I could get Python to play nice with the game files. I do think that where a new modding language would really shine is in the general syntax/structure (loops, conditionals, etc) and readability.

Like, this is what it takes to get a kitcode ID in WeiDu

COPY_EXISTING ~kitlist.2da~ ~override~
	COUNT_2DA_COLS k_cols // amount of columns
	READ_2DA_ENTRIES_NOW k_rows k_cols // read all file into memory  
	FOR (k_row = 1; k_row < k_rows; ++k_row) BEGIN // iterate over rows
		READ_2DA_ENTRY_FORMER k_rows k_row 1 ~k_kit~ // read column value
		PATCH_IF ~%k_kit%~ STRING_EQUAL_CASE ~%kit_name%~ BEGIN
			SET kit_row = %k_row%
			READ_2DA_ENTRY_FORMER k_rows kit_row 9 kit_code
		END
	END
BUT_ONLY

Whereas in Python, it could be as simple as the following.

kitlist.loc[kitlist['NAME'] == kitname,'ID']

kjeron

Aquadrizzt wrote: »

Whereas in Python, it could be as simple as the following.

kitlist.loc[kitlist['NAME'] == kitname,'ID']

This is the equivalent code in Weidu:

IDS_OF_SYMBOL(~KIT~ ~NAME~)

While weidu doesn't have similar built-in reads for 2da files, they could be made such that you can get those values with something similar:

$KITLIST(~ID~ ~NAME~)

The only real obstacle for 2da files is that using "NAME" (instead of "ROW#") is that it's not necessarily unique - several 2da files don't rely on unique labels for each row, instead only caring about the actual row# (and even if the first column is a row number, it's sometimes just for show).

ALIEN

@kjeron It's not only about what the code can do, it is also about how easy is to understand/write/expand etc and the whole process of edition game resources via code. I thing you underestimate the amount of improvements which new weidu could do, I mean, there must be a reason why some modders fallback to LUA/C# or even JAVA right?

Aquadrizzt good luck with you project!

Alonso

ALIEN wrote: »

there must be a reason why some modders fallback to LUA/C# or even JAVA right?

You can mod IE in Java?!!! Wow, how does that work? I speak Java!

@Aquadrizzt: That looks very promising. Do you have a thread or web page for PyDU?

OlvynChuru

Alonso wrote: »

ALIEN wrote: »

there must be a reason why some modders fallback to LUA/C# or even JAVA right?

You can mod IE in Java?!!! Wow, how does that work? I speak Java!

By using NearInfinity, maybe? That's written in Java.

Alonso

Aquadrizzt wrote: »

PyDU isn't "ready" by any means, but it is able to handle simple bitcode editing of existing files.

Have you considered creating a thread on the topic and maybe asking for help to speed up your project? I doubt I have the knowledge required, but if I could help in any way I'd be happy to do it.