[Project] A new language for writing mods
Circonflexe
Member Posts: 6
in BG:EE Mods
I have started to implement a new way of writing mods, using an existing programming language instead of WeiDU.
Here is the repository and here is the documentation.
The language is Julia; this is portable to at least as many platforms as EE games, while offering all the required syntactic sugar and being reasonably efficient. Short teaser:
I am perfectly aware that WeiDU exists and of the huge codebase that it has collected over the last 20 years. However, I also think that this tool starts to show its age, and hope that InfinityEngine games (and mods) will remain alive for at least the next 20 years, so it is not too late to try and write something even better than WeiDU. Moreover, distant development goals include some level of compatibility with WeiDU
as well as semi-automatic translation (e.g. this should be possible at least for .d/.tra dialogue files).
The repository's README (linked above) contains a longer discussion of all the possible advantages
which such a tool could eventually offer relatively to WeiDU. These importantly include gains in robustness (e.g. no mod crash because of missing tildes in .tra file) and portability (no need to include shell code in mod tp2, Julia makes everything portable by default). I also hope that a more accessible syntax, making life easier for mod authors and translators, would leave them more energy for writing many good mods.
This is not usable yet. I only post the link here because the project is now advanced enough to start discussing it in public. Although most core concepts are mostly implemented, for now applications are strictly limited to editing items and dialogs.
So: I am now welcoming any form of constructive input about this tool, be it about design (which is not closed yet), testing, or even (for the brave) help in developing.
Final remark: I am not a mod author, but I did play all the IE games since the very first CD-ROM editions and Weimer's first mods. Since however I can program a bit, think of this project as an eventual “thank you” to all the people who can write good mods.
Here is the repository and here is the documentation.
The language is Julia; this is portable to at least as many platforms as EE games, while offering all the required syntactic sugar and being reasonably efficient. Short teaser:
# Let's create a big sword!
bigsword = Longsword(_"Really Big Sword", +1, weight = 200,
description = _"A sword so big that it does crushing-type damage!")
bigsword.abilities[1].damage = Crushing(2d6+1)
# Translations are handled by `.po` files, no need to bother
# with @XXX strings.
# Keevor is a bit protective about Imoen:
from("imoen", 1)
actor("reevor") interject(_"Hey, you! Be gentle with Imoen!")
I am perfectly aware that WeiDU exists and of the huge codebase that it has collected over the last 20 years. However, I also think that this tool starts to show its age, and hope that InfinityEngine games (and mods) will remain alive for at least the next 20 years, so it is not too late to try and write something even better than WeiDU. Moreover, distant development goals include some level of compatibility with WeiDU
as well as semi-automatic translation (e.g. this should be possible at least for .d/.tra dialogue files).
The repository's README (linked above) contains a longer discussion of all the possible advantages
which such a tool could eventually offer relatively to WeiDU. These importantly include gains in robustness (e.g. no mod crash because of missing tildes in .tra file) and portability (no need to include shell code in mod tp2, Julia makes everything portable by default). I also hope that a more accessible syntax, making life easier for mod authors and translators, would leave them more energy for writing many good mods.
This is not usable yet. I only post the link here because the project is now advanced enough to start discussing it in public. Although most core concepts are mostly implemented, for now applications are strictly limited to editing items and dialogs.
So: I am now welcoming any form of constructive input about this tool, be it about design (which is not closed yet), testing, or even (for the brave) help in developing.
Final remark: I am not a mod author, but I did play all the IE games since the very first CD-ROM editions and Weimer's first mods. Since however I can program a bit, think of this project as an eventual “thank you” to all the people who can write good mods.
0
Comments
This looks like it has more of a focus on readability from the author POV.
To paraphrase some thoughts I recently had in the g3 discord:
What if a new tool kept weidu/weinstall compatability for the rollback (--uninstall) operation?
Then you'd just need weidu/weinstall to have a small switch as it reads a file - if it sees tp2, it runs weinstall as it currently does. If it sees a new file suffix intended for your program, it'd run that one instead.
That may allow seamless integration into weidu setups (as long as the various weidu side-effects were mirrored - writing to weidu.log, creating the backup/ files in a way weidu proper can use them).
It might actually greatly open up the novel tooling space (I could write some weird installer, and have it play nicely with yours and weidu).
I think a simple JSON -> weidu transpiler might be very valuable as well - I would have a much simpler time writing custom tooling around BG items if they were represented as basic JSON or xml data.
So instead of:
You may have (using xml here, slightly more readable than JSON on a forum):
<items> <item name="BigSword" type="longsword" desc="Really Big Sword" damage="2d6 + 1" weight=200 enchantment=1> <abilities>(whatever custom effects)</abilities> </item> ....(all the other package specific items).... </items>WeiDU started out to edit the dialogues and everything else just grew on top of it. Yes, it's syntax is weird and uncommon. On the other hand, it allows to do everything and half of the shipped internal functions are written in WeiDU inside it. This is important, because in the older days, lowlevel stuff and direct byte manipulation were more common, and while I get this one tries to abstract away the things, abstraction without explanations is bad and shouldn't be the default.
WeiDU's evolution was that it started from the lowest level. The teaser already implies this goes another way, opting for internal magic numbers, looks restrictive from the get-go and has way too many assumptions on stuff it shouldn't assume about (a great example is the 2.6 soundset additions - if a mod wants to use those, WeiDU doesn't offer a constant, but you can just go with the classic byte offset manipulation and call it a day, while this teaser implies me that ahwell, tough spot). This isn't robustness, this is a case of "I don't understand the needs of my target audience".
On the other hand, the B solution with a TP2 wrapper is doable and there are mods doing it - The Horde is written in Java and does exactly this, the tp2 calls the Java executable. (This also means The Horde's bugs can't be fixed at all, because the Java executable lacks source code and one needs to reverse-engineer it outright to fix the issues and yes, the mod is buggy). In fact, The Horde is outright the reason why I wouldn't even support a language which needs a compiler. I also participated in a modding community for a decade where there were actual effort wasted to introduce obfuscators in the scene, preventing modders to learn from each other due to asset protection. I've seen the toxicity that can lead to and having a compiler in the ecosystem provides a massive starting point for this mindset.
Note that WeiDU offers everything for a mod manager running on top of it, as the BWS proved, and PI/subtledoctor's Mac toolkit proves as well. There is room for another one in this layer though, since PI's mindset isn't novice-friendly - but it should really follow PI's metadata format proposal as a standard.
There's already SCS's SSL as an alternative option which is somewhat based on Perl's syntax and invokes that, but since it still keeps everything on the interpreter stage, this allows other modders to learn and pick it up and incorporate into their mods, like how Made in Heaven did.
Also see https://www.gibberlings3.net/forums/topic/32890-a-turing-machine-in-weidu/ and https://www.gibberlings3.net/forums/topic/32862-alternatives-to-weidu-or-how-to-write-a-smarter-sound-set-installer/ which also covers this topic. Although the latter one was atleast using a language I wouldn't consider fringe outright.
Please see the discussion linked above, but here are some examples:
1. .po files the industry standard for translations, whereas .tra files are quite brittle (e.g. any translation missing even one string crashes the mod). Also, the current quality of mod translations is often quite poor, because .tra strings are translated without context; e.g. translators see only "Slow" and don't know whether this is an adjective or a verb, and despite their best efforts this produces nonsensical translations. This is basically impossible to fix without changing the language.
2. Right now a lot of mods invoke shell scripts or batch scripts depending on OS. This forces the authors to write (and maintain) several sets of functions while often being able to validate only one of them. This is of course a great source of bugs. Hence I suggest a solution which totally dispenses with shell scripts and makes everything portable by default, by running on a platform which is already as portable as the game itself. Again, this could be fixed by implementing a few shell functions (cp, mv, rm) inside WeiDU, but at this point we are already converting WeiDU into a new tool aren't we?
2.5 While we are on portability: WeiDU is quite bad with case-sensitive filesystems (so that e.g. installing mods on a Linux install requires mounting a separate ad-hoc ciopfs, which significantly slows down the game). I am trying to write something which works natively with any filesystem and (this is important) without requiring any extra effort from mod authors.
3. (A more minor point) This module seems (for now; tests are obviously not complete) to be running circles around WeiDU or InfinityExplorer when it comes to speed (on my ordinary laptop it loads the whole game database in < 5 milliseconds, and this is not a SSD) — this becomes significant when some mods litterally take hours to install (SCS...) and one may need to reinstall the whole stack a few times to tweak some installation options.
4. Any existing language is, by definition, strictly less obscure than WeiDU. I won't discuss style here (although WeiDU would lose, even without mentioning OCaml or SCS's Perl) but the exact choice of a non-WeiDU language does not matter much, as long as it is easy to learn. In this respect most modern script-like languages are very much alike: had I used (say) Lua or Python instead of Julia, the teaser above would still have looked almost exactly the same. (But if you want to write a Lua variant of my project, please do, and I will probably even join you — this could possibly have been an even better choice, although the task of writing all of this in a very minimal language looks a bit daunting to me).
5. About magic numbers: what exactly makes you believe that it is not possible to assign a value at a given offset with this program? This would actually be a quite simple feature to implement. However, I believe that this is would be a *bad* feature, because abstracting the format of the game resource database from mod authors is extremely useful. Here is a little thought experiment to support this: imagine that in some future version the binary format of ITM file changes. Now all .tp2 files which use magic offset constants need to be rewritten. On the other hand, the InfinityEngine module just needs a simple patch to learn about (and detect) the new binary format and any code written by mod authors then remains valid automatically. (Right now the module only knows one version of game resource database, but extending to several versions is in the pipes to support other games than BGEE. Future changes like in the thought experiment would naturally hook into this mechanism and be very straightforward to write).
Oh and by the way, of course this does not need a compiler for individual mods: compilation is JIT so that this is used exactly as if it were a scripting language. The intended development cycle is exactly the same as that of WeiDU mods. And the goal here is to make mod source less obfuscated; this should be obvious when comparing the nonsensical example above with its WeiDU equivalent.
I am perfectly aware of WeiDU's “organic” growth, and I believe that all of the disadvantages of WeiDU listed above are artifacts of its growth, since it originated as a single-OS, single-author, single-language, (almost) single-mod tool.
Now we have the advantage of hindsight: while Westley Weimer had to invent everything from scratch, we know which features are actually used by mod authors and how to smooth everything, and this requires re-thinking the toolset globally. For one more example, mod stack management is more of an afterthought for WeiDU: claiming that WeiDU offers “everything needed” is a bit exaggerated when even language codes are not standardized between mods, components options are passed through terminal interactions, and mod stack managers tend to work despite WeiDU's conception. Making a WeiDU mod BWS (etc.)-friendly brings even more constraints on its author.
Besides, the fact that a mod manager is even needed at all is one more deficiency of WeiDU that I intend to remedy (and mod managers themselves are not portable — I never succedded running PI on Linux and I was not even aware of Mac-specific solutions). I had to write my own mod manager, some code of which will probably be reused in this project.
1. Translations. ".po" files are the industry standard for localization. In comparison, ".tra" files are extremely brittle: for example, a single missing string crashes the mod on install (this can easily happen when translations are not updated as fast as the mod's core).
Moreover, the quality of mod translations is currently sometimes quite poor, and this is not the translators' fault: ".tra" files offer absolutely no context for translated strings. When a translator sees, say, "Slow", they don't know whether this is a verb or an adjective, and this does lead quite often to nonsensical translations. This is basically impossible to fix in WeiDU as it is now. On the other hand, I already have some quite simple code for providing translation contexts.
2. Portability. A lot of mods use OS functions for moving files around, which requires writing the same function several times in various languages (shell script/batch). Of course most authors only use *one* of those languages, so the other ones become obsolete very fast: this is one more great source for bugs.
It would naturally be possible to fix this by including e.g. "mv", "cp" functions in WeiDU (but this already counts as “a new modding language”).
3. Portability/performance. WeiDU is notoriously bad with case-sensitive filesystems, to the point when running it on Linux requires mounting an ad-hoc ciopfs. This significantly slows down the game. Again, since file names are determined by mod authors, this would be very hard to fix from within WeiDU.
On the other hand, I am writing something which completely abstracts the file names from mod authors and natively solves this problem.
4. Magic numbers. What makes you believe that this tool cannot support writing offsets by hand? This would actually be a quite simple feature to write. But I believe that going down in abstraction would be a very bad idea, much in the same way as replacing a C struct by a char[] of the same size and assigning pointers by hand, and not only for convenience reasons.
Here is a little thought experiment in support of this: imagine that in some future release, the binary format of ITM files is modified. Now mods using magic offset numbers need to be completely rewritten, and this is very brittle (every single use of magic offsets need to be trakced down, etc.). On the other hand, if the modding language offers an abstraction for items, then only the language itself needs to be aware of the new format. This is actually something that is eventually going to be present in the InfinityEngine module anyway, since various games use different resource formats, so adding just one more format can plug into
this system.
(On a side note: I play without sound, so I am totally unaware of those 2.6 soundsets. Do you have a pointer to this?).
5. Mod stack management. The fact that an external tool is even needed at all is already a disadvantage of WeiDU. Moreover, modding tools tend to work *despite* WeiDU's conception and not with it: even something as simple as language codes is not standardized across mods; component options are passed by simulating terminal IO; mod metadata was added as an afterthought; mod conflicts are left to be guessed; obtaining the mod README is hard (for my own mod manager I needed to patch a "--list-readme" option into WeiDU) and component READMEs are impossible, etc.
Moreover, mod tools tend to be not portable (I was unable to run PI under Linux and you also mention a Mac-only tool), to the point where I needed to write my own mod manager to install a big WeiDU stack on Linux. (I expect to reuse some of that code in this new project).
6. Speed. This is a more minor point, but still significant since some mods (SCS) can require a few hours to install, and with WeiDU's model tweaking a few options may need to reinstall the whole stack. Of course the benchmarks are not definitive yet, but on my (non-gamer, not SSD) laptop my module can load all game resources in less than 5 milliseconds.
I am fully aware of WeiDU's history (as said above, I used it from the very beginning): it began as a single-author, single-OS, single-language, and (almost) single-mod tool. Again, I am not saying anything bad about WW's work: he started everything and we are all grateful to him for this. But now we happen to have the benefit of hindsight; we know which features are used by mod authors, we know how many mods (and of which kinds) compose a typical installation, we know how to make everything robust, so we can try and design something better from start.
Note that none of the points above is about syntax. But now that I have established why a new tool can be an improvement, we might just as well try and make life easier for mod authors. In this respect, almost any existing language is by definition less obscure than TP2 (even LISP
game itself and with an easy learning curve (I believe most people would indeed find it easier than TP2, WeiDU's OCaml, or SCS's Perl — and I say this while generally being a defender of OCaml or Perl).
I am not married to the Julia language (or to anyone named Julia for that matter). If by chance you happen to prefer, say, Lua, then feel free to re-use my work to write a Lua modding library -- I might even join you on that project (I believe Lua could be another interesting choice for the final product, but writing all of this in such a minimal language is a bit too intimidating for me alone).
Last note: I fully agree with your remark on compiled and obfuscated mods. Note however that this exactly not what I am suggesting; Julia is JIT-compiled, so that in practice it behaves as a scripting language, and the development process for mods in this system would be almost exactly the same as with WeiDU (with maybe minor changes in the translation process, to the advantage of translators). On the contrary, I try very hard to make the syntax as simple as possible, not only to help write good mods, but also to help maintain them.
I would not be so sure about that (but I'm not claiming to be 100% sure on what I say next either). The unmodded EE game works without a case insensitive file system. Then, I've modded stuff manually and with NearInfinity, and the game did not require it either. Even when changing the case of file names inside "override" on purpose. Then added changes via WeiDU, and suddenly the game needs it to run. More: you just need the case insensitive file system to _start_ running WeiDU, because is unable to locate the dialog.tlk file. Maybe this is fixable in WeiDU, I don't know, but I think WeiDU is part of the problem.
This is an interesting project, definitely worth pursuing. Though you do have your work cut out for you. Your program is going to need to re-implement several of WeiDU's features before it can replace WeiDU. You've likely solved some of these already, but it's worth checking:
1. You'll need to implement ways of modifying 2DA and IDS files. Mods will occasionally have to read or write values in these files, or add or remove rows.
2. There should be some way of matching and replacing text in files (akin to WeiDU's REPLACE_TEXTUALLY and REPLACE_EVALUATE). These functions can be used to dynamically modify BAF files, the BAF chunks in DLG files, UI.MENU, the INI files used for animations, as well as LUA files. This also requires you to implement some way of decompiling and compiling the BCS files (modifying the BAF chunks in DLG files also requires you to implement some way of decompiling and compiling the DLG files, as WeiDU is already capable of doing).
3. Replacing the system of writing at offsets for editing fields with an object-oriented system is okay, but make sure your program has a comprehensive list of variable names for every field in ARE, CHR, CHU, CRE, DLG, EFF, ITM, PRO, SPL, STO, VVC, WED, and WMP files. Otherwise, I might run into situations where I want to modify a field that you forgot to include a variable for. I remember this was a big problem when I was making mods for Skyrim; the in-game functions don't let you modify all the fields in spell files, so I had to use a third-party modding utility to modify those things.
4. Even if specifying offsets to write at like in WeiDU isn't the primary way of editing files, it should still be an option, just in case someone wants to edit a file type you didn't plan for. In the process of developing Icewind Dale 2: Enhanced Edition, for example, I had to modify BMP files with WeiDU at one point (this was specifically to modify search maps, which have gameplay implications).
5. Make sure that you have ways of adding new data blocks to all file types that have those (e.g. new headers to items and spells, new effects to item and spell headers, new actors, containers, doors, regions, etc. to areas, new items to stores), as well as ways of removing them.
6. Also, just checking: does your program have a way to make copies of existing files and modify the copies, like WeiDU does? This is generally a much better idea than trying to make a new sword item file from scratch.
7. Does your program have a way to copy all the new files from the mod folder into the override folder with a single line of code? WeiDU can do this. My bigger mods typically have a line of code that says this:
COPY ~%mod_folder%/Allfiles~ ~override~
It would be good to have some equivalent of that.
8. Does your program have some equivalent to COPY_EXISTING_REGEXP that lets you modify all files in BIF and override that match a specific regexp?
9. Can a mod coded using your program be uninstalled, undoing the changes of that mod but not the changes of mods installed before that one?
10. Can a mod coded using your program have multiple components (potentially with subcomponents), with some way of allowing the user to choose which components or subcomponents they want to install?
11. You also might need to add equivalents of WeiDU's various functions that do special or complex things (e.g. ADD_KIT, ADD_PROJECTILE, BUT_ONLY_IF_IT_CHANGES, IF_EVAL).
I know close to nothing about Julia, and I don't feel quite motivated to learn it, but certainly the availability of a tool to make modding easier could make me think about it. Some other scripting language could be much better, though.
My "plan" (more like a pipe dream, to be honest) was to complete the support for IE file formats in the library of my application, Moebius Toolkit. I've been progressing very little on it, but I plan a more aggressive strategy for the next release, as I want to have basic browsing in the application. For that, I would only need read support, but write will be a bit simpler, I think, as the main deal has been me being very panicky about breaking compatibility, and having unit tests and the like.
The next step was making a toy mod, for my own use, in C++ (the language in which the app and library are written), then add scripting support for it. Lua is the first on the list, as it seems the most appropriate (simple, popular in games, used in EEex and the InfinityEngine console, etc.). JavaScript was also on the options because popularity and availability.
I think scripting is better for this kind of project (mods), but I surely don't have a strong opinion. I don't see the problem with compiled languages given that WeiDU, EEex, BG2RadarOverlay or NearInfinity are all compiled (and open source or at least have source available). The only problem with opaque binaries is that they can do shady things. My answer to that would be to skip this mods entirely.
How does your proposed alternative interact with GemRB, a revised Infinity Engine, and EEex, a default Infinity Engine extender?
As I said, I focused first, as proofs-of-concept, on items (code written here will be reusable for many resource types) and on dialogs (efining and editing state machines needs special syntax). I do have some code for reading 2da and ids (not writing them though, but this will be the easy part, just use printf). I will also need some way of using IDs in-script (this will be quite easy, the language has dictionaries).
This is actually one of the reasons behind the language choice, since Julia has native regular expressions, whereas Lua does not.
replace(string, "pattern" => "substitution") is native Julia syntax.
By construction it does; for an example, you can look in the git repository (file src/InfinityEngine.jl) how "struct Item" is defined.
About BMP files: this module does not yet handle binary resources. I thought that the most we needed to do with them was copying and renaming files; your example does show that there may be a few more needs. However, this is (again) one of the joys of using an actual existing language: of course it already has features for editing a binary file and it won't be much work to use that.
This is already the case (well, for those resources which are already implemented, namely actors and items): adding new effects and abilities to items are trivial since these have the API of a Vector (a native construct in the language), we can simply push onto the list etc.
Of course it does, and this is actually shown in the example above: Longsword() is really shorthand for cloning the basic long sword item; the actual function definition is (almost verbatim) Longsword(args...) = copy("sw1h04.itm", args...), and we can use just any item name in there.
Not yet (although it would be straightforward to implement since the language natively contains "cp") but the goal is to have something slightly different, and (intended as) more author-friendly:
- game resources are identified by any author-chosen string and (semi-)automatically namespaced to prevent name collision (rendering the 2-byte prefixes useless and names more explicit) (however nothing prevents author from still choosing 2+6-byte names if they really want to).
- I suggest defining resources from within the script instead on from an external file (having a text version is more edit-friendly and even more maintenance-friendly: it can be versioned, diff-ed, etc.). I think of this as having access to the source code of resources vs. only a compiled form (even though “decompiling” is here quite simple).
Given all of this it is still possible to have a one-line copy, of course.
Just like above: scripts are not yet a handled resource type, but the Julia language has very easy access to regexps, so this will be very simple to implement.
This is definitely on the TODO list, and one of the points which I think deserves discussion, hence my post here. I think that this is actually the single hardest remaining problem. WeiDU's stack-based concept has the advantage of simplicity but is not too efficient; when you want to tweak an option deep in the stack and you need to uninstall + reinstall everything (including taking ages to reinstall SCS) you regret it.
Ideally I would like to have some system where each game resource is “owned” by either the base game or a mod: this would prevent mod conflict, help sort out the “mod install order” problem, and make uninstall trivial (just restore the original resource). However the needed granularity could be very fine (e.g. must each single cell in a 2da file have an owner?). I must test how this works in practice (e.g. using data compression I could probably store the (mod) -> (set of owned resources) map) before committing to a particular solution.
Same answer as 10. Moreover, what I have now is not yet a proper modding system (“WeiDU”), but an Infinity Engine-access interface (“InfinityExplorer”): the way to define actual mods is not defined yet.
This is for me one of the greatest advantages of using a “real” programming language: once we have basic access to resources, implementing that kind of bigger changes becomes very simple (just write a function in that language). Of course there will eventually be some form of ADD_KIT etc., but that is the easy part.
@GraionDilach : I maintain my point 2 and your answer is quite obviously nonsense since the InfinityEngine *does* run on a case-sensitive filesystem. The problem is not with the game engine but with mods. With the tool I am developing, I did succeed in modding a few resources without ciopfs and having them recognized with the game. My experiments actually suggest something very close to @suy 's answer: I am under the impression that the game engine does ignore case when opening resources and that the problem lies entirely within WeiDU's ability to open files (and author's ability to use case-consistent filenames, which is quite poor in practice).
I selected this language because it has useful features (regular expressions, introspection, somewhat extensible syntax) and probably decent staying power. Obviously, knowing more about the Julia language would of course be needed to maintain the tool itself. (However, this is still an improvement over WeiDU).
Yes, using a lowercase size suffix will not work on Windows. That feature is picky enough that in all my setups where I am using it, I explicitly opt for mixed cases, with everything else excluding the size suffix being lowercase.
Doesn't sound like a case insensitive logic to me.
Also, newsflash: WeiDU has this ownership model. That's what --change-log runs on. Have an example.
EDIT: Come to think of it, it is possible that the case-sensitivity I experienced comes from the Lua sandbox running the UI code (because I know for a fact that the variables inside the Lua code segments are very much case-sensitive there, had to document this within the EE Soundset Tool even).