NWN file formats in C++

Liareth

I wanted a decent way to read NWN file formats in C++ for personal NWN-related projects so I decided to write a library to handle it. It's open source and completely free for anyone to copy, use, and change with or without attribution. It should compile and work on all platforms and x86 architectures (probably not on ARM though) but I haven't tried it on Linux so there may be a few compile fixes needed there.

Note that this is a library for C++ developers who want to parse Aurora engine file formats, and right now only GFF is implemented.

https://github.com/Liareth/NWNFileFormats

You can see how it works here:

https://github.com/Liareth/NWNFileFormats/blob/master/Examples/Example_Gff.cpp

Liareth

This now fully supports ERF per the spec.

Example:

https://github.com/Liareth/NWNFileFormats/blob/master/Examples/Example_Erf.cpp

tgx

I like how you call it "Friendly API". Looks good. Hopefully we'll see some useful applications down the road that make use of these. Better to have an API than to do it again and again. I suspect this will be useful.

GreenWarlock

my personal peeve - you could avoid a lot of 'assert(not null)', and be correct by-design, by passing references rather than pointers. Not a good idea for arrays, where iteration/indexing is expected, but when reading/writing a single object, for example, it usually leads to slightly cleaner code.

Of course, coding convention is in the eye of the beholder, and I am stuck with this asserting null pointers at work too, which might be why I am so sensitive

GreenWarlock

BTW, love that you are using C++17 features already - I am a standards geek and love to see our stuff being adopted, even if in only a small way

Liareth

@GreenWarlock Good feedback, thanks! (And for the PR too, assuming that was you!)

I do prefer references in most cases, but for functions that take out parameters, I prefer raw pointers because it's more explicit to me that the intent is to return a value that way. In this library, I've used raw pointers only for out parameters (Gff/Erf::ReadFromBytes and Gff::ReadField), and everything else takes references. I'm not allowed to use exceptions at work so I find myself writing code in this way rather and returning error codes rather than returning by value.

GreenWarlock

We have the same conventions where I work too, so totally understood, and yes, those were my PRs as well

Liareth

This now supports KEY and BIF file formats too, and I've created a proof of concept that just extracts all the game resources from a KEY.

https://github.com/Liareth/NWNFileFormats/blob/master/Examples/Example_KeyBifExtractor.cpp
https://github.com/Liareth/NWNFileFormats/blob/master/Examples/Example_Key.cpp
https://github.com/Liareth/NWNFileFormats/blob/master/Examples/Example_Bif.cpp

GreenWarlock

Darn - struggling to get this to build on my Mac as I cannot persuade cmake to use my latest Clang checkout, which has the C++17 libraries. Any pointers on configuring make for noobies appreciated! In the meantime, you probably want to change the file permissions on the build.sh file to add the execution bit on Unix platforms.

Liareth

@GreenWarlock I think CMake uses the CC and CXX environment variables, so if you set those to the path of clang and clang++, that should work. You can do also cmake -DCMAKE_C_COMPILER=/my/clang -DCMAKE_CXX_COMPILER/my/clang++ .. theoretically but I've never tried it.

Liareth

This now has TLK support. I've also optimized the BIF and ERF support to be much more memory efficient. The implementations now use memory mapped files and don't duplicate data all over the place. When extracting all of the game resources, this reduced my peak memory usage to 250 mb from 900 mb.

@GreenWarlock I played around with compiling it on Clang today. You need Clang 5.0 or later. I use the unstable branch of Debian which has Clang 6.0 and GCC 7.2 available as a package, both of which compile the library out of the box.

If you have a manual compiler install, you can do it like:

CC=/usr/bin/clang-6.0 CXX=/usr/bin/clang++-6.0 ./build.sh

TarotRedhand

Any updates on this? Pity it's on Unix from my perspective, being a windows user. Do you plan on tackling the .mdl format?

TR

Sherincall

It should work on Windows just fine. You just need to install the latest visual studio and compile it from source.

Liareth

@TarotRedhand Yes it should work out of the box in latest VS2017 - that's how I developed it, with Linux support coming secondary. I'm done with the project for now as I'm able to read all of the NWN formats I need. I will be adding support for writing GFF at some point in the future.

TarotRedhand

Thanks for the update.

TR

Liareth

I've been adding some more support to this library lately. It now supports editing and saving GFF, TLK, and 2DA. I've also added a couple of tools to the repository - most notably a utility to merge two 2DAs together.

hmudd67

Now you just need to have .uti file support.

shadguy

@hmudd67 .uti and most of the other module assets are GFF format with a filelname extension change to help specify what they're used for.

-Dave

Trasd

@Liareth

Is your code based on bioware_aurora_engine_file_formats.zip? Or, is there an updated reference I missed somewhere?

Thanks and, thanks for the work!

Sherincall

There have been no updates to those formats since 1.69, so the spec is still valid.

Trasd

@Sherincall

Thanks.

That's what I thought (backwards compatible and all). But, I wanted to confirm.