New resource data formats
TarotRedhand
Member Posts: 1,481
There is a proposal on the trello boards that people are voting for yet until this thread, it does not appear to have been discussed elsewhere. Not only that but I do not recall a thread anywhere on these forums where all of these have even been proposed together. This is the proposal -
I think all of these have problems so I'll take them one at a time.
Data:
I'll deal with the two proposals here individually followed by some overall observations.
- UTF-8 2da's?
OK, I've done some checking on this one as it is very specific (This web page is succinct). The proposal is that 2sa files should change from a one character = one byte encoding system to a system where one character can be stored on between one (minimum) and four bytes (maximum). The question here is why would you want to do such a thing? Is there a hidden agenda here? Is it being proposed that Chinese/Korean/Japanese/hieroglyphics be used in 2da files? 2da files are essentially just data files, why would you need to waste space changing something that not only works but works well?
- .csv (comma separated values) 2da optional alternative?
A reasonable proposal at first glance. This would allow 2da files to be created, stored and edited using a spreadsheet program. The thing is there already exist tools created by users to do just this anyway. Also needing a spreadsheet adds an extra tool needed. Currently, 2da files are easily edited using just a plain text editor (I use notepad++ for this).
Data: Overall
I suppose I should be thankful for small mercies here. At least it is not proposed that 2da files should need to be compiled like they are in the Knights of the Old Republic single player games. Apart from that, I think these proposals are somewhat pointless exercises in fixing something that isn't broken. Before you ask what about for other languages in the case of the UTF-8 proposal, remember that 2da files are predominantly data files. Language specific stuff belongs in tlk files, that's what they are there for. In respect of the csv proposal - Now I don't know if it's just MS Excel 2007 that has this little quirk (I suspect not) but while it expects strings with spaces to be enclosed in quotation marks in csv files that it imports, it strips them on export. Oh, fun and games.
So no, these get the thumbs down from me. As far as I can see it's either a case of "Ooooh! Shiny, New" or (in respect of the user made tools that already exist) "Not Invented Here" - can't trust them.
Art:
- 32 bit PCM wav for SFX/voice sounds
Now I question why on earth anyone would feel the need to have such an overly accurate representation of the sound of water droplets falling into a pool of water. You do realise that at 32 bits the level of accuracy is 65,536 times that of of CD music (16 bit Pulse Code Modulation). A level of accuracy that will be lost on by far the vast majority of people (is your PC/Mac hooked up to a Bang and Olufsen sound system?).
- "open" resolutions for mp3 audio (128 - 320 CBR/VBR) in .mp3 or "lossless" formats
This specific proposal needs to be expanded. Now I have no problem with the mp3 part (all right - there's no need to be sarcastic - get up and stop pretending you've fainted). After all NwN has used mp3 files from the get go. The only difference between a BMU file and an MP3 file of the same sounds is that a BMU file has eight extra bytes. These extra bytes are inserted at the very beginning of an mp3 file to change it into a bmu file. The actual bytes are the same in every case - "BMU V1.0" (just drag and drop any bmu file onto the VLC player and it treats it as what it essentially is - an mp3). Anyway, the reason I say this needs to expanded is that we need to know specifically which lossless format(s) we are talking about here if a meaningful discussion is to be had.
- Optional use of Industry standard header DDS block compression format
Nice idea in theory, might be problematic in practice. When Bioware adopted the Direct Draw Surface (aka dds) image format it was nut just cutting edge technology, it was the bleeding edge. So much so that MS altered the specification for it, not that long after. This means that (as far as I know) NwN is the only game that uses this version of dds. This being so there are thousands (no really) of textures in this obsolete format. Now I will assume that it is reasonably easy to differentiate between these two distinct forms in code. If not, this risks breaking an awful lot of existing custom content. But the real problem arises when little johnny/jenny comes along and wants to use and edit an existing dds texture. With this proposal you would escalate the "texture is broken" threads somewhat dramatically. I do have a counter proposal here that I will delineate after the next proposal.
- PNG compression format available for optional TGA replacement
Given that hard drives are commonly in the terabytes of size these days and TGA shrinks really well when compressed into an archive, why would you feel the need to replace an industry standard file format?
Taking these 2 proposals together I see a desire for change. The thing is to not get too carried away here. One thing to bare in mind is that should a new format be added it must, above all other considerations work in those 3D modelling programs that are most commonly used. Now I've tried to seek for a list of which image formats are most commonly used and at this point I have to say that google is absolutely useless for finding this out. So if you've got a link please post it. Now if was just a simple matter of getting the smallest image that would do the job I would suggest gif. Problem is I don't know if 3D modelling programs can handle it. Using the free Gimp, you can create gif files with an optimised palette. If you worry about the image quality, don't. I use it all the time for screen shots to post and in the vast majority of cases I can't see the difference. Also all patents on the format lapsed by the end of 2006. Of course if the £D programs can't use it, the point would be moot.
- New video formats and resolutions (needs discussion)
This one puzzles me. Haven't we just broken compatibility due to licensing issues? Why would we need to do so again? Or is this about the image size and aspect ratio rather than the file format. Definitely needs more than mere discussion, it needs clarification.
- New optional 3d model formats (FBX, X3d) with easier access to universal development tools for creating, editing, skinning, and animating models. (needs discussion)
Now before I address this I have to ask one question of the people who have voted for this - Have you any experience of 3D modelling at all? If not, why are you voting for this? A change that as far as I am aware of, nobody in the custom content creation community has actually asked for.
Enough of that. Specify which universal development tools are being spoken about here. I do hope you are not talking about that laughable Paint 3D that is part of Windows 10. Anyway, what pray is more universal than a plain text editor? The fact that the Bioware .mdl file format (there are other .mdl formats out there) exists in a plain text version is one of its greatest strengths and you want to throw that away for more "Ooooh! Shiny" stuff that requires a complex program just to open... A program that in all likelihood costs you an arm and a leg to buy... Of the free programs (that I am aware of) that cover this on the PC, there are only two that are worth considering anyway. Both of which are supported by the community. There is Blender. A free, open source, continuously maintained and updated 3D modelling program. While this is primarily a program for the creation of cgi movies, it is used by modellers for many different games and has many professional quality tools within it. The other universally used free tool is gmax. A cut-down version of 3ds max. Actually that is a bit unfair. It was actually designed purely for use by modding communities in general. They just left out what they thought wasn't essential for making 3D stuff for games. The only possible disadvantage is that it is not maintained. I believe that on the mac there were efforts to give Cheetah 3D the ability to handle the .mdl format but I might be mistaken.
Suffice to say I am definitely against this unneeded and unasked for (by the people who actually make the stuff) proposal.
At this point I'll stop not because I have run out of stuff to say but more because I could rant for ages.
TR
New resource data formats for higher performance, quality, and compatibility
Description
Data:
- UTF-8 2da's?
- .csv (comma separated values) 2da optional alternative?
Art:
- 32 bit PCM wav for SFX/voice sounds
- "open" resolutions for mp3 audio (128 - 320 CBR/VBR) in .mp3 or "lossless" formats
- Optional use of Industry standard header DDS block compression format
- PNG compression format available for optional TGA replacement
- New video formats and resolutions (needs discussion)
- New optional 3d model formats (FBX, X3d) with easier access to universal development tools for creating, editing, skinning, and animating models. (needs discussion)
Description
Data:
- UTF-8 2da's?
- .csv (comma separated values) 2da optional alternative?
Art:
- 32 bit PCM wav for SFX/voice sounds
- "open" resolutions for mp3 audio (128 - 320 CBR/VBR) in .mp3 or "lossless" formats
- Optional use of Industry standard header DDS block compression format
- PNG compression format available for optional TGA replacement
- New video formats and resolutions (needs discussion)
- New optional 3d model formats (FBX, X3d) with easier access to universal development tools for creating, editing, skinning, and animating models. (needs discussion)
I think all of these have problems so I'll take them one at a time.
Data:
I'll deal with the two proposals here individually followed by some overall observations.
- UTF-8 2da's?
OK, I've done some checking on this one as it is very specific (This web page is succinct). The proposal is that 2sa files should change from a one character = one byte encoding system to a system where one character can be stored on between one (minimum) and four bytes (maximum). The question here is why would you want to do such a thing? Is there a hidden agenda here? Is it being proposed that Chinese/Korean/Japanese/hieroglyphics be used in 2da files? 2da files are essentially just data files, why would you need to waste space changing something that not only works but works well?
- .csv (comma separated values) 2da optional alternative?
A reasonable proposal at first glance. This would allow 2da files to be created, stored and edited using a spreadsheet program. The thing is there already exist tools created by users to do just this anyway. Also needing a spreadsheet adds an extra tool needed. Currently, 2da files are easily edited using just a plain text editor (I use notepad++ for this).
Data: Overall
I suppose I should be thankful for small mercies here. At least it is not proposed that 2da files should need to be compiled like they are in the Knights of the Old Republic single player games. Apart from that, I think these proposals are somewhat pointless exercises in fixing something that isn't broken. Before you ask what about for other languages in the case of the UTF-8 proposal, remember that 2da files are predominantly data files. Language specific stuff belongs in tlk files, that's what they are there for. In respect of the csv proposal - Now I don't know if it's just MS Excel 2007 that has this little quirk (I suspect not) but while it expects strings with spaces to be enclosed in quotation marks in csv files that it imports, it strips them on export. Oh, fun and games.
So no, these get the thumbs down from me. As far as I can see it's either a case of "Ooooh! Shiny, New" or (in respect of the user made tools that already exist) "Not Invented Here" - can't trust them.
Art:
- 32 bit PCM wav for SFX/voice sounds
Now I question why on earth anyone would feel the need to have such an overly accurate representation of the sound of water droplets falling into a pool of water. You do realise that at 32 bits the level of accuracy is 65,536 times that of of CD music (16 bit Pulse Code Modulation). A level of accuracy that will be lost on by far the vast majority of people (is your PC/Mac hooked up to a Bang and Olufsen sound system?).
- "open" resolutions for mp3 audio (128 - 320 CBR/VBR) in .mp3 or "lossless" formats
This specific proposal needs to be expanded. Now I have no problem with the mp3 part (all right - there's no need to be sarcastic - get up and stop pretending you've fainted). After all NwN has used mp3 files from the get go. The only difference between a BMU file and an MP3 file of the same sounds is that a BMU file has eight extra bytes. These extra bytes are inserted at the very beginning of an mp3 file to change it into a bmu file. The actual bytes are the same in every case - "BMU V1.0" (just drag and drop any bmu file onto the VLC player and it treats it as what it essentially is - an mp3). Anyway, the reason I say this needs to expanded is that we need to know specifically which lossless format(s) we are talking about here if a meaningful discussion is to be had.
- Optional use of Industry standard header DDS block compression format
Nice idea in theory, might be problematic in practice. When Bioware adopted the Direct Draw Surface (aka dds) image format it was nut just cutting edge technology, it was the bleeding edge. So much so that MS altered the specification for it, not that long after. This means that (as far as I know) NwN is the only game that uses this version of dds. This being so there are thousands (no really) of textures in this obsolete format. Now I will assume that it is reasonably easy to differentiate between these two distinct forms in code. If not, this risks breaking an awful lot of existing custom content. But the real problem arises when little johnny/jenny comes along and wants to use and edit an existing dds texture. With this proposal you would escalate the "texture is broken" threads somewhat dramatically. I do have a counter proposal here that I will delineate after the next proposal.
- PNG compression format available for optional TGA replacement
Given that hard drives are commonly in the terabytes of size these days and TGA shrinks really well when compressed into an archive, why would you feel the need to replace an industry standard file format?
Taking these 2 proposals together I see a desire for change. The thing is to not get too carried away here. One thing to bare in mind is that should a new format be added it must, above all other considerations work in those 3D modelling programs that are most commonly used. Now I've tried to seek for a list of which image formats are most commonly used and at this point I have to say that google is absolutely useless for finding this out. So if you've got a link please post it. Now if was just a simple matter of getting the smallest image that would do the job I would suggest gif. Problem is I don't know if 3D modelling programs can handle it. Using the free Gimp, you can create gif files with an optimised palette. If you worry about the image quality, don't. I use it all the time for screen shots to post and in the vast majority of cases I can't see the difference. Also all patents on the format lapsed by the end of 2006. Of course if the £D programs can't use it, the point would be moot.
- New video formats and resolutions (needs discussion)
This one puzzles me. Haven't we just broken compatibility due to licensing issues? Why would we need to do so again? Or is this about the image size and aspect ratio rather than the file format. Definitely needs more than mere discussion, it needs clarification.
- New optional 3d model formats (FBX, X3d) with easier access to universal development tools for creating, editing, skinning, and animating models. (needs discussion)
Now before I address this I have to ask one question of the people who have voted for this - Have you any experience of 3D modelling at all? If not, why are you voting for this? A change that as far as I am aware of, nobody in the custom content creation community has actually asked for.
Enough of that. Specify which universal development tools are being spoken about here. I do hope you are not talking about that laughable Paint 3D that is part of Windows 10. Anyway, what pray is more universal than a plain text editor? The fact that the Bioware .mdl file format (there are other .mdl formats out there) exists in a plain text version is one of its greatest strengths and you want to throw that away for more "Ooooh! Shiny" stuff that requires a complex program just to open... A program that in all likelihood costs you an arm and a leg to buy... Of the free programs (that I am aware of) that cover this on the PC, there are only two that are worth considering anyway. Both of which are supported by the community. There is Blender. A free, open source, continuously maintained and updated 3D modelling program. While this is primarily a program for the creation of cgi movies, it is used by modellers for many different games and has many professional quality tools within it. The other universally used free tool is gmax. A cut-down version of 3ds max. Actually that is a bit unfair. It was actually designed purely for use by modding communities in general. They just left out what they thought wasn't essential for making 3D stuff for games. The only possible disadvantage is that it is not maintained. I believe that on the mac there were efforts to give Cheetah 3D the ability to handle the .mdl format but I might be mistaken.
Suffice to say I am definitely against this unneeded and unasked for (by the people who actually make the stuff) proposal.
At this point I'll stop not because I have run out of stuff to say but more because I could rant for ages.
TR
2
Comments
Also, IIRC there's a proposal to allow adding strings into 2das as well as references to tlk entries. I know this sucks for multiple languages, but it would make prototyping things easier; it's annoying as hell to have to mess around with tlk entries when you're in the middle of messing with adding new 2da entries. Um, I'd argue that it's the opposite of not invented here; 2das are NIH, CSVs are probably the 2nd file format ever invented (after .txt).
Having said that, please use something better than CSVs. They're really, really awful to work with. I still have nightmares about having to debug a CSV upload feature for a market research platform. CSVs don't have any metadata, nor anyway to know with certainty what character encoding they're using.
Excel will also, helpfully, silently change the text encoding, default to terrible Windows ISO-whatever-the-fail or UTF-16 (also known as "the worst of all unicode worlds") and, when finally coerced into using UTF-8, screw it up by adding a byte order mark. I also remember the pound (£) sign being especially fun because it could have 3 different code points depending on character encoding and one of them was the same as ” in UTF-8.
I repeat; do not use CSVs, here be dragons. Opening disclaimer: I'm not a sound engineer, nor have I done anything more than play around with the toolset. Take my thoughts here with a pinch of salt.
I've always been under the impression that the reason you had higher bitrates (and sample rates) with digital audio was to allow it to be edited without adding compression artifacts, clipping and other errors to the final result.
I don't know how much dynamic editing you can do for sounds in the toolset/scripting, but if you can do a decent amount it makes sense for it to have a higher sample rate/bitrate to avoid artifacts. Just thought you should know, the last MP3 patent expired last month. It's now a royalty free codec. Install lame and don't feel guilty.
Also, I'm guessing flac for the lossless codec. It's royalty free, lossless and in quite widespread usage. It could be Apple's lossless AAC as that's in common usage, is royalty free and their implementation is Apache licensed now IIRC. TGAs are a pain for normal people to work with. Heck, industry standard tools like Photoshop doesn't work with them without plugins.
Also I don't know a thing about them, but you saying "they compress well into an archive" tells me they either have no compression or terrible compression. Files that have been well compressed don't shrink when you throw them at a general compressor (because information theory). Creating content is a lot easier when there are tools to do it. Is it really good use of module creators time to go hunting the internet for some ancient program to create bink videos? Webm is royalty free, widely available and supported by most of the heavy hitters of the web. You can spit out webms even with VLC.
I'd imagine there's still bink video support otherwise, like you say, compatibility is broken and Beamdog have a "no breaking peoples' modules" policy. Personally, I'm of the opinion that using standardised file formats (which people actually use nowadays and can expect their software to support) lowers the barriers to entry for custom content. Lower barriers to entry means more people can play with it. More people playing with it means more people making shiny new content.
Not to mention it stops you having to hunt around dodgy shovelware sites looking for ancient conversion programs that probably install keyloggers.
Also, IIRC there's a proposal to allow adding strings into 2das as well as references to tlk entries. I know this sucks for multiple languages, but it would make prototyping things easier; it's annoying as hell to have to mess around with tlk entries when you're in the middle of messing with adding new 2da entries.
2da files are formatted plain text files that contain data in a tabular fashion (2da stands for 2 dimensional array). tlk files on the other hand are compiled, which is why you need special tools to edit them. Also you can already have strings in 2da files, you just enclose them in quotes. That being the reason I mentioned that excel behaviour.
TR
TR
You're also right on the 2da strings; I totally forgot about that! It's probably because I've only ever really seen strings used for unused comment fields in things like cls_*_feat.2da files.
While I'm posting, just want to reiterate: No CSVs. Just no. Not while Microsoft Excel still infests the earth. Use another spreadsheet format that specifies what the damned character encoding is.
Fun fact about Excel: If you make a .csv file with "ID" (sans quotes) in the cell A1, Excel will refuse to open it. This is a good feature and not at all silly because no one would ever want to have a table where the first column is some form of ID.
He, he neither am I but I do know a bit about electronics and sound.
Not so I'm afraid. The bitrates and sample rates are there to determine the size of the difference in height (when viewed in a graph of the waveform) between one point and the next. All digital audio makes use of analogue filter technology to smooth these transitions so as to simulate analogue sounds.
Currently all you can do is to alter the volume. For anything else you need an external editor such as Audacity.
This probably because TGA is regarded by them as an obsolescent format. It didn't used to be that way. Also the Gimp and paint.net have no trouble with TGAs whatsoever.
It's a bit of both. TGAs have an optional compression scheme (RLE compression) that 1.69 (and therefore by extension EE) doesn't like one little. As far as I can tell TGA is a lossless format. At least it didn't screw up certain textures that I made unlike Bioware dds...
Bink, is still used to create videos for modern games and is still maintained. All it does is to convert compatible video formats to bik. A smooth yet more compressed version of the originals. 1.69 made use of an obsolete decoder and a more modern version needs to be licensed for a fee. Hence the switch.
Nope, no longer supported in EE. I've converted my videos already as soon as the patch was released and the announcement made. One slight wrinkle here is that apparently the toolset wasn't updated and you can only specify bik files in the toolset.
I am of the opposite opinion here. I feel that the fact that uncompiled model files (e.g. plain text) lowers any such barriers in that such models can be edited with a simple text editor and it facilitates the creation of simple tools that would be much harder to create for other formats.
Since the conversion programs are created, maintained and hosted by members of the community this last statement could be construed as being borderline insulting. I am sure this was not your intent.
TR
My point was it's "files I can open in software I've heard of" vs "things that, when I google their file extensions, all I get is hits to websites with names like isthisavirus.com". (the latter is a total strawman and I'm not apologising; it matches my life experiences too well
Incidentally is there a community made bink converter? It was that example in particular that made me think of those halcyon days of trying to get random .mpg files to play in the early 2000s. God the world is such a better place for VLC and it's ilk.
The second place where misconception appears to have crept in is over this last point we have been discussing. When I said -
I was still talking about the proposed changes to 3D model formats and not the the entirety of all the proposals. I think this is where the misunderstanding may have crept in.
There was a short lived thread on the subject of video conversion here. Also there isn't a community made tool for this yet but I think one may be on the way look here.
TR
Presumably the engine will work with any format it supports, the old formats and new formats will get converted into the same internal structure.
I think you missed the word 'Optional' at the start of the sentence.
Having the option of using industry standard formats in addition to the currently supported version would be a blessed relief and potentially save a step in the process of generating custom content.
Well, it's a tool I asked for!
I think you're looking at the word 'Universal' in the wrong way, the idea isn't to support a single specific 'universal' tool that everyone uses, but to support 'universal' file formats that can be created in many different software packages.
Yes, Blender is good and free, and yes GMax can still be downloaded. Both require scripts and plugins and a measure of fiddling about externally to the toolset to get content ready to go in.
The request instead is for the workload of bringing CC into NWN to be removed from the existing limited pool of software options and placed into the toolset. Being able to import and convert .obj, .fbx, .X3d or whatever directly into the toolset opens up a huge amount of options for content creators without requiring them to install and get used to new programs.
As with the .dds format it's about options - options on how to create, texture and import content with new modern tools, rather than relying on the existing older options.
LINK
.obj does not support animation
(best bet).fbx is propriety but does have good range of features and a lot of program support import/export.
.X3d low adoption rate and not many programs support import/export
.dds / some new programs came out that are more modern then gimp and photoshop? Wooh that do not have to have plugins to work with import/export .dds vs exporting tga or a simple plugin that kicks out nwn .DDS format? There is no new options with this format it still be a mipmap. If it has performance improvements vs nwn dds compression (I doubt it) than sure, but it no big wow change.
I feel even if BD started cross support(continue support of mdl) for any new 3d model file formats that they probably could only go so far (possibility of big engine changes) that any of these formats/programs would have to have special plugins/tinkering to get to work.
I would love to hear from someone from inside BD (Trent or programmer/3d modeler) thoughts on this.
As for the audio and video, the updates that BD has already did are perfectly fine.
Any 3d file format must also support hierarchy and meta data for each object in addition to animations like ShadowM said. That completely rules out obj. Fbx is doable, but if you don't want to use plugins then it is out of question too as Maya, Zbrush and Blender all use plugins for fbx.
Most feasible thing I can think of is using an external fbx to mdl converter. Which would have to be created, something the community would do if needed - we'll see how much need there is.
Texture formats:
tga
Photoshop and Gimp both support tga. I haven't seen any 3d modeling software that doesn't support tga. I don't see what the problem is here.
dds
This I can understand. No program supports NWN-dds format and it' a bit of a hassle to have to convert it each time (although the converters come bundled with NWN). I don't know how much overhead it would cause to detect the dds format on each load, if the engine were to support it.
That isn't the biggest issue however: I dread the mess it would create in haks. If i extract a texture, how do I know what format that dds file is in? I suppose I'll know it when gimp or the converter throw an error message, each time, no thank you. That's why I'm against supporting the standard dds format.
Instead I suggest to adapt the gimp dds plugin to add support for NWN-dds format. Doable, I've dabbled with it and hacked in rudimentary reading support a while ago (it's mostly about the header). It's messy and needs more work though, I'll get to it when I find the time - which could take a while. The original plugin seems to have some build problems too.
Standalone converter from multiple file formats, i.e. from standard-dds, png, tga to NWN-dds might be better. AFAIK most custom content creators use tga during development and convert to dds once after they've finalized everything, so it's not too big of a hassle.
Also i agree with symmetric on the Texture formats. Its better for community to create plugins that flooding developers with useless work that might mess the game up.
.fbx for pretty much anything else. Sure it's a propriety format but the Autodesk SDK is free and open to use. It's hugely supported by pretty much every vaguely modern piece of 3d software.
Together they're the primary formats used by most current gen commercial game engines - UE4, Unity, Cryengine, Lumberyard, Stingray with .fbx being the preferred format for all of those. It's a little disingenuous to call out the need for plugins to handle .fbx when all the software you've listed ships with those plugins activated as standard.
If we have the chance to get official, integrated and functioning tools or the capacity to use current file formats in addition to the existing stuff then I honestly fail to see the reason to oppose such a development.
The easier it is to create custom content and get it up and running in game then the more attractive a proposition it becomes for people to do. The more steps and applications you need to handle content, the less enjoyable it is, if we can get to the stage where it's just 'Save in package, open in toolset' then that's just peachy.
We not talking the programs out there we talking where the files have to go to a custom 3d engine.
We talking ease of implementation/ IE best bang for your buck.
Let go over this from the more realistic side of BD
BD not got to rework there engine to read .obj files just so very few placables that are static(most have open/close/on/off and special animations and even tileset can have animations. So BD going to say pass on support for that. On top of that blender/gmax support .obj imports and a simple putting down aurorabase and export and you done.
.fbx again redoing the engine to support for reading mesh/animations/vfx etc.. would be a high cost of man hours and resources and may even cause some snags that would not work with every type of the sub-format so BD might say we got basic meshes in, but you we cannot get the engine to work with the animation or special effects unless they are setup a certain way in any of the programs (example special export of X program) You have to remember we did not even have support for .mdl from bioware when it was a bigger company and at peak company members. A member of the community had to do the work to get a modeling program to work with it. As an example symmetric work on getting mdl to work with blender.
So like you say if their a chance. I realistically telling you there a low chance of it happening and even if it does happen there is a high chance that it going to be limited. I would love to be wrong and it all just comes together nice and neat maybe after a year or two of work on it from BD
I not trying to be a downer because there is option of getting mdl import/export plugins for other programs with more eyes on the game EE release.
Let's turn our attention to model formats. Someone complained about plug-ins being needed for the .mdl format. I've got news for you. Whatever format is used will need plug-ins to make it compatible. The reason is wok, pwk and dwk - in other words walkmeshes. Also possibly nodes (not sure of this one). Let's turn our attention to fbx. Now according to this wikipedia article, fbx is a propriety format just like max (and both owned by the same company). I'll take a wild stab here but I'm guessing this means that BD would have to buy a license to use it? Not going to happen when they already have a working format they can use. Then we come to the little matter of of which version of fbx would you want to use? Now milkshape can load 2 versions (fbx and fbx 2010) but not the version that NwN2 can use (I was interested in a recent release and thought I'd take a look). Then there are the modern versions...
In defence of the .mdl format. I am guessing that most people who will have an opinion on the matter will probably, on average, make no more than 4 versions of a model. I, on the other hand tend to work on a more industrial scale - 270 separate graffiti placeables for this ccc alone (plus the discussion via wayback machine). I can only do this because the .mdl format comes in a plain text format as well as a binary one. All it took was a good plain text editor and a quite a bit of work. Just think how long the graffiti would have taken if I'd had to do it in a specialised 3d modelling program. Plus, because it is plain text it makes writing little utilities to automate some processes a breeze (comparatively). The proposal mentions universal modelling software. There is nothing more universal than a plain text editor.
One thing I have realised (and find interesting) about the original proposal is that both fbx and DDS are used by NwN2... Now that couldn't have anything to do with it, could it?
TR
Besides, it's not like adding assets to Unity and Unreal is one click. The model alone doesn't do anything, there's always some meta data needed.
It seems to me like you'd like in-toolset editors for hak, 2DA and tlk files to edit said meta data without having to resort to (multiple) 3rd party programs.
That would require major changes to the toolset, maybe even to rewrite it from the ground up. I don't see that happening. The way haks are loaded means you can't edit them while a module using them is opened (or at the very least the changes won't show up until you reload that module).
The easier and maybe better approach is to have a extra tool, which can open tlks, haks and allows editing the contained 2DAs or add/view assets on the fly. Here we can talk about conversion of other 3d model formats to MDL. Something to make it easier for people to gather and package the custom content for their module. Of course I'm not opposed to that.
I'll be more specific, why OBJ won't work or at the very least not without providing additional data. It's not just about animations:
- Walkmeshes for tiles: Which material is the ground? Can I walk there? You also need to create an aabb tree put it in a wok file.
- Walkmeshes for placeables and doors: For open state, for closed state and all the other states.
- Should a mesh be rendered, should it cast a shadow (AFAIK obj files have this one), should it fade when the player is near, ...
- Myriad of other thing I'm too lazy to list here
So many things have to be added that manually specifying them is just not feasible, let alone convenient for all but the most primitive models. And I'm talking really primitive.With FBX it's a the very least possible to have custom properties, but I don't know if exporters can handle them. Whether importing them can be done in a convenient fashion is another matter entirely and there are the same walkmesh issues as with OBJ. At least I can see a possibility here.
I'm trying to point out where the problems lie and the reasons why something is unlikely to work.
pwk/wok/dwk could be handled through naming convention or material.
It's not about replacing one format with the other. Adding support for importing / converting other formats in once place (preferably with better .2da editing in the toolset too). Having the capacity to import stuff from .fbx won't magically make the .mdl files become anything but ,mdl so you can do whatever you fancy with the older tools. Rather it's about increasing integrated options and making the whole process easier no matter what software you're using and reduce the amount of 3rd party bits and bobs you need to use. .fbx wouldn't exclude .mdl and having the ability to use the common .dds format wouldn't exlude the current one.
As for NWN2, that uses .dds, .mdl and .gr2 files. I believe there is a program released very recently that allows the conversion of .fbx into .gr2 but it's never been a native option. I don't see what you're getting at here.
Given that the aim is enhancement; 64 bit, new shader model, much higher polycounts, 2gb of video memory, it just seems odd to not want enhanced ways of getting this shiny new content into play. It's about what can be done with all the new capabilities and giving new workflows to creators both new and old.
Overall I'll live in hope that given the clamor for updated textures and higher resolution models that's all over various parts of the forums BD will integrate an easier way to import them rather than relying the existing community 3rd party tools or obsolete software.
TR
Each format raises complex issues and merits its own discussion.
Also, it seems strange that Trello only allows votes "for" but not "against".
Some innovation would be helpful, provided old content remains compatible and tools are updated.
3rd party plug-ins. OK, this one I totally don't get. All modern browsers have the facility for them. Notepad++ has some great ones. So what's the problem. It couldn't be a slight case of hypocrisy, could it?
Expensive professional 3d modelling software. Lucky you. Now for us lesser mortals. Actually I have a question. Where are the community editions of this stuff, because I am not aware of any. If you do programming there's the community edition of Visual Studio and the jetbrains community editions of their tools as a top-of-my-head examples. So we are left with the unsupported gmax, (possibly) milkshape 3D (shareware & I still need the lost plug-in) or Blender which is still maintained. So rather than expanding possibilities this proposal risks shrinking them for the less fortunate.
Oh and the "it's only optional" cry. What happens if I'd like to modify something created in these "modern" formats but I am stuck with something that can't load it?
TR
EDIT: I'm not sure it answers your exact question, but I did find this link which seems to answer it in the negative.
TR
I've no major opposition to 3rd party plugins, the issue I have is that we currently need to rely on people with spare time, capability and drive to make plugins for each individual bit of software available. Having the import/conversion tools built into the NWN toolset eliminates that need and opens up the capacity to make content to a huge range of software.
The more steps there are to getting content functioning the more off putting it is for new people, having to convert textures to unique formats, restricting what software can and cant be used to import - It all puts people off.
Community software editions - I guess that depends on what you mean by this. A lot of the traditionally expensive software packages are available in much lower cost Indie formats marketed at small independent, amateur or hobbiest creators.
Maya LT, Modo Indie (11 is coming apparently and will be freed from Steam), Houdini Indie, 3dCoat Amateur to name some.
If you mean free, then yeah, Blender.
As an aside, for video stuff I always suggest Black Magic Fusion, it's fantastic and it's free.
Again, it's not about limiting possibilities it's about expanding them; adding the capacity to import/convert .fbx or other file formats wouldn't be to the exclusion of the existing formats.
Make it easier for more people to use more packages to make more stuff for EE without fiddly steps or specific software limitations.
- Were FBX supported, how would all the NWN special nodes be delivered alongside the FBX format?
- And how would this new system be optimized to work in parallel with an optimized rendering of existing content?
My take to this suggestion is that while it is probably well meant, it does not solve its implied purpose: to enable new CC makers to use modern and better supported 3D software.Any 3D software that you use for NWN will require a plugin for exporting your work to NWN, and to work with existing content you will need a plugin for importing MDL. If you don't understand why, reread my questions up above and think about them for a bit. Then look at what a tool like Unity does with your assets. Embedding game engine specific info is what the NWN plugins for Blender and Gmax also do. We can't get around this.
You either have the game engine specific info in ancillary files associated with the raw model and animation data, or you embed all this into a single model file. NWN's MDL format simply embeds it all in one file.
So rather than waste our time with this pointless recommendation, lets approach this from the problem and work towards a solution that will work for all of us:
Problem:
A limited set of tools to work with, Gmax or Blender, creates a barrier to entry for new CC makers.
Why is this the case?
Because those two tools are the only ones with publicly released import and export scripts.
Solution:
create more plugins/tools for other modelling software.
Ramble: I wrote my own import export script prototype for Cheetah3D, but never finished because I was busy. All I had was basic model data and texture data. No animations, no particle system, no lights... etc.... And I only had this working for text files as I didn't master the binary parsing and compilation. This takes a lot of work. Just ask Symmetric. But - and this is the important take away - it is achievable even by an entry level skilled coder like me if they are willing to put in the time.
Alternative Solution:
Develop a middleware import/export tool which abstracts the MDL into categories of things: (1) the stuff handled by most model formats (geometry, tvert mapping), (2) animations, and (3) NWN specific data nodes.
This tool would then allow a user to do the following:
- Import a MDL and break it into parts
- Import geometry and animations in FBX format and maybe others
- Export geometry and animations to FBX format and maybe others
- Combine all the parts and export as a MDL
Ramble: Ofcourse there is also more complex stuff we'd want it to do ... like handle tile and tileset editing and creation. But that would be like a second tier of features to implement after we had the first phase working.
Ideally this would be part of an new NWN toolset and HAK editor combo which worked like a modern game dev environment. Hahahahah. Yeah I am getting crazy with this one. Think of this last one as shooting for the moon, and the first part of this solution as the realistic target.
This would be more involved to develop, but its an all in one solution that would not require the community or beamdog to write and maintain import/export scripts.
FBX shouldn't replace or likely even exist in-game along side the .MDL format, but rather be easy to import/convert within the toolset (or, grudgingly, in a stand alone converter).
Individual scripts/plugins for the many software packages out there shouldn't be an option, partly because there's too many packages and partly because a lot of the indie style packages don't support/allow them.
Besides the idea is to break away from specific software dependency and get a more universal tool.
(This could possibly do with being broken out into a separate topic?)
Now we know from the last stream that these kind of tools will be developed, so this is our chance to shape them to fit our exact needs.
So, ideal world, what tools or workflow would make sense for you?
The way I see it something like an 'Import Asset Wizard' built into the toolset would be the best option (or as a plugin, if the toolset can be opened to allow people to create plugins)
Allowing you to choose .FBX, maybe other formats to import/convert to nwn format.
Allow you to choose the type of asset you're importing and spit out an appropriate row of info for you to cut and paste into the appropriate .2da.
Tool should handle attributes or other meshes via suffix (for walkmesh, LoD, and so on) or via tag or map.
Ideally a part of a larger set of CC management/creation tools.